Each mammalian female cell transcriptionally inactivates one X chromosome to balance X-linked gene dosage between males and females. This phenomenon, called X chromosome inactivation, is a perfect epigenetic event, in which two chromosomes with identical DNA sequences are solely distinguished by epigenetic modifications. In this case, epigenetic marks, such as histone modifications, histone variants, DNA methylation, and ncRNAs, are all enriched on one chromosome, the inactive X chromosome (Xi), to establish its chromosome-wide gene silencing. At face value, it seems that the gene silencing mechanism of Xi is well understood. However, the "silence" of Xi in somatic cells is so tightly maintained that it remains largely intact even after almost all known epigenetic modifications are artificially depleted. To understand how the gene silence of Xi is maintained in soma is a major challenge in current research. We summarize the current knowledge related with this issue and discuss future research directions.