X-Chromosome Inactivation (XCI) is the process whereby one, randomly chosen X becomes transcriptionally silenced in female cells. XCI is governed by the Xic, a locus on the X encompassing an array of genes which interact with each other and with key molecular factors. The mechanism, though, establishing the fate of the X's, and the corresponding alternative modifications of the Xic architecture, is still mysterious. In this study, by use of computer simulations, we explore the scenario where chromatin conformations emerge from its interaction with diffusing molecular factors. Our aim is to understand the physical mechanisms whereby stable, non-random conformations are established on the Xic's, how complex architectural changes are reliably regulated, and how they lead to opposite structures on the two alleles. In particular, comparison against current experimental data indicates that a few key cis-regulatory regions orchestrate the organization of the Xic, and that two major molecular regulators are involved.