A new level of chromosome organization, Topologically Associating Domains (TADs), was recently uncovered by chromosome-confirmation-capture (3C) techniques. To explore TAD structure and function, we developed a polymer model that can extract the full repertoire of chromatin conformations within TADs from population-based 3C data. This model predicts actual physical distances and to what extent chromosomal contacts vary between cells. It also identifies interactions within single TADs that stabilize boundaries between TADs and allows us to identify and genetically validate key structural elements within TADs. Combining the model’s predictions with high-resolution DNA FISH and quantitative RNA FISH for TADs within the X-inactivation center (Xic), we dissect the relationship between transcription and spatial proximity to cis-regulatory elements. We demonstrate that contacts between potential regulatory elements occur in the context of fluctuating structures rather than stable loops and propose that such fluctuations may contribute to asymmetric expression in the Xic during X inactivation.