Inside the nucleus, DNA replication is organized at discrete sites called replication factories, consisting of DNA polymerases and other replication proteins. Replication factories play important roles in coordinating replication and in responding to replication stress. However, it remains unknown how replicons are organized for processing at each replication factory. Here we address this question using budding yeast. We analyze how individual replicons dynamically organized a replication factory using live-cell imaging and investigate how replication factories were structured using super-resolution microscopy. Surprisingly, we show that the grouping of replicons within factories is highly variable from cell to cell. Once associated, however, replicons stay together relatively stably to maintain replication factories. We derive a coherent genome-wide mathematical model showing how neighboring replicons became associated stochastically to form replication factories, which was validated by independent microscopy-based analyses. This study not only reveals the fundamental principles promoting replication factory organization in budding yeast, but also provides insight into general mechanisms by which chromosomes organize sub-nuclear structures. © 2013 Shigeoka et al.