Significance Colorectal cancer (CRC) is a heterogeneous disease, with significant variation in genotype and phenotype within each individual tumor. This intratumor heterogeneity emerges during tumor development due to clonal evolution and in part can explain therapy resistance in CRC. However, a detailed understanding of the spatiotemporal development of tumors underlying cancer evolution and intratumor heterogeneity remains absent. Here, we use lineage-tracing experiments of human CRC cells transplanted into immunocompromised mice, in combination with computational modeling, to study the growth mode of CRC. We found that the clonal position is crucial for clonal outgrowth. This demonstrates that, in addition to the genetic composition, the environment and the geometry of tumor growth play a significant role in shaping tumor evolution.