Southern China is the birthplace of rice-cultivating agriculture and different language families and has also witnessed various human migrations that facilitated cultural diffusions. The fine-scale demographic history in situ that forms present-day local populations, however, remains unclear. To comprehensively cover the genetic diversity in East and Southeast Asia, we generated genome-wide SNP data from 211 present-day Southern Chinese and co-analyzed them with ∼1,200 ancient and modern genomes. In Southern China, language classification is significantly associated with genetic variation but with a different extent of predictability, and there is strong evidence for recent shared genetic history particularly in Hmong–Mien and Austronesian speakers. A geography-related genetic sub-structure that represents the major genetic variation in Southern East Asians is established pre-Holocene and its extremes are represented by Neolithic Fujianese and First Farmers in Mainland Southeast Asia. This sub-structure is largely reduced by admixture in ancient Southern Chinese since > ∼2,000 BP, which forms a “Southern Chinese Cluster” with a high level of genetic homogeneity. Further admixture characterizes the demographic history of the majority of Hmong–Mien speakers and some Kra-Dai speakers in Southwest China happened ∼1,500–1,000 BP, coeval to the reigns of local chiefdoms. In Yellow River Basin, we identify a connection of local populations to genetic sub-structure in Southern China with geographical correspondence appearing > ∼9,000 BP, while the gene flow likely closely related to “Southern Chinese Cluster” since the Longshan period (∼5,000–4,000 BP) forms ancestry profile of Han Chinese Cline.