Abstract The human brain is a complex organ comprised of distinct cell types, and the contribution of the 3D genome to lineage specific gene expression remains poorly understood. To decipher cell type specific genome architecture, and characterize fine scale changes in the chromatin interactome across neural development, we compared the 3D genome of the human fetal cortical plate to that of neurons and glia isolated from the adult prefrontal cortex. We found that neurons have weaker genome compartmentalization compared to glia, but stronger TADs, which emerge during fetal development. Furthermore, relative to glia, the neuronal genome shifts more strongly towards repressive compartments. Neurons have differential TAD boundaries that are proximal to active promoters involved in neurodevelopmental processes. CRISPRi on CNTNAP2 in hIPSC-derived neurons reveals that transcriptional inactivation correlates with loss of insulation at the differential boundary. Finally, re-wiring of chromatin loops during neural development is associated with transcriptional and functional changes. Importantly, differential loops in the fetal cortex are associated with autism GWAS loci, suggesting a neuropsychiatric disease mechanism affecting the chromatin interactome. Furthermore, neural development involves gaining enhancer-promoter loops that upregulate genes that control synaptic activity. Altogether, our study provides multi-scale insights on the 3D genome in the human brain.