Significance Our physical senses are separated not only into distinct experiences but also into specialized regions within the cerebral cortex. Synesthesia is a neurological phenomenon that causes unusual links between sensory experiences, and its molecular basis is completely unknown. We demonstrate that three families who experience color when listening to sounds are connected by rare genetic variants affecting genes that contribute to axonogenesis, a process essential for neuronal connections within and across brain regions. Multiple genes with similar activity patterns during neural development fall within parts of the genome previously linked to the condition. Our results connect synesthetes’ altered structural and functional connectivity to genes that support the development of those connections.