Over 90% of children with Autism Spectrum Disorders (ASD) demonstrate atypical sensory behaviors. In fact, hyper- or hyporeactivity to sensory input or unusual interest in sensory aspects of the environment is now included in the DSM-5 diagnostic criteria. However, there are children with sensory processing differences who do not meet an ASD diagnosis but do show atypical sensory behaviors to the same or greater degree as ASD children. We previously demonstrated that children with Sensory Processing Disorders (SPD) have impaired white matter microstructure, and that this white matter microstructural pathology correlates with atypical sensory behavior. In this study, we use diffusion tensor imaging (DTI) fiber tractography to evaluate the structural connectivity of specific white matter tracts in boys with ASD (n = 15) and boys with SPD (n = 16), relative to typically developing children (n = 23). We define white matter tracts using probabilistic streamline tractography and assess the strength of tract connectivity using mean fractional anisotropy. Both the SPD and ASD cohorts demonstrate decreased connectivity relative to controls in parieto-occipital tracts involved in sensory perception and multisensory integration. However, the ASD group alone shows impaired connectivity, relative to controls, in temporal tracts thought to subserve social-emotional processing. In addition to these group difference analyses, we take a dimensional approach to assessing the relationship between white matter connectivity and participant function. These correlational analyses reveal significant associations of white matter connectivity with auditory processing, working memory, social skills, and inattention across our three study groups. These findings help elucidate the roles of specific neural circuits in neurodevelopmental disorders, and begin to explore the dimensional relationship between critical cognitive functions and structural connectivity across affected and unaffected children.