Children with autism spectrum disorder (ASD) are highly variable in their language abilities, but the neural bases of these individual differences are poorly understood. Structural magnetic resonance imaging (MRI) and magnetic resonance diffusion tensor imaging (DTI) tractography were used to examine asymmetries in language-related gray- and white-matter and their relationships to language ability in a sample of 20 children with ASD, aged 4-7 years, and a reference sample of 20 typically developing (TD) children, aged 6-11 years. Children with ASD did not differ significantly from TD children in gray matter asymmetries, but were significantly less left-lateralized than TD children in the volume and radial diffusivity (RD) of the arcuate fasciculus (AF). They did not differ in the fractional anisotropy (FA) or the mean or axial diffusivity of the AF. Within the ASD group, exploratory analyses revealed that decreased leftward/increased rightward asymmetry of pars opercularis was associated with higher language ability and bilaterally increased FA and decreased RD of the AF. In conclusion, children with ASD exhibited atypical asymmetry in language-related white-matter structure as well as an atypical pattern of brain-language relationships that suggest that they may meet language milestones and acquire normal language via a different neurodevelopmental trajectory from TD children.