Abstract Background Neuroanatomical differences between individuals with and without autism spectrum disorder (ASD) were inconsistent in the literature. Such heterogeneity may substantially originate from age-differential effects. Methods Voxel-based morphometry was applied in 86 males with ASD and 90 typically developing control (TDC) males (aged 7 to 29 years). Three steps of statistical modeling (model 1, multiple regression with age as a covariate; model 2, multiple regression further considering diagnosis-by-age interaction; model 3, age-stratified analyses) were performed to dissect the moderating effects of age on diagnostic group differences in neuroanatomy. Results Across ages, males with and without ASD did not differ significantly in total gray matter (GM) or white matter (WM) volumes. For both groups, total GM volumes decreased and WM volumes increased with age. For regional volume, comparing with the model only held the age constant (model 1), the main effect of group altered when diagnosis-by-age interaction effects were considered (model 2). Here, participants with ASD had significantly greater relative regional GM volumes than TDC in the right inferior orbitofrontal cortex and bilateral thalamus; for WM, participants with ASD were larger than TDC in the bilateral splenium of corpus callosum and right anterior corona radiata. Importantly, significant diagnosis-by-age interactions were identified at the bilateral anterior prefrontal cortex, bilateral cuneus, bilateral caudate, and the left cerebellum Crus I for GM and left forceps minor for WM. Finally, age-stratified analyses (model 3) showed distinct patterns in GM and WM volumetric alterations in ASD among subsamples of children, adolescents, and adults. Conclusions Our findings suggest that the heterogeneous reports on the atypical neuroanatomy of ASD may substantially originate from age variation in the study samples. Age variation and its methodological and biological implications have to be carefully delineated in future studies of the neurobiology of ASD.