Smaller total brain and subcortical volumes have been linked to psychopathology including attention-deficit/hyperactivity disorder (ADHD). Identifying mechanisms underlying these alterations, therefore, is of great importance. We investigated the role of gene-environment interactions (GxE) in interindividual variability of total gray matter (GM), caudate, and putamen volumes. Brain volumes were derived from structural magnetic resonance imaging scans in participants with (N = 312) and without ADHD (N = 437) from N = 402 families (age M = 17.00, SD = 3.60). GxE effects between DAT1, 5-HTT, and DRD4 and social environments (maternal expressed warmth and criticism; positive and deviant peer affiliation) as well as the possible moderating effect of age were examined using linear mixed modeling. We also tested whether findings depended on ADHD severity. Deviant peer affiliation was associated with lower caudate volume. Participants with low deviant peer affiliations had larger total GM volumes with increasing age. Likewise, developmentally sensitive GxE effects were found on total GM and putamen volume. For total GM, differential age effects were found for DAT1 9-repeat and HTTLPR L/L genotypes, depending on the amount of positive peer affiliation. For putamen volume, DRD4 7-repeat carriers and DAT1 10/10 homozygotes showed opposite age relations depending on positive peer affiliation and maternal criticism, respectively. All results were independent of ADHD severity. The presence of differential age-dependent GxE effects might explain the diverse and sometimes opposing results of environmental and genetic effects on brain volumes observed so far.