Background Infants with congenital heart disease (CHD) are at risk of neurodevelopmental impairments, which may be associated with impaired brain growth. We characterized how perioperative brain growth in infants with CHD deviates from typical trajectories and assessed the relationship between individualized perioperative brain growth and clinical risk factors. Methods and Results A total of 36 infants with CHD underwent preoperative and postoperative brain magnetic resonance imaging. Regional brain volumes were extracted. Normative volumetric development curves were generated using data from 219 healthy infants. Z‐ scores, representing the degree of positive or negative deviation from the normative mean for age and sex, were calculated for regional brain volumes from each infant with CHD before and after surgery. The degree of Z ‐score change was correlated with clinical risk factors. Perioperative growth was impaired across the brain, and it was associated with longer postoperative intensive care stay (false discovery rate P <0.05). Higher preoperative creatinine levels were associated with impaired brainstem, caudate nuclei, and right thalamus growth (all false discovery rate P =0.033). Older postnatal age at surgery was associated with impaired brainstem and right lentiform growth (both false discovery rate P =0.042). Longer cardiopulmonary bypass duration was associated with impaired brainstem and right caudate growth (false discovery rate P <0.027). Conclusions Infants with CHD can have impaired brain growth in the immediate postoperative period, the degree of which associates with postoperative intensive care duration. Brainstem growth appears particularly vulnerable to perioperative clinical course, whereas impaired deep gray matter growth was associated with multiple clinical risk factors, possibly reflecting vulnerability of these regions to short‐ and long‐term hypoxic injury.