In children who are born small for gestational age (SGA), an adverse intrauterine environment has led to underdevelopment of both the body and the brain. The delay in body growth is (partially) restored during the first two years in a majority of these children. In addition to a negative influence on these physical parameters, decreased levels of intelligence and cognitive impairments have been described in children born SGA. In this study, we used magnetic resonance imaging to examine brain anatomy in 4- to 7-year-old SGA children with and without complete bodily catch-up growth and compared them to healthy children born appropriate for gestational age. Our findings demonstrate that these children strongly differ on brain organisation when compared with healthy controls relating to both global and regional anatomical differences. Children born SGA displayed reduced cerebral and cerebellar grey and white matter volumes, smaller volumes of subcortical structures and reduced cortical surface area. Regional differences in prefrontal cortical thickness suggest a different development of the cerebral cortex. SGA children with bodily catch-up growth constitute an intermediate between those children without catch-up growth and healthy controls. Therefore, bodily catch-up growth in children born SGA does not implicate full catch-up growth of the brain.