Abstract Long-term disturbances in cortisol levels might affect brain structure in individuals with autoimmune Addison’s disease (AAD). This study investigated gray and white matter brain structure in a cohort of young adults with AAD. T1- and diffusion-weighted images were acquired for 52 individuals with AAD and 70 healthy controls, aged 19–43 years, using magnetic resonance imaging. Groups were compared on cortical thickness, surface area, cortical gray matter volume, subcortical volume (FreeSurfer), and white matter microstructure (FSL tract-based spatial statistics). Individuals with AAD had 4.3% smaller total brain volume. Correcting for head size, we did not find any regional structural differences, apart from reduced volume of the right superior parietal cortex in males with AAD. Within the patient group, a higher glucocorticoid (GC) replacement dose was associated with smaller total brain volume and smaller volume of the left lingual gyrus, left rostral anterior cingulate cortex, and right supramarginal gyrus. With the exception of smaller total brain volume and potential sensitivity of the parietal cortex to GC disturbances in men, brain structure seems relatively unaffected in young adults with AAD. However, the association between GC replacement dose and reduced brain volume may be reason for concern and requires follow-up study.