<i>Background:</i> The apolipoprotein E (ApoE) genotype has been confirmed as the major genetic risk factor for late-onset Alzheimer’s disease (AD). How the ApoE genotype and brain morphology relate to each other is only partly understood, particularly in mild cognitive impairment, the assumed prestage of AD. <i>Methods:</i> A total of 83 subjects with mild cognitive impairment (aging-associated cognitive decline criteria) were investigated with optimized voxel-based morphometry (VBM). We tested for differences in gray and white matter densities between groups according to their ApoE status, i.e. Ε4 allele noncarriers (n = 42), subjects with one Ε4 allele (n = 27) and subjects with two Ε4 alleles (n = 14). <i>Results:</i> In individuals carrying two Ε4 alleles, VBM revealed a decline in gray matter density predominantly in the medial temporal lobe region. Subjects with a single copy of the Ε4 allele exhibited gray matter atrophy in the right inferior frontal gyrus. With respect to white matter changes, atrophy was only found in subjects homozygous for Ε4 and confined to the right superior and middle temporal gyrus. <i>Conclusion:</i> Our findings support the hypothesis that the ApoE genotype in mild cognitive impairment might be associated with structural changes typically found in the early stages of AD.