Effects of age and sex, and their interaction on the structural brain anatomy of healthy elderly were assessed thanks to a cross-sectional study of a cohort of 662 subjects aged from 63 to 75 years. T1- and T2-weighted MRI scans were acquired in each subject and further processed using a voxel-based approach that was optimized for the identification of the cerebrospinal fluid (CSF) compartment. Analysis of covariance revealed a classical neuroanatomy sexual dimorphism, men exhibiting larger gray matter (GM), white matter (WM), and CSF compartment volumes, together with larger WM and CSF fractions, whereas women showed larger GM fraction. GM and WM were found to significantly decrease with age, while CSF volume significantly increased. Tissue probability map analysis showed that the highest rates of GM atrophy in this age range were localized in primary cortices, the angular and superior parietal gyri, the orbital part of the prefrontal cortex, and in the hippocampal region. There was no significant interaction between "Sex" and "Age" for any of the tissue volumes, as well as for any of the tissue probability maps. These findings indicate that brain atrophy during the seventh and eighth decades of life is ubiquitous and proceeds at a rate that is not modulated by "Sex".