To characterize the neuroanatomical changes in healthy older adults is important to differentiate pathological from normal brain structural aging. The present study investigated the annualized rate of GM atrophy in a large sample of older participants, focusing on the hippocampus, and searching for modulation by age and sex. In this 4-year longitudinal community cohort study, we used a VBM analysis to estimate the annualized rate of GM loss, at both the global and regional levels, in 1,172 healthy older adults (65–82 years) scanned at 1.5T. The global annualized rate of GM was −4.0 cm3/year (−0.83%/year). The highest rates of regional GM loss were found in the frontal and parietal cortices, middle occipital gyri, temporal cortex and hippocampus. The rate of GM atrophy was higher in women (−4.7 cm3/year, −0.91%/year) than men (−3.3 cm3/year, −0.65%/year). The global annualized rate of GM atrophy remained constant throughout the age range of the cohort, in both sexes. This pattern was replicated at the regional level, with the exception of the hippocampi, which showed a rate of GM atrophy that accelerated with age (2.8%/year per year of age) similarly for men and women. The present study reports a global and regional description of the annualized rate of grey matter loss and its evolution after the age of 65. Our results suggest greater anatomical vulnerability of women in late life and highlight a specific vulnerability of the hippocampus to the aging processes after 65 years of age.