Study Objectives: To investigate the contribution of sleep duration and quality to age-related changes in brain structure and cognitive performance in relatively healthy older adults. Design: Community-based longitudinal brain and cognitive aging study using a convenience sample. Setting: Participants were studied in a research laboratory. Participants: Relatively healthy adults aged 55 y and older at study commencement. Interventions: N/A. Measurements and Results: Participants underwent magnetic resonance imaging and neuropsychological assessment every 2 y. Subjective assessments of sleep duration and quality and blood samples were obtained. Each hour of reduced sleep duration at baseline augmented the annual expansion rate of the ventricles by 0.59% (P = 0.007) and the annual decline rate in global cognitive performance by 0.67% (P = 0.050) in the subsequent 2 y after controlling for the effects of age, sex, education, and body mass index. In contrast, global sleep quality at baseline did not modulate either brain or cognitive aging. High-sensitivity C-reactive protein, a marker of systemic inflammation, showed no correlation with baseline sleep duration, brain structure, or cognitive performance. Conclusions: In healthy older adults, short sleep duration is associated with greater age-related brain atrophy and cognitive decline. These associations are not associated with elevated inflammatory responses among short sleepers.