BACKGROUND: Elevated cortisol levels due to hypothalamic-pituitary-adrenal (HPA) axis stress response have been associated with cognitive impairment. However, the causal relationship between stress and subsequent cognitive impairment remains unclear, notably because of the small number of gender-stratified prospective studies. METHOD: Salivary cortisol secretion was evaluated in 197 non-depressed community-dwelling elderly people at three time points on the day of hospital attendance for a clinical examination and again on the following day at home, in a distinct environmental context. Cognitive performance was evaluated at baseline and at 2- and 4-year follow-up. RESULTS: Cross-sectional logistic analyses adjusted for age and education indicated that men with high morning cortisol at the hospital had higher risk of low cognitive performance in verbal fluency [odds ratio (OR) 3.0, p=0.05] and visuospatial performance (OR 5.1, p=0.03). Impairment in verbal fluency was observed in women with moderate high morning cortisol (OR 3.6, p=0.05) or moderate slow diurnal rhythm (OR 3.7, p=0.04). In longitudinal analyses, slow diurnal rhythm (flatter slope) was associated with decline over 4 years in visuospatial performance (OR 7.7, p=0.03) and visual memory (OR 4.1, p=0.03) in men, and in verbal fluency (OR 6.0, p=0.01) in women. High morning cortisol was associated with decline in visual memory in women (OR 5.1, p=0.06). CONCLUSIONS: HPA axis dysregulation seems to be associated with low cognitive performance in the elderly. Slower cortisol elimination rates could predict cognitive decline affecting principally non-verbal functioning in men and verbal functioning in women. The effects are independent of environmental context, apolipoprotein E (ApoE) genotype or psychopathology. Interventions blocking this pathway may provide new therapeutic options to prevent cognitive decline.