Cognitive decline and dementia due to Alzheimer's disease (AD) have been associated with genetic, lifestyle, and environmental factors. A number of potentially modifiable risk factors should be taken into account when preventive or ameliorative interventions targeting dementia and its preclinical stages are investigated. Bone mineral density (BMD) and body composition are two such potentially modifiable risk factors, and their association with cognitive decline was investigated in this study. 164 participants, aged 34–87 years old (62.78 ± 9.27), were recruited for this longitudinal study and underwent cognitive and clinical examinations at baseline and after 3 years. Blood samples were collected for apolipoprotein E (APOE) genotyping and dual energy x-ray absorptiometry (DXA) was conducted at the same day as cognitive assessment. Using hierarchical regression analysis, we found that BMD and lean body mass, as measured using DXA were significant predictors of episodic memory. Age, gender, APOE status, and premorbid IQ were controlled for. Specifically, the List A learning from California Verbal Learning Test was significantly associated with BMD and lean mass both at baseline and at follow up assessment. Our findings indicate that there is a significant association between BMD and lean body mass and episodic verbal learning. While the involvement of modifiable lifestyle factors in human cognitive function has been examined in different studies, there is a need for further research to understand the potential underlying mechanisms.