The rhythms of activity across the 24-h sleep-wake cycle, determined in part by the circadian clock, change with aging. Few large-scale studies measured the activity rhythm objectively in the general population. The present population-based study in middle-aged and elderly persons evaluated how activity rhythms change with age, and additionally investigated sociodemographics, mental health, lifestyle, and sleep characteristics as determinants of rhythms of activity. Activity rhythms were measured objectively with actigraphy. Recordings of at least 96 h (138 ± 14 h, mean ± SD) were collected from 1734 people (age: 62 ± 9.4 yrs) participating in the Rotterdam Study. Activity rhythms were quantified by calculating interdaily stability, i.e., the stability of the rhythm over days, and intradaily variability, i.e., the fragmentation of the rhythm relative to its 24-h amplitude. We assessed age, gender, presence of a partner, employment, cognitive functioning, depressive symptoms, body mass index (BMI), coffee use, alcohol use, and smoking as determinants. The results indicate that older age is associated with a more stable 24-h activity profile (β = 0.07, p = 0.02), but also with a more fragmented distribution of periods of activity and inactivity (β = 0.20, p < 0.001). Having more depressive symptoms was related to less stable (β = -0.07, p = 0.005) and more fragmented (β = 0.10, p < 0.001) rhythms. A high BMI and smoking were also associated with less stable rhythms (BMI: β = -0.11, p < 0.001; smoking: β = -0.11, p < 0.001) and more fragmented rhythms (BMI: β = 0.09, p < 0.001; smoking: β = 0.11, p < 0.001). We conclude that with older age the 24-h activity rhythm becomes more rigid, whereas the ability to maintain either an active or inactive state for a longer period of time is compromised. Both characteristics appear to be important for major health issues in old age.