(1) Cognitive decline differs among individuals and cognition function domains. We sought to identify distinct groups of immediate and delayed verbal memory in two age subsamples (50–64, 65+ years), and to analyze associated factors. (2) Latent class mixed models were used to identify verbal memory trajectories in a sample of Spanish community-dwelling individuals over 8 years’ follow up. Chi-square and Kruskal–Wallis tests were used to assess differences among trajectories. (3) Different trajectories were identified. In the case of immediate verbal memory, these were: very low/decline (6.3%), low/stable (38.2%), medium/slow decline (43.4%), and high/slow decline (12.2%) in the middle-aged group, and low/decline (20.4%), medium/slow decline (60.4%), and high/slow decline (19.2%) in the older subsample. In delayed verbal memory, more distinct patterns were found: very low/decline (12.4%), low/stable (51.4%), medium/accelerated decline (24.7%), and high/slow increase (11.4%) in the younger group, and low/slow decline (34.4%), medium/decline (52.7%), and high/slow decline (12.9%) in the older group. (4) Overall, low initial performance and decline were associated with older age, lower education, and higher diabetes/stroke prevalence. Differences found suggests heterogeneity in cognitive ageing. The high prevalence of cardiovascular diseases in those with worse cognition suggests that early interventions to prevent those conditions should be targeted in midlife to delay cognitive decline.