Sleep problems have been associated with cognition, both cross-sectionally and longitudinally. Specific genes have been also associated with both sleep regulation and cognition. In a large group of older non-demented adults, we aimed to (a) validate the association between Sleep Polygenic Risk Score (Sleep PRS) and self-reported sleep duration, and (b) examine the association between Sleep PRS and cognitive changes in a three-year follow-up. Participants were drawn from the Hellenic Longitudinal Investigation of Aging and Diet (HELIAD). A structured, in-person interview, consisting of a medical history report and physical examination, was conducted for each participant during each of the visits (baseline and first follow-up). In total, 1376 participants were included, having all demographic, genetic, and cognitive data, out of which, 688 had at least one follow-up visit. In addition, an extensive neuropsychological assessment examining five cognitive domains (memory, visuo-spatial ability, attention/speed of processing, executive function, and language) was administered. A PRS for sleep duration was created based on previously published, genome-wide association study meta-analysis results. In order to assess the relationship between the Sleep PRS and the rate of cognitive change, we used generalized estimating equations analyses. Age, sex, education, ApolipoproteinE-ε4 genotype status, and specific principal components were used as covariates. On a further analysis, sleep medication was used as a further covariate. Results validated the association between Sleep PRS and self-reported sleep duration (B = 1.173, E-6, p = 0.001). Further, in the longitudinal analyses, significant associations were indicated between increased Sleep PRS and decreased visuo-spatial ability trajectories, in both the unadjusted (B = −1305.220, p = 0.018) and the adjusted for the covariates model (B = −1273.59, p = 0.031). Similarly, after adding sleep medication as a covariate (B = −1372.46, p = 0.019), none of the associations between Sleep PRS and the remaining cognitive domains were significant. PRS indicating longer sleep duration was associated with differential rates of cognitive decline over time in a group of non-demented older adults. Common genetic variants may influence the association between sleep duration and healthy aging/cognitive health.