Background: With the exception of APOE, genetic variants associated with increased Alzheimer’s disease (AD) risk are characterized by small effect sizes. Polygenic risk scores (PRS) have shown utility in predicting AD risk; however, their utility for predicting decline in cognition at preclinical stages of AD is poorly understood. Objective: To validate associations of a 22-variant AD-risk-weighted PRS with AD risk and related biomarkers and to assess its utility to predict cognitive decline. Methods: The PRS was evaluated with respect to brain amyloid-β (Aβ) burden, cerebrospinal fluid (CSF) Aβ42, total-tau, and phospho-tau, and decline in cognition in 643 (570 cognitively normal (CN), 73 AD) PET-imaged participants from the longitudinal Australian Imaging, Biomarkers and Lifestyle (AIBL) Study of Ageing. Cognition was assessed using three composite measures; global cognition, verbal episodic memory, and a Pre-Alzheimer’s Cognitive Composite (PACC). Results: PRS, both with and without APOE, were positively correlated with brain Aβ burden, CSF total-tau, and phospho-tau in CN older adults. Further, in CN biomarker positive (Aβhigh) participants, significant associations were observed with baseline and longitudinal cognition. However, this association was not observed after the removal of APOE. Partitioning the PRS into quartiles revealed that the PRS associations with cognitive decline in Aβhigh CN older adults is due to a saturating effect of APOE genotype. Conclusions: An AD-risk-weighted PRS is associated with cognitive decline in CN older adults. However, this association is absent when APOE genotype is excluded from the PRS, suggesting that associations with cognitive decline in this model of polygenic risk are driven by APOE genotype alone. Further research is needed to define appropriate PRSs with greater utility for predicting preclinical AD cognitive decline.