Abstract Background and Objectives Sleep difficulties are common among older adults and are associated with cognitive decline. We used data from a large, nationally representative longitudinal survey of adults aged older than 50 in the United States to examine the relationship between specific sleep difficulties and cognitive function over time. Research Design and Methods Longitudinal data from the 2004–2014 waves of the Health and Retirement Study were used in the current study. We examined sleep difficulties and cognitive function within participants and across time (n = 16 201). Sleep difficulty measures included difficulty initiating sleep, nocturnal awakenings, early morning awakenings, and waking up feeling rested from rarely/never (1) to most nights (3). The modified Telephone Interview for Cognitive Status was used to measure cognitive function. Generalized linear mixed models were used with time-varying covariates to examine the relationship between sleep difficulties and cognitive function over time. Results In covariate-adjusted models, compared to “never” reporting sleep difficulty, difficulty initiating sleep “most nights” was associated with worse cognitive function over time (Year 2014: b = −0.40, 95% CI: −0.63 to −0.16, p < .01) as was difficulty waking up too early “most nights” (Year 2014: b = −0.31, 95% CI: −0.56 to −0.07, p < .05). In covariate-adjusted analyses, compared to “never” reporting waking up feeling rested, cognitive function was higher among those who reported waking up feeling rested “some nights” (Year 2010: b = 0.21, 95% CI: 0.02 to 0.40, p < .05). Discussion and Implications Our findings highlight an association between early morning awakenings and worse cognitive function, but also an association between waking up feeling rested and better cognitive function over time.