Accumulating research indicates that handgrip strength is associated with cognitive function. Studies have also shown the difference in cognitive decline between males and females. We investigated the association between baseline handgrip strength and later cognitive function in older adults according to sex using the dataset from Korean Longitudinal Study of Aging (2006–2018). Overall, 9707 observations of 1750 participants (989 males and 761 females) over 65 years of age were sampled from the first wave, followed by six consecutive waves. The Korean version of the Mini-Mental State Examination and baseline handgrip strength scores were assessed. Sociodemographic and health-related variables were also included as covariates in the multivariable linear mixed models. Males in the lowest quartile of the baseline handgrip strength decreased in cognitive function (β = −0.54, standard error (SE) = 0.16, p < 0.001), compared to males in the highest quartile. For females, those in the second lowest quartile (β = −0.65, SE = 0.19, p < 0.001) and the lowest quartile (β = −0.53, SE = 0.19, p< 0.01) decreased in cognitive function. Handgrip strength may be positively associated with later cognitive function, but the association may be non-linear and differ between sexes. Sex-specific preventive assessment of handgrip strength may help identify older adults at risk for cognitive impairment.