Abstract Background Increasingly, lifestyle factors in midlife are reported to impact health and functional status in old age. This work examines associations between dietary trajectories in middle age and subsequent impairments in physical function. Method Data are from 851 participants (61% men, mean age at first dietary assessment = 47 years, range 30–59 years) from the Baltimore Longitudinal Study of Aging. We used latent class analysis to derive dietary trajectories based on adherence to the Alternative Healthy Eating Index-2010 (AHEI), and further classified them based on tertiles, as poor (score <39.3), intermediate (39.3–48.9), or good (>48.9). Physical function was assessed with the Short Physical Performance Battery (SPPB). Random effects tobit regression models were used to examine associations between dietary trajectories and later physical function. Results Two latent classes of AHEI scores were generated and labeled “greatly improved” or “moderately improved.” In the greatly improved class, participants showed a trend in overall AHEI score from poor/intermediate to good diet categories across dietary assessments with age, over time. In the moderately improved class, the overall AHEI score shifted from poor to intermediate diet categories over time, and the prevalence of the good diet category remained low. Mean AHEI score between ages 30 and 59 years was higher in the greatly, than moderately, improved class. The moderately improved class had 1.6 points lower SPPB score (indicating poorer physical function) at older age than the greatly improved class (p = .022). Conclusions Findings suggest that improving diet quality in middle age may contribute to better physical function in older age.