Abstract Background Vision loss is associated with increased risk of falls and restricted physical activity, yet the relationship between multiple vision measures and objectively measured physical activity, especially activity patterns, in mid-to-late life is not well understood. Method This study included 603 participants aged 50 years and older (mean age = 73.5) in the Baltimore Longitudinal Study of Aging who had the following assessments: presenting and best-corrected visual acuity, contrast sensitivity, visual fields, stereo acuity, and free-living physical activity using a wrist-worn ActiGraph accelerometer for 7 days. Linear regression models were used to examine the association between vision measures and daily activity counts, active minutes, and activity fragmentation (defined as an active-to-sedentary transition probability), adjusting for potential confounders. Mixed-effects models estimated differences in activity by time of day comparing those with and without each visual impairment. Results In the fully adjusted model, worse presenting visual acuity, contrast sensitivity, and visual fields were associated with fewer activity counts, less active time, and more fragmented activity patterns (p < .05 for all). Participants with presenting or best-corrected visual acuity impairment had 19.2 and 29.3 fewer active minutes (p = .05 and p = .03, respectively) per day. Visual field impairment was associated with 268 636 fewer activity counts (p = .02), 46.2 fewer active minutes (p = .02) per day, and 3% greater activity fragmentation (p = .009). Differences in activity levels tended to be greatest from 6 am to 6 pm (p < .05). Conclusions Older adults with visual impairment have restricted and more fragmented patterns of daily activity. Longitudinal studies to quantify the long-term impacts of visual impairments on activity decline are warranted.