Abstract Background Motor function affects ability to perform daily activities and maintain independence. Yet, the interrelatedness of upper and lower extremity motor impairments and the magnitude of their contribution to slow gait and mobility difficulty are not well investigated. Methods Participants in the Baltimore Longitudinal Study of Aging (N = 728, aged 50–99) completed motor and physical function tests including grip and knee extension strength, pegboard, finger tapping, standing balance, chair stands, fast-paced 400-m walk, and usual gait speed. Slow gait was defined as usual gait speed <1.0 m/s. Mobility difficulty was defined as self-reported difficulty walking ¼ mile or climbing stairs. Structural equation modeling examined the interrelationships of motor measures and their contributions to slow gait and mobility difficulty, adjusting for demographics and comorbidities. Results Poorer manual dexterity (−0.571 standard deviation [SD] units, p < .001) and lower muscle strength (upper and lower extremity; −0.447 SD units, p = .014) were most strongly associated with slow gait speed, followed by slower chair stand pace (−0.195 SD units, p = .002) and greater lap time variation (0.102 SD units, p = .028). Lower muscle strength (−0.582 SD units, p = .001) was most strongly associated with mobility difficulty, followed by slower chair stand pace (−0.322 SD units, p < .001), slower gait speed (−0.247 SD units, p < .001), and poorer standing balance (−0.190 SD units, p = .043). Conclusions Components of manual dexterity and strength were the strongest correlates of slow gait and mobility difficulty in mid-to-late life. Longitudinal studies examining relationships between changes in these motor parameters and mobility are needed to elucidate possible causal effects.