Time spent out-of-home can increase physical activity (PA) levels. However, the association between the purpose for leaving home and the mode of transport on time out-of-home and PA are not straightforward. In a large sample of community-dwelling older adults (≥65 years), daily walking duration was measured prospectively over 1 week using body-fixed sensors and time-out-of-home questionnaires. Data from 1,277 participants yielded 6,500 full days for analysis. The following statistically significant associations were seen: public transport use increased the time spent out-of-home by 88 min and daily walking duration by 16 min. Social contacts or the use of a car increased the time out-of-home, but decreased PA. Shopping or “going by foot” decreased the time spent out-of-home by 19 and 62 min, respectively, while both increased the daily walking duration by 5 min. The association between time out-of-home and PA (daily walking duration) is strongly dependent on the activity and mode of transport.