The need to retain individuals longer in the workforce is acknowledged in many high-income countries. The present study therefore aimed to examine the importance of physically demanding work tasks (PDWT) and physically hazardous work environment (PHWE) in relation to retirement timing among pensionable workers (≥61 years). A particular question was whether PDWT and PHWE increased in importance with age. Six waves (2008–2018) of the Swedish Longitudinal Occupational Survey of Health (SLOSH) were used (n = 5201; 56% women and 44% men; mean age at first survey was 61.0 (SD 2.0) years). Discrete time-event history analysis, stratified by socioeconomic position and gender, showed that among blue-collar workers, PDWT and PHWE were associated with an increased likelihood of retiring within the next two years. With increasing age, high-level PHWE was associated with higher probability of retiring among blue-collar men, whereas heavy PDWT was associated with lower probability of retiring among blue-collar women. Among white-collar workers, having at least some PDWT compared to no PDWT was associated with a lower likelihood of retiring within the next two years. With increasing age, exposure to PHWE was associated with higher probability of retiring among white-collar women. These results suggest that to delay retirements, organizations could offer their older employees, especially blue-collar workers and the oldest white-collar women, alternatives to PDWT and PHWE.