Abstract Purpose Persistent fatigue among colorectal cancer (CRC) patients might be associated with unfavorable body composition, but data are sparse and inconsistent. We studied how skeletal muscle index (SMI), skeletal muscle radiodensity (SMR), visceral adipose tissue (VAT), and subcutaneous adipose tissue (SAT) at diagnosis are associated with fatigue up to 24 months post-diagnosis in stage I–III CRC patients. Methods SMI, SMR, VAT, and SAT were assessed among 646 CRC patients using pre-treatment computed tomography images. Fatigue at diagnosis, at 6, and 24 months post-diagnosis was assessed using the European Organization for Research and Treatment of Cancer Quality of Life Questionnaire. The association of SMI, SMR, VAT, and SAT with fatigue (yes/no) was assessed using confounder-adjusted restricted cubic spline analyses. Results Prevalence of fatigue at diagnosis was 18%, at 6 months 25%, and at 24 months 12%. At diagnosis, a significant (p = 0.01) non-linear association of higher levels of SAT with higher prevalence of fatigue was observed. Lower levels of SMR were linearly associated with higher prevalence of fatigue at 6 months post-diagnosis (overall association p = 0.02). None of the body composition parameters were significantly associated with fatigue at 24 months. Conclusion Having more SAT was associated with more fatigue at diagnosis, while low levels of SMR were associated with more fatigue at 6 months post-diagnosis. Implications for Cancer Survivors Our results suggest that it may be interesting to investigate whether interventions that aim to increase SMR around the time of diagnosis may help to lower fatigue. However, more knowledge is needed to understand the mechanisms behind the association of SMR with fatigue.