Fatigue is a main symptom of chronic diseases, including immune-mediated inflammatory diseases (IMIDs), such as inflammatory bowel disease (IBD) and inflammatory arthritis (IA); however, the pathophysiological mechanisms are not completely understood. The aim of this study was to assess the prevalence of fatigue and the associated factors in an IMIDs population. A control group, IBD, and IA patients, were enrolled. The PROMIS® fatigue questionnaire was used to evaluate the symptoms. Information on demographics, anthropometrics, disease characteristics, and medications was collected for each participant. A total of 471 subjects (137 with IBD, 103 with IA, and 206 controls) were enrolled. IBD and IA patients reported greater fatigue than controls (p < 0.001, each). In univariate regression, patients with anxiety and depression were more likely to report fatigue (p = 1.40 × 10−9 and p = 3.80 × 10−11, respectively). Males, holding a high school diploma, and being employed were inversely correlated to the domain (p = 1.3 × 10−5; p = 0.003 and p = 0.005, respectively). The use of steroids and disease activity determined increased fatigue (p = 0.014 and p = 0.019; respectively). In the multivariate analysis, anxiety and depression remained associated (p = 0.002 and p = 1.3 × 10−5, respectively). IMIDs patients present increased fatigue compared with healthy subjects. Anxiety and depression are the main associated factors, suggesting a psychological component of the symptom; thus, a holistic management should be established.