Androgen deprivation therapy (ADT) is a cornerstone of treatment for prostate cancer and, in recent years, androgen receptor (AR)-targeted therapies (abiraterone and enzalutamide) have both been used for the treatment of castration-resistant prostate cancer (CRPC). In our study, we sought to investigate the association between ADT and immune disorders, considering a potential role of androgens in the immune modulation. We retrospectively evaluated CRPC patients treated with abiraterone/enzalutamide between July 2011 and December 2018. We assessed the risk of developing immune alterations and their impact on outcome. We included 844 CRPC patients receiving AR-directed therapies, of whom 36 (4.3%) had autoimmune diseases and 47 (5.6%) second tumors as comorbidities. Median age was 70 years [interquartile range (IQR) = 63–75)]. We showed higher significant incidence of autoimmune diseases during their hormone sensitive status (p = 0.021) and the presence of autoimmune comorbidities before starting treatment with abiraterone/enzalutamide was significantly associated with worse overall survival (OS) (10.1 vs. 13.7 months, HR = 1.59, 95% CI 1.03–2.27, p = 0.038). In a multivariate analysis, the presence of autoimmune disorders was an independent predictor of OS (HR = 1.65, 95% CI 1.05–2.60, p = 0.031). In conclusion, CRPC patients with autoimmune alterations before starting AR-directed therapies may have worse prognosis. Further prospective studies are warranted to assess the role of immune modulation in the management of prostate cancer patients.