Testosterone plays a significant role in maintaining the tumor microenvironment. The role of the target serum testosterone (TST) level in enzalutamide- (Enza) and abiraterone (Abi)-treated castration-resistant prostate cancer (CRPC) patients was studied. In total, 107 patients treated with Enza and/or Abi at Chiba University Hospital and affiliated hospitals were studied. The relationships between progression-free survival (PFS), overall survival (OS), and clinical factors were studied by Cox proportional hazard and Kaplan–Meier models. In the Abi and Enza groups overall, TST ≥ 13 ng/dL (median) (Hazard Ratio (HR) 0.43, p = 0.0032) remained an independent prognostic factor for PFS. In the Enza group, TST ≥ 13 ng/dL (median) was found to be a significant prognostic factor (HR 0.28, p = 0.0044), while, in the Abi group, TST ≥ 12 ng/dL (median) was not significant (HR 0.40, p = 0.0891). TST showed significant correlation with PFS periods (r = 0. 32, p = 0.0067), whereas, for OS, TST ≥ 13 ng/dL (median) showed no significant difference in the Abi and Enza groups overall. According to Kaplan–Meier analysis, a longer PFS at first-line therapy showed a favorable prognosis in the Enza group (p = 0.0429), while no difference was observed in the Abi group (p = 0.6051). The TST level and PFS of first-line therapy may be considered when determining the treatment strategy for CRPC patients.