The identification of early or primary resistance to androgen signaling inhibitors (ASIs) is of great value for the treatment of metastatic castration-resistant prostate cancer (mCRPC). This study evaluates the predictive value of prostate-specific antigen (PSA) response at dour weeks of first-line ASIs treatment for mCRPC patients. A total of 254 patients treated with ASIs (abiraterone acetate: AA and enzalutamide: Enz) at the first-line treatment are retrospectively analyzed. Patients are stratified according to the achievement of >30% PSA decline at 4 and 12 weeks from the treatment initiation. At four weeks of the treatment, 157 patients (61.8%) achieved >30% PSA decline from the baseline. Thereafter, 177 patients (69.7%) achieved >30% PSA decline at 12 weeks of the treatment. A multivariate analysis exhibits >30% PSA decline at four weeks as an independent predictor for overall survival (OS). We note that 30 of 97 (30.9%) patients who did not achieve >30% PSA decline at four weeks consequently achieved >30% PSA decline at 12 weeks, and had a comparable favorable three years OS rate as the 147 patients achieving >30% PSA decline at both 4 and 12 weeks. To identify the variables that discriminate the patient survival in 97 patients without achieving >30% PSA decline at four weeks, a multivariate analysis is performed. The duration of androgen deprivation therapy before CRPC ≤ 12 months and Eastern Cooperative Oncology Group Performance Status ≥ 1 are identified as independent predictors for shorter OS for those patients. These data offer a concept of early treatment switch after four weeks of first-line ASIs when not observing >30% PSA decline at four weeks—particularly in patients with a modest effect of ADT and poor performance status.