(1) Background: the aim of this study was to create a score to predict the incidence of CPAP failure in COVID-19 patients early. (2) Methods: in this retrospective observational study, we included all consecutive adult patients admitted between February and April 2021. The main outcome was the failure of CPAP support (intubation or death). (3) Results: two-hundred and sixty-three COVID-19 patients were managed with CPAP. The population was divided in short-CPAP (CPAP days ≤ 10; 72.6%) and long-CPAP (>10; 27.4%) groups. After balancing the entire population using a stabilized IPTW method, we applied a multivariable logistic regression analysis to identify the risk factors for CPAP failure. We used the identified covariates to create a mathematical model, the CPAP Failure Score (CPAP-FS). The multivariable logistic regression analysis identified four variables: SpO2 (OR = 0.86; p-value = 0.001), P/F ratio (OR = 0.99; p-value = 0.008), the Call Score (OR = 1.44; p-value = 0.02), and a pre-existing chronic lung disease (OR = 3.08; p-value = 0.057). The beta-coefficients obtained were used to develop the CPAP-FS, whose diagnostic ability outperformed other relevant COVID-19-related parameters (AUC = 0.87; p-value < 0.0001). We validated the CPAP-FS using a 10-fold internal cross-validation method which confirmed the observed results (AUCs 0.76–0.80; p-values < 0.0001). (4) Conclusions: the CPAP-FS can early identify COVID-19 patients who are at risk of CPAP failure.