COVID-19 vaccine hesitancy (VH) has caused concerns due to the possible fluctuations that may occur directly impacting the control of the pandemic. In this study, we aimed to estimate the prevalence and factors associated with COVID-19 VH in Portuguese-speaking countries. We developed a web survey (N:6,843) using an online, structured, and validated questionnaire. We used Measurement Models, Exploratory Factor Analysis, Exploratory Structural Equation Models, and Confirmatory Factor Analysis for the data analysis. The overall prevalence of COVID-19 VH in Portuguese-speaking countries was 21.1%. showed a statistically significant direct effect for VH: vaccine-related conspiracy beliefs (VB) (β = 0.886), perceived stress (PS) (β = 0.313), COVID-19 Misinformation (MIS) (β = 0.259) and individual responses to COVID-19 (CIR) (β = −0.122). The effect of MIS and CIR for VH was greater among men and of PS and VB among women; the effect of PS was greater among the youngest and of VB and CIR among the oldest. No discrepant differences were identified in the analyzed education strata. In conclusion, we found that conspiracy beliefs related to the vaccine strongly influence the decision to hesitate (not to take or to delay the vaccine). Specific characteristics related to gender, age group, social and cognitive vulnerabilities, added to the knowledge acquired, poorly substantiated and/or misrepresented about the COVID-19 vaccine, need to be considered in the planning of vaccination campaigns. It is necessary to respond in a timely, fast, and accurate manner to the challenges posed by vaccine hesitancy.