The reported incidence of immediate hypersensitivity reactions (IHR) including anaphylaxis after COVID-19 vaccination is 10-fold higher than for other vaccines. Several patient groups are theorized to be at particular risk. Since specific vaccination guidelines for these patients are based on expert opinion, we performed a retrospective monocentric analysis of the tolerability of adenoviral vector and mRNA-based COVID-19 vaccines in a cohort of patients allegedly at high risk of IHR. Reactions were assessed immediately on-site by allergists during a monitored vaccination protocol and after 3–7 days through telephone interviews. The cohort included 196 patients (aged 12–84 years) with primary mast cell disease (pMCD, 50.5%), idiopathic anaphylaxis (IA, 19.9%), hereditary angioedema (HAE, 5.1%) or miscellaneous indications (24.5%). Twenty-five immediate reactions were observed in 221 vaccine doses (11.3%). Most occurred in IA or miscellaneous patients. None fulfilled anaphylaxis criteria and most were mild and self-limiting. Reaction occurrence was significantly associated with female sex. In total, 13.5% of pMCD patients reported mast cell activation-like symptoms within 72 h post-vaccination. All pediatric pMCD patients (n = 9, 12–18 years) tolerated both mRNA-based vaccine doses. In summary, adenoviral vector and mRNA-based COVID-19 vaccines were safe and well-tolerated in patients with pMCD, HAE, and IA. No anaphylaxis was observed. The mild and subjective nature of most reactions suggests a nocebo effect associated with vaccination in a medicalized setting. Patients with pMCD could experience mild flare-ups of mast cell activation-like symptoms, supporting antihistamine premedication.