Residents in residential care homes for the elderly (RCHEs) are at high risk of severe illnesses and mortality, while staff have high exposure to intimate care activities. Addressing vaccine hesitancy is crucial to safeguard vaccine uptake in this vulnerable setting, especially amid a pandemic. In response to this, we conducted a cross-sectional survey to measure the level of vaccine hesitancy and to examine its associated factors among residents and staff in RCHEs in Hong Kong. We recruited residents and staff from 31 RCHEs in July–November 2022. Of 204 residents, 9.8% had a higher level of vaccine hesitancy (scored ≥ 4 out of 7, mean = 2.44). Around 7% of the staff (n = 168) showed higher vaccine hesitancy (mean = 2.45). From multi-level regression analyses, higher social loneliness, higher anxiety, poorer cognitive ability, being vaccinated with fewer doses, and lower institutional vaccination rates predicted residents’ vaccine hesitancy. Similarly, higher emotional loneliness, higher anxiety, being vaccinated with fewer doses, and working in larger RCHEs predicted staff’s vaccine hesitancy. Although the reliance on self-report data and convenience sampling may hamper the generalizability of the results, this study highlighted the importance of addressing the loneliness of residents and staff in RCHEs to combat vaccine hesitancy. Innovative and technology-aided interventions are needed to build social support and ensure social interactions among the residents and staff, especially amid outbreaks.