Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) Beta variant of concern (VOC) resists neutralization by major classes of antibodies from COVID-19 patients and vaccinated individuals. In this study, serum of Beta-infected patients revealed reduced cross-neutralization of wild-type virus. From these patients, we isolated Beta-specific and cross-reactive receptor-binding domain (RBD) antibodies. The Beta-specificity results from recruitment of VOC-specific clonotypes and accommodation of mutations present in Beta and Omicron into a major antibody class that is normally sensitive to these mutations. The Beta-elicited cross-reactive antibodies share genetic and structural features with wild type–elicited antibodies, including a public VH1-58 clonotype that targets the RBD ridge. These findings advance our understanding of the antibody response to SARS-CoV-2 shaped by antigenic drift, with implications for design of next-generation vaccines and therapeutics.