Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has evolved variants with substitutions in the spike receptor-binding domain (RBD) that affect its affinity for angiotensin-converting enzyme 2 (ACE2) receptor and recognition by antibodies. These substitutions could also shape future evolution by modulating the effects of mutations at other sites—a phenomenon called epistasis. To investigate this possibility, we performed deep mutational scans to measure the effects on ACE2 binding of all single–amino acid mutations in the Wuhan-Hu-1, Alpha, Beta, Delta, and Eta variant RBDs. Some substitutions, most prominently Asn ⁵⁰¹ →Tyr (N501Y), cause epistatic shifts in the effects of mutations at other sites. These epistatic shifts shape subsequent evolutionary change—for example, enabling many of the antibody-escape substitutions in the Omicron RBD. These epistatic shifts occur despite high conservation of the overall RBD structure. Our data shed light on RBD sequence-function relationships and facilitate interpretation of ongoing SARS-CoV-2 evolution.