Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) continues to be a global problem in part because of the emergence of variants of concern that evade neutralization by antibodies elicited by prior infection or vaccination. Here we report on human neutralizing antibody and memory responses to the Gamma variant in a cohort of hospitalized individuals. Plasma from infected individuals potently neutralized viruses pseudotyped with Gamma SARS-CoV-2 spike protein, but neutralizing activity against Wuhan-Hu-1-1, Beta, Delta, or Omicron was significantly lower. Monoclonal antibodies from memory B cells also neutralized Gamma and Beta pseudoviruses more effectively than Wuhan-Hu-1. 69% and 34% of Gamma-neutralizing antibodies failed to neutralize Delta or Wuhan-Hu-1. Although Class 1 and 2 antibodies dominate the response to Wuhan-Hu-1 or Beta, 54% of antibodies elicited by Gamma infection recognized Class 3 epitopes. The results have implications for variant-specific vaccines and infections, suggesting that exposure to variants generally provides more limited protection to other variants.