A comprehensive understanding of the kinetics and evolution of the human B cell response to SARS-CoV-2 infection will facilitate the development of next-generation vaccines and therapies. Here, we longitudinally profiled this response in mild and severe COVID-19 patients over a period of five months. Serum neutralizing antibody (nAb) responses waned rapidly but spike (S)-specific IgG ⁺ memory B cells (MBCs) remained stable or increased over time. Analysis of 1,213 monoclonal antibodies (mAbs) isolated from S-specific MBCs revealed a primarily de novo response that displayed increased somatic hypermutation, binding affinity, and neutralization potency over time, providing evidence for prolonged antibody affinity maturation. B cell immunodominance hierarchies were similar across donor repertoires and remained relatively stable as the immune response progressed. Cross-reactive B cell populations, likely re-called from prior endemic beta-coronavirus exposures, comprised a small but stable fraction of the repertoires and did not contribute to the neutralizing response. The neutralizing antibody response was dominated by public clonotypes that displayed significantly reduced activity against SARS-CoV-2 variants emerging in Brazil and South Africa that harbor mutations at positions 501, 484 and 417 in the S protein. Overall, the results provide insight into the dynamics, durability, and functional properties of the human B cell response to SARS-CoV-2 infection and have implications for the design of immunogens that preferentially stimulate protective B cell responses.