Background Telehealth has become widely used as a novel way to provide outpatient care during the COVID-19 pandemic, but data about telehealth use in primary care remain limited. Studies in other specialties raise concerns that telehealth may be widening existing health care disparities, requiring further scrutiny of trends in telehealth use. Objective Our study aims to further characterize sociodemographic differences in primary care via telehealth compared to in-person office visits before and during the COVID-19 pandemic and determine if these disparities changed throughout 2020. Methods We conducted a retrospective cohort study in a large US academic center with 46 primary care practices from April-December 2019 to April-December 2020. Data were subdivided into calendar quarters and compared to determine evolving disparities throughout the year. We queried and compared billed outpatient encounters in General Internal Medicine and Family Medicine via binary logic mixed effects regression model and estimated odds ratios (ORs) with 95% CIs. We used sex, race, and ethnicity of the patient attending each encounter as fixed effects. We analyzed socioeconomic status of patients in the institution’s primary county based on the patient’s residence zip code. Results A total of 81,822 encounters in the pre–COVID-19 time frame and 47,994 encounters in the intra–COVID-19 time frame were analyzed; in the intra–COVID-19 time frame, a total of 5322 (11.1%) of encounters were telehealth encounters. Patients living in zip code areas with high utilization rate of supplemental nutrition assistance were less likely to use primary care in the intra–COVID-19 time frame (OR 0.94, 95% CI 0.90-0.98; P=.006). Encounters with the following patients were less likely to be via telehealth compared to in-person office visits: patients who self-identified as Asian (OR 0.74, 95% CI 0.63-0.86) and Nepali (OR 0.37, 95% CI 0.19-0.72), patients insured by Medicare (OR 0.77, 95% CI 0.68-0.88), and patients living in zip code areas with high utilization rate of supplemental nutrition assistance (OR 0.84, 95% CI 0.71-0.99). Many of these disparities persisted throughout the year. Although there was no statistically significant difference in telehealth use for patients insured by Medicaid throughout the whole year, subanalysis of quarter 4 found encounters with patients insured by Medicaid were less likely to be via telehealth (OR 0.73, 95% CI 0.55-0.97; P=.03). Conclusions Telehealth was not used equally by all patients within primary care throughout the first year of the COVID-19 pandemic, specifically by patients who self-identified as Asian and Nepali, insured by Medicare, and living in zip code areas with low socioeconomic status. As the COVID-19 pandemic and telehealth infrastructure change, it is critical we continue to reassess the use of telehealth. Institutions should continue to monitor disparities in telehealth access and advocate for policy changes that may improve equity.