Abstract Background Telehealth delivery expanded quickly during the COVID-19 pandemic after the reduction of payment and regulatory barriers, but older adults are the least likely to benefit from this expansion. Little is known about physician experiences initiating telehealth and factors that fostered or discouraged adoption during the COVID-19 pandemic with older adult patients. Therefore, our objective was to understand experiences of frontline physicians caring for older adults via telehealth during the COVID-19 pandemic. Methods We conducted semi-structured interviews from September 2020 to November 2020 with 48 physicians. We recruited a diverse sample of geriatricians (n = 18), primary care (n = 15), and emergency (n = 15) physicians from all United Stated (US) regions, rural-urban settings, and academic-community practices who cared for older adult patients during the pandemic using purposive sampling methods. We completed framework analysis of the transcribed interviews to identify emerging themes and used the Quadruple Aim to organize themes. Results Frontline physicians described telehealth as a more flexible, value-based, and patient-centered mode of health care delivery. Benefits of using telehealth to treat older adults included reducing deferred care and increasing timely care, improving efficiency for physicians, enhancing communication with caregivers and patients, reducing patient travel burdens, and facilitating health outreach and education. Challenges included unequal access for rural, older, or cognitively impaired patients. Physicians noted that payment parity with in-person visits, between video and telephone visits, and relaxation of restrictive regulations would enhance their ability to continue to offer telehealth. Conclusions Frontline physicians who treated older adults during the COVID-19 pandemic were largely in favor of continuing telehealth use beyond the pandemic; however, they noted that sustainability would depend on enacting policies that address access inequities and reimbursement concerns. Our data provide policy insights that if placed into action could facilitate the long-term success of telehealth and encourage a more flexible healthcare delivery system in the US.