Abstract Background Cognitive complaints are common in patients recovering from Coronavirus Disease 2019 (COVID-19), yet their etiology is often unclear. We assess factors that contribute to cognitive impairment in ambulatory versus hospitalized patients during the sub-acute stage of recovery. Methods In this cross-sectional study, participants were prospectively recruited from a hospital-wide registry. All patients tested positive for SARS-CoV-2 infection using a real-time reverse transcriptase polymerase-chain-reaction assay. Patients ≤ 18 years-of-age and those with a pre-existing major neurocognitive disorder were excluded. Participants completed an extensive neuropsychological questionnaire and a computerized cognitive screen via remote telemedicine platform. Rates of subjective and objective neuropsychological impairment were compared between the ambulatory and hospitalized groups. Factors associated with impairment were explored separately within each group. Results A total of 102 patients (76 ambulatory, 26 hospitalized) completed the symptom inventory and neurocognitive tests 24 ± 22 days following laboratory confirmation of SARS-CoV-2 infection. Hospitalized and ambulatory patients self-reported high rates of cognitive impairment (27–40%), without differences between the groups. However, hospitalized patients showed higher rates of objective impairment in visual memory (30% vs. 4%; p = 0.001) and psychomotor speed (41% vs. 15%; p = 0.008). Objective cognitive test performance was associated with anxiety, depression, fatigue, and pain in the ambulatory but not the hospitalized group. Conclusions Focal cognitive deficits are more common in hospitalized than ambulatory patients. Cognitive performance is associated with neuropsychiatric symptoms in ambulatory but not hospitalized patients. Objective neurocognitive measures can provide essential information to inform neurologic triage and should be included as endpoints in clinical trials.