A large proportion of patients exhibit persistently reduced pulmonary diffusion capacity after COVID-19. It is unknown whether this is due to a post-COVID restrictive lung disease and/or pulmonary vascular disease. The aim of the current study was to investigate the association between initial COVID-19 severity and haemoglobin-corrected diffusion capacity to carbon monoxide (DLco) reduction at follow-up. Furthermore, to analyse if DLco reduction could be linked to pulmonary fibrosis (PF) and/or thromboembolic disease within the first months after the illness, a total of 67 patients diagnosed with COVID-19 from March to December 2020 were included across three severity groups: 12 not admitted to hospital (Group I), 40 admitted to hospital without intensive care unit (ICU) admission (Group II), and 15 admitted to hospital with ICU admission (Group III). At first follow-up, 5 months post SARS-CoV-2 positive testing/4 months after discharge, lung function testing, including DLco, high-resolution CT chest scan (HRCT) and ventilation-perfusion (VQ) single photon emission computed tomography (SPECT)/CT were conducted. DLco was reduced in 42% of the patients; the prevalence and extent depended on the clinical severity group and was typically observed as part of a restrictive pattern with reduced total lung capacity. Reduced DLco was associated with the extent of ground-glass opacification and signs of PF on HRCT, but not with mismatched perfusion defects on VQ SPECT/CT. The severity-dependent decline in DLco observed early after COVID-19 appears to be caused by restrictive and not pulmonary vascular disease.