ObjectiveTo determine which pathologic process could be responsible for the acceleration of cognitive decline during the course of multiple sclerosis (MS), using longitudinal structural MRI, which was related to cognitive decline in relapsing-remitting MS (RRMS) and progressive MS (PMS).MethodsA prospective cohort of 230 patients with MS (179 RRMS and 51 PMS) and 59 healthy controls was evaluated twice with 5-year (mean 4.9, SD 0.94) interval during which 22 patients with RRMS converted to PMS. Annual rates of cortical and deep gray matter atrophy as well as lesion volume increase were computed on longitudinal (3T) MRI data and correlated to the annual rate of cognitive decline as measured using an extensive cognitive evaluation at both time points.ResultsThe deep gray matter atrophy rate did not differ between PMS and RRMS (−0.82%/year vs −0.71%/year, p = 0.11), while faster cortical atrophy was observed in PMS (−0.87%/year vs −0.48%/year, p < 0.01). Similarly, faster cognitive decline was observed in PMS compared to RRMS (p < 0.01). Annual cognitive decline was related to the rate of annual lesion volume increase in stable RRMS (r = −0.17, p = 0.03) to the rate of annual deep gray matter atrophy in converting RRMS (r = 0.50, p = 0.02) and annual cortical atrophy in PMS (r = 0.35, p = 0.01).ConclusionsThese results indicate that cortical atrophy and cognitive decline accelerate together during the course of MS. Substrates of cognitive decline shifted from worsening lesional pathology in stable RRMS to deep gray matter atrophy in converting RRMS and to accelerated cortical atrophy in PMS only.