Background: Differences exist regarding post-stroke cognitive outcomes. Objective: The aim of this study investigates the potential factors associated with post-stroke cognitive performance and trajectories. Methods: We performed a prospective cohort study using serial monitoring of cognitive function over a 1-year period after a first-ever ischemic stroke. Small vessel disease (SVD) burden and hippocampal atrophy (HA) were evaluated using the modified cerebral small vessel disease scores (mCSVD) and medial temporal atrophy score (MTA) scores. A generalized estimating equation (GEE) model and a group-based trajectory model (GBTM) was used to analyze the potential factors associated with post-stroke cognitive outcomes. Results: A total of 112 patients were enrolled. The GEE model showed that all patients, regardless of initial cognitive performance, had a tendency to show an increase in the Montreal Cognitive Assessment over time. The cognitive performance was better in male patients with higher education levels (p = 0.046 and p < 0.001, respectively), but tended to be worse in patients with higher SVD burden and HA. The GBTM model grouped patients into low, intermediate, and high performance (LP, IP, and HP) after stroke. A higher SVD burden, rather than HA and initial stroke severity and location, independently predicted a higher odds of poor post-stroke cognitive trajectory (being in the LP group) after stroke (adjusted odds ratio 2.74, 95%CI 1.09–6.86). Conclusion: In patients with first-ever mild stroke, cognitive improvement over time was evident. The detrimental impact of the SVD burden may outweigh the effect of HA or acute stroke insult on the post-stroke cognitive trajectory during the 1-year follow-up.