Cross-sectional results have suggested a linear association between the number of dietary exclusions and depressive symptoms. This longitudinal study aimed to examine the direction of this association. Methods: In the population-based Constances cohort, depressive symptoms were defined by a score ≥19 on the Centre of Epidemiologic Studies-Depression (CES-D) scale. Diet was measured with a 24-item qualitative food frequency questionnaire (FFQ). Both variables were available at inclusion (from 2012 to 2014) and on follow-up (2015 for the CES-D and 2017 for diet). Food exclusion was categorized into five different groups: No exclusion, exclusion of one, two, three, or ≥4 food groups according to the self-reported number of food groups rarely or never consumed. Logistic regressions were conducted, either taking depressive symptoms as the outcome on follow-up with dietary exclusions at baseline as predictor or with the opposite, adjusting for age, sex, education, income, alcohol intake, smoking, physical activity, and anemia. The path analysis included outcomes and covariates in one model. Results: The median follow-up was three years. A total of 29,337 participants (53.4% women, 48.15 ± 12.9 y.o.) had complete CES-D data and 25,356 (53.56% women, 49.05 ± 12.8 y.o.) FFQ data. Dietary exclusion at inclusion predicted depressive symptoms at follow-up (odds ratio [95% confidence interval]: 2.35 [1.62–3.40] for ≥4 excluded items compared to no exclusions). Depressive symptoms at inclusion predicted dietary exclusions at follow-up (3.45 [1.93–6.16] for ≥4 excluded items). In the path analysis, the standardized estimate of the association between dietary exclusions at inclusion and depressive symptoms at follow-up was by far higher than the opposite (0.1863 and 0.00189, respectively, both p < 0.05). Conclusions: The association of dietary exclusion with subsequent depression is stronger than the opposite association.