Background: To assess the association between anxiety and risk of vascular dementia (VaD), as well as potential sex differences, in a community-based cohort. Methods: A random sample of 4057 dementia-free community participants aged 55 or older, from the longitudinal, community-based Zaragoza Dementia and Depression Project (ZARADEMP) study were followed for 4.5 years. Geriatric Mental State B (GMS)-Automated Geriatric Examination for Computer Assisted Taxonomy (AGECAT) was used for the assessment and diagnosis of anxiety, and a panel of research psychiatrists diagnosed the incident cases of VaD according to DSM-IV (Diagnostic and Statistical Manual of mental disordes). Multivariate survival analysis with competing risk regression model was performed. Results: In men, the incidence rate of VaD was significantly higher among anxiety subjects compared with non-anxiety subjects (incidence rate ratio (IRR) (95% confidence interval (CI)): 3.24 (1.13–9.35); p = 0.029), and no difference was observed in women (IRR (95%CI): 0.68 (0.19–2.23); p = 0.168). In the multivariate model, for men, cases of anxiety had 2.6-fold higher risk of VaD (subdistribution hazard ratio (SHR): 2.61; 95%CI: 0.88–7.74) when all potential confounding factors were controlled, with no statistical significance (p = 0.084), but a clinically relevant effect (Cohen’s d: 0.74). No association was found in women. Conclusions: In men, but not in women, risk of VaD was higher among individuals with anxiety, with a clinically relevant effect. Potential anxiety-related preventive interventions for VaD might be tailored to men and women separately.