(1) Background: Caffeine is one of the most consumed psychoactive stimulants worldwide. It has been suggested that caffeine intake at large doses can induce anxiety, whereas evidence of the role of low to moderate caffeine intake is scarce and inconsistent. Therefore, we aimed to assess the association between caffeine intake and general anxiety in adults recruited from the general population. (2) Methods: Participants from the French NutriNet-Santé web cohort with data on caffeine intake and general anxiety (assessed during 2013–2016 through the trait subscale of Spielberger’s State-Trait Anxiety Inventory Form Y; STAI-T, sex-specific top quartile = high trait anxiety) were included in this cross-sectional analysis (n = 24,197; 74.1% women; mean age = 53.7 ± 13.9 years). Mean dietary intake was estimated using ≥2 self-reported 24-h dietary records. Sex-specific tertiles of caffeine intake and low/high trait anxiety were calculated. Multivariable logistic regression models were fitted to assess the odds ratio (OR) and 95% confidence interval (CI) for the association between caffeine intake and general anxiety by sex. (3) Results: In the total sample, the mean caffeine intake (mg/day) from all dietary sources combined was 220.6 ± 165.0 (women = 212.4 ± 159.6; men = 243.8 ± 177.7, p < 0.01). Women in the highest tertile of caffeine intake showed significantly higher odds for high trait anxiety compared to those in the lowest tertile (reference), even after adjustment for potential confounders (OR: 1.13; 95% CI: 1.03–1.23). No significant associations were detected among men. Sensitivity analyses according to perceived stress level and sugar intake, respectively, showed similar results. (4) Conclusions: The results suggest that higher caffeine intake is associated with higher odds of general anxiety among women but not among men. Further research is needed to confirm the sex-specific findings and elucidate the potential causal relationship between caffeine intake and anxiety status.