Background: The relationship between midlife dietary habits and risk of dementia remains unclear. Objective: To investigate the association between dietary fish and n-3 polyunsaturated fatty acid (PUFA) consumption in midlife and risk of dementia in later life. Methods: This population-based cohort study assessed food frequency (average intake in 1995 and 2000) and cognition (2014-2015) in 1,127 participants (aged 45–64 in 1995). We used logistic regression analyses to calculate odds ratios (ORs) for dementia and mild cognitive impairment (MCI) diagnoses for consumption quartiles of fish, PUFA-rich fish, total n-3 PUFAs, total n-6 PUFAs, types of PUFAs, and n-3/n-6 PUFA ratio. Estimated ORs were adjusted for age; sex; education; smoking status; alcohol consumption frequency; physical activity; histories of cancer, myocardial infarction, and diabetes mellitus; and depression. Results: Significantly reduced risks of dementia over non-dementia (MCI plus cognitively normal) were observed in the second (OR = 0.43 (95% CI = 0.20–0.93)), third (OR = 0.22 (95% CI = 0.09–0.54)), and highest quartiles (OR = 0.39 (95% CI = 0.18–0.86)) for fish; the third (OR = 0.39 (95% CI = 0.16–0.92)) and highest quartiles (OR = 0.44 (95% CI = 0.19–0.98)) for eicosapentaenoic acid (EPA); the second (OR = 0.39 (95% CI = 0.18–0.84)), third (OR = 0.30 (95% CI = 0.13–0.70)), and highest quartiles (OR = 0.28 (95% CI = 0.12–0.66)) for docosahexaenoic acid (DHA); and the third (OR = 0.36 (95% CI = 0.16–0.85)) and highest quartiles (OR = 0.42 (95% CI = 0.19–0.95)) for docosapentaenoic acid (DPA). Conclusion: High intake of fish in midlife might aid in preventing dementia.