The Mediterranean dietary pattern (MPD) has shown promise in preventing low-grade systemic inflammation (LGSI). This study tested if a high adherence to the MDP by younger and older Brazilian adults is associated with lower LGSI and investigated which Mediterranean food components may contribute to these associations. We performed a secondary study on 2015 ISA-Nutrition (290 younger adults (20–59 years old) and 293 older adults (≥60 years old)), a cross-sectional population-based study in São Paulo, SP, Brazil. The adherence to the MDP was assessed using the Mediterranean Diet Score (MedDietScore), obtained from two non-consecutive 24 h dietary recalls (24HDRs). The LGSI score (from plasma CRP, TNF-α, and adiponectin) identified the inflammatory status. Linear regression models assessed the association between LGSI and the MedDietScore. In older adults only, a high adherence to the MDP signified an 11.5% lower LGSI score. Older adults, classified with high adherence to the MDP, differed by consuming lower meat intake and full-fat dairy. Between older adults, the intake of vegetables and olive oil was inversely associated with the levels of LGSI. Thus, among older adults, the intake of some specific Mediterranean food determined high adherence to the MDP and was associated with decreased LGSI.