Normal brain aging is an inevitable and heterogeneous process characterized by a selective pattern of structural changes. Such heterogeneity arises as a consequence of cumulative effects over the lifespan, including stress and mood effects, which drive different micro- and macro-structural alterations in the brain. Investigating these differences in healthy age-related changes is a major challenge for the comprehension of the cognitive status. Herein we addressed the impact of normal aging, stress, mood, and their interplay in the brain gray and white matter (WM) structure. We showed the critical impact of age in the WM volume and how stress and mood influence brain volumetry across the lifespan. Moreover, we found a more profound effect of the interaction of aging/stress/mood on structures located in the left hemisphere. These findings help to clarify some divergent results associated with the aging decline and to enlighten the association between abnormal volumetric alterations and several states that may lead to psychiatric disorders.