Abstract Purpose Amygdalar metabolic activity was shown to independently predict cardiovascular outcomes. However, little is known about age- and sex-dependent variability in neuronal stress responses among individuals free of cardiac disease. This study sought to assess age- and sex-specific differences of resting amygdalar metabolic activity in the absence of clinical cardiovascular disease. Methods Amygdalar metabolic activity was assessed in 563 patients who underwent multimodality imaging by ¹⁸F-fluorodeoxyglucose (¹⁸F-FDG) positron emission tomography/computed tomography and echocardiography for the evaluation of cardiac function. Results After exclusion of 294 patients with structural or functional cardiovascular pathologies, 269 patients (128 women) remained in the final population. ¹⁸F-FDG amygdalar activity significantly decreased with age in men (r = − 0.278, P = 0.001), but not in women (r = 0.002, P = 0.983). Similarly, dichotomous analysis confirmed a lower amygdalar activity in men ≥ 50 years as compared to those < 50 years of age (0.79 ± 0.1 vs. 0.84 ± 0.1, P = 0.007), which was not observed in women (0.81 ± 0.1 vs. 0.82 ± 0.1, P = 0.549). Accordingly, a fully adjusted linear regression analysis identified age as an independent predictor of amygdalar activity only in men (B-coefficient − 0.278, P = 0.001). Conclusion Amygdalar activity decreases with age in men, but not in women. The use of amygdalar activity for cardiovascular risk stratification merits consideration of inherent age- and sex-dependent variability.