Background: Extra-virgin olive oil (EVOO) is the main source of seasoning fat in the Mediterranean diet and it is one of the components with known protective factors on chronic-degenerative disease. We aimed to evaluate the effect of a medium-high level of oil consumption on mortality in a cohort with good adherence to the Mediterranean diet. Methods: A total of 2754 subjects who had completed the food questionnaire in the Multicenter Italian study on Cholelithiasis (MICOL) cohort were included in the study. EVOO consumption was categorized in four levels (<20 g/die, 21–30 g/die, 31–40 g/die, >40 g/die). We performed a flexible parametric survival model to assess mortality by EVOO consumption level adjusted for some covariates. We also performed the analysis on subjects with and without non-alcoholic fatty liver disease (NAFLD) to evaluate the effects of oil in this more fragile sub-cohort. Results: We found a statistically significant negative effect on mortality for the whole sample when EVOO consumption was used, both as a continuous variable and when categorized. The protective effect was stronger in the sub-cohort with NAFLD, especially for the highest levels of EVOO consumption (HR = 0.58 with p < 0.05). Conclusions: Our study has shown a protective effect of EVOO consumption towards all causes of mortality. Despite the higher caloric intake, the protective power is greater for a consumption >40 g/day in both the overall cohort and the sub-cohorts with and without NAFLD.