Extra virgin olive oil (EVOO) has been reported to have a distinct influence on gut microbiota in comparison to other fats, with its physiological benefits widely studied. However, a large proportion of the population consumes olive oil after a depurative process that not only mellows its taste, but also deprives it of polyphenols and other minority components. In this study, we compare the influence on the intestinal microbiota of a diet high in this refined olive oil (ROO) with other fat-enriched diets. Swiss Webster mice were fed standard or a high-fat diet enriched with EVOO, ROO, or butter (BT). Physiological parameters were also evaluated. At the end of the feeding period, DNA was extracted from feces and the 16S rRNA was pyrosequenced. The group fed ROO behaved differently to the EVOO group in half the families with statistically significant differences among the diets, with higher comparative levels in three families—Desulfovibrionaceae, Spiroplasmataceae, and Helicobacteraceae—correlating with total cholesterol. These results are again indicative of a link between specific diets, certain physiological parameters and the prevalence of some taxa, but also support the possibility that polyphenols and minor components of EVOO are involved in some of the proposed effects of this fat through the modulation of the intestinal microbiota