Food webs are the complex networks of trophic interactions that stoke the metabolic fires of life. To understand what structures these interactions in natural communities, ecologists have developed simple models to capture their main architectural features. However, apparently realistic food webs can be generated by models invoking either predator–prey body-size hierarchies or evolutionary constraints as structuring mechanisms. As a result, this approach has not conclusively revealed which factors are the most important. Here we cut to the heart of this debate by directly comparing the influence of phylogeny and body size on food web architecture. Using data from 13 food webs compiled by direct observation, we confirm the importance of both factors. Nevertheless, phylogeny dominates in most networks. Moreover, path analysis reveals that the size-independent direct effect of phylogeny on trophic structure typically outweighs the indirect effect that could be captured by considering body size alone. Furthermore, the phylogenetic signal is asymmetric: closely related species overlap in their set of consumers far more than in their set of resources. This is at odds with several food web models, which take only the view-point of consumers when assigning interactions. The echo of evolutionary history clearly resonates through current food webs, with implications for our theoretical models and conservation priorities.