Despite the large body of research devoted to understanding the role of Quaternary glacial cycles in the genetic divergence of European trees, the differential contribution of geographic isolation and/or environmental adaptation in creating population genetic divergence remains unexplored. In this study, we used a long-lived tree (Taxus baccata) as a model species to investigate the impact of Quaternary climatic changes on genetic diversity via neutral (isolation-by-distance) and selective (isolation-by-adaptation) processes. We applied approximate Bayesian computation to genetic data to infer its demographic history, and combined this information with past and present climatic data to assess the role of environment and geography in the observed patterns of genetic structure. We found evidence that yew colonized Europe from the East, and that European samples diverged into two groups (Western, Eastern) at the beginning of the Quaternary glaciations, c. 2.2 Myr before present. Apart from the expected effects of geographical isolation during glacials, we discovered a significant role of environmental adaptation during interglacials at the origin of genetic divergence between both groups. This process may be common in other organisms, providing new research lines to explore the effect of Quaternary climatic factors on present-day patterns of genetic diversity.