Significance The simultaneous flow of multiple fluid phases through a porous solid occurs in many natural and industrial processes. Microscale physical mechanisms such as the relative affinity of the solid for the fluids (i.e., wettability), capillarity, and viscosity combine with pore geometry to produce a wide variety of macroscopic flow patterns. Pore-scale modeling is an essential tool to connect microscale mechanisms with macroscopic patterns, but quantitative comparisons between different models, and with experimental data, are lacking. Here, we perform an unprecedented comparison of state-of-the-art models from 14 leading groups with a recent experimental dataset. The results underscore the challenges of simulating multiphase flows through porous media, highlighting specific areas for further effort in what is already a flourishing field of research.