Abstract The origins of most stellar streams in the Milky Way are unknown. With improved proper motions provided by Gaia EDR3, we show that the orbits of 23 Galactic stellar streams are highly clustered in orbital phase space. Based on their energies and angular momenta, most streams in our sample can plausibly be associated with a specific (disrupted) dwarf galaxy host that brought them into the Milky Way. For eight streams we also identify likely globular cluster progenitors (four of these associations are reported here for the first time). Some of these stream progenitors are surprisingly far apart, displaced from their tidal debris by a few to tens of degrees. We identify stellar streams that appear spatially distinct, but whose similar orbits indicate they likely originate from the same progenitor. If confirmed as physical discontinuities, they will provide strong constraints on the mass loss from the progenitor. The nearly universal ex situ origin of existing stellar streams makes them valuable tracers of galaxy mergers and dynamical friction within the Galactic halo. Their phase-space clustering can be leveraged to construct a precise global map of dark matter in the Milky Way, while their internal structure may hold clues to the small-scale structure of dark matter in their original host galaxies.