Adiabatic quantum pumping in noninteracting, phase coherent quantum dots is elegantly described by Brouwer's formula. Interactions within the dot, while suppressing phase coherence, make Brouwer's formalism inapplicable. In this Letter, we discuss the nature of the physical processes forcing a description of pumping beyond Brouwer's formula, and develop, using a controlled adiabatic expansion, a useful formalism to study the effect of interactions within a generic perturbative scheme. The pumped current consists of a first contribution, analogous to Brouwer's formula and accounting for the remanent coherence, and of interaction corrections describing inelastic scattering. We apply the formalism to study the effect of interaction with a bosonic bath on a resonant level pump and discuss the robustness of the quantization of the pumped charge in turnstile cycles.