We investigate the evolution of interacting Rydberg gases in the limit of strong noise and dissipation. Starting from a description in terms of a Markovian quantum master equation we derive effective equations of motion that govern the dynamics on a "coarse-grained" timescale where fast dissipative degrees of freedom have been adiabatically eliminated. Specifically, we consider two scenarios which are of relevance for current theoretical and experimental studies --- Rydberg atoms in a two-level (spin) approximation subject to strong dephasing noise as well as Rydberg atoms under so-called electromagnetically induced transparency (EIT) conditions and fast radiative decay. In the former case we find that the effective dynamics is described by classical rate equations up to second order in an appropriate perturbative expansion. This drastically reduces the computational complexity of numerical simulations in comparison to the full quantum master equation. When accounting for the fourth order correction in this expansion, however, we find that the resulting equation breaks the preservation of positivity and thus cannot be interpreted as a proper classical master rate equation. In the EIT system we find that the expansion up to second order retains information not only on the "classical" observables, but also on some quantum coherences. Nevertheless, this perturbative treatment still achieves a non-trivial reduction of complexity with respect to the original problem. ; Comment: 31 pages, 5 figures