The quality of super resolution images obtained by stochastic single-molecule microscopy critically depends on image analysis algorithms. We find that the choice of background estimator is often the most important determinant of reconstruction quality. A variety of techniques have found use, but many have a very narrow range of applicability depending upon the characteristics of the raw data. Importantly, we observe that when using otherwise accurate algorithms, unaccounted background components can give rise to biases on scales defeating the purpose of super-resolution microscopy. We find that a temporal median filter in particular provides a simple yet effective solution to the problem of background estimation, which we demonstrate over a range of imaging modalities and different reconstruction methods.