The elimination of so-called background fields is an essential step in phase MRI and quantitative susceptibility mapping (QSM). Background fields, which are caused by sources outside the region of interest (ROI), are often one to two orders of magnitude stronger than tissue-related field variations from within the ROI, hampering quantitative interpretation of field maps. This paper reviews the current literature on background elimination algorithms for QSM and provides insights into similarities and differences between the many algorithms proposed. We discuss the basic theoretical foundations and derive fundamental limitations of background field elimination. Copyright © 2016 John Wiley & Sons, Ltd.