We mechanically clean graphene devices using an atomic force microscope (AFM). By scanning an AFM tip in contact mode in a broom-like way over the sample, resist residues are pushed away from the desired area. We obtain atomically flat graphene with a root mean square (rms) roughness as low as 0.12 nm after this procedure. The cleaning also results in a shift of the charge-neutrality point toward zero gate voltage, as well as an increase in charge carrier mobility. (C) 2012 American Institute of Physics. [http://dx.doi.org/10.1063/1.3695451]