Manipulation of individual atoms and molecules by scanning probe microscopy offers the ability of controlled assembly at the single-atom scale. However, the driving forces behind atomic manipulation have not yet been measured. We used an atomic force microscope to measure the vertical and lateral forces exerted on individual adsorbed atoms or molecules by the probe tip. We found that the force that it takes to move an atom depends strongly on the adsorbate and the surface. Our results indicate that for moving metal atoms on metal surfaces, the lateral force component plays the dominant role. Furthermore, measuring spatial maps of the forces during manipulation yielded the full potential energy landscape of the tip-sample interaction.