A dielectric material’s response to light is microscopically defined by field-cycle-driven motion of electron densities in the restoring forces of the atomic environment. Here we apply single-cycle mid-infrared pulses to drive the nonlinear motion of valence electrons in a heteronuclear crystal with asymmetric structure and report how the macroscopic optical response can be tracked back to the real-space electron dynamics in the symmetry-breaking potential along the chemical bonds. Whether our single-cycle field drives electrons from the less electronegative to the more electronegative element or vice versa controls the appearance of a smooth nonlinear output spectrum or one with even and odd harmonic orders. Crystal angle scans reveal the absolute orientation of the asymmetric bonds. Directional motion of valence charges controlled by a single cycle of light can therefore be used for spectroscopically exploring the binding potential, to understand and design novel materials for nonlinear optics, or to eventually process information at the frequency of light.