The GRAVITY instrument on the ESO VLTI pioneers the field of high-precision near-infrared interferometry by providing astrometry at the 10−100 μas level. Measurements at this high precision crucially depend on the control of systematic effects. We investigate how aberrations introduced by small optical imperfections along the path from the telescope to the detector affect the astrometry. We develop an analytical model that describes the effect of these aberrations on the measurement of complex visibilities. Our formalism accounts for pupil-plane and focal-plane aberrations, as well as for the interplay between static and turbulent aberrations, and it successfully reproduces calibration measurements of a binary star. The Galactic Center observations with GRAVITY in 2017 and 2018, when both Sgr A* and the star S2 were targeted in a single fiber pointing, are affected by these aberrations at a level lower than 0.5 mas. Removal of these effects brings the measurement in harmony with the dual-beam observations of 2019 and 2020, which are not affected by these aberrations. This also resolves the small systematic discrepancies between the derived distance R₀ to the Galactic Center that were reported previously.