The detection of the first electromagnetic counterpart to the binary neutron star (BNS) merger remnant GW170817 established the connection between short $γ$-ray bursts and BNS mergers. It also confirmed the forging of heavy elements in the ejecta (a so-called kilonova) via the r-process nucleosynthesis. The appearance of non-thermal radio and X-ray emission, as well as the brightening, which lasted more than 100 days, were somewhat unexpected. Current theoretical models attempt to explain this temporal behavior as either originating from a relativistic off-axis jet or a kilonova-like outflow. In either scenario, there is some ambiguity regarding how much energy is transported in the non-thermal electrons versus the magnetic field of the emission region. Combining the VLA (radio) and Chandra (X-ray) measurements with observations in the GeV-TeV domain can help break this ambiguity, almost independently of the assumed origin of the emission. Here we report for the first time on deep H.E.S.S. observations of GW170817 / GRB 170817A between 124 and 272 days after the BNS merger with the full H.E.S.S. array of telescopes, as well as on an updated analysis of the prompt (<5 days) observations with the upgraded H.E.S.S. phase-I telescopes. We discuss implications of the H.E.S.S. measurement for the magnetic field in the context of different source scenarios.