We explore the field-temperature phase diagram of the XY pyrochlore antiferromagnet Er$_2$Ti$_2$O$_7$, by means of magnetization and neutron diffraction experiments. Depending on the field strength and direction relative to the high symmetry cubic directions $[001], [1\bar{1}0]$ and $[111]$, the refined field induced magnetic structures are derived from the zero field $ψ_2$ and $ψ_3$ states of the $Γ_5$ irreducible representation which describes the ground state of XY pyrochlore antiferromagnets. At low field, domain selection effects are systematically at play. In addition, for $[001]$, a phase transition is reported towards a $ψ_3$ structure at a critical field $H_c^{001}=$ 43 mT. For $[1\bar{1}0]$ and $[111]$, the spins are continuously tilted by the field from the $ψ_2$ state, and no phase transition is found while domain selection gives rise to sharp anomalies in the field dependence of the Bragg peaks intensity. For $[1\bar{1}0]$, these results are confirmed by high resolution inelastic neutron scattering experiments, which in addition allow to determine the field dependence of the spin gap. This study agrees qualitatively with the scenario proposed theoretically by Maryasin et al. [Phys. Rev. B 93, 100406(R) (2016)], yet the strength of the field induced anisotropies is significantly different from theory. ; Comment: 13 pages, 10 figures