The fine structure of the rotational ground state of molecular ortho-hydrogen confined inside the fullerene cage C60 is investigated by inelastic neutron scattering (INS). The INS line corresponding to transitions between the three sub-levels comprising the ortho ground state to the non-degenerate para ground state was studied as a function of temperature down to 60 mK in neutron energy gain. The experiments show that at ambient pressure the three ortho sub-levels are split into a low energy non-degenerate level and a high energy doubly degenerate level separated by 0.135 ± 0.010 meV. This observation is consistent with hydrogen molecules being located at sites with axial symmetry superseding the icosahedral symmetry of isolated rigid C60 cages in the solid phase. To gain insight into the role of inter-cage interactions in determining the symmetry breaking potential, the effects of hydrostatic pressure on the fine structure of the line was also investigated. The analysis of the INS spectra shows that the potential and the energy levels of H2 are sensitive to the orientation of neighbouring cages, consistent with the low-temperature crystalline phase of C60.