We study the ultrametric structure of phase space of one-dimensional Ising spin glasses with random power-law interaction in an external random field. Although in zero field the model in both the mean-field and non-mean-field universality classes shows an ultrametric signature [Phys. Rev. Lett. 102, 037207 (2009)], when a field is applied ultrametricity seems only present in the mean-field regime. The results for the non-mean field case in an external field agree with data for spin glasses studied within the Migdal-Kadanoff approximation. Our results therefore suggest that the spin-glass state might be fragile to external fields below the upper critical dimension. ; Comment: 5 pages, 4 figures, 1 table