The confinement of high frequency phonons approaching 1 GHz is demonstrated in phonon-trapping acoustic cavities at cryogenic temperatures using a low-coupled network approach. The frequency range is extended by nearly an order of magnitude, with excitation at greater than the 200$^{\text{th}}$ overtone achieved for the first time. Such high frequency operation reveals Rayleigh-type phonon scattering losses due to highly diluted lattice impurities and corresponding glass-like behaviour, with a maximum $Q_L\times f$ product of $8.6\times 10^{17}$ at 3.8K and $4\times10^{17}$ at 15mK. This suggests a limit on the $Q\times f$ product due to unavoidable crystal disorder. Operation at 15 mK is high enough in frequency that the average phonon occupation number is less than unity, with a loaded quality factor above half a billion. This work represents significant progress towards the utilisation of such acoustic cavities for hybrid quantum systems.