We used THz (terahertz) and INS (inelastic neutron scattering) spectroscopies to study the interaction between an endohedral noble gas atom and the C60 molecular cage. The THz absorption spectra of powdered A@C60 samples (A = Ar, Ne, Kr) were measured in the energy range from 0.6 to 75 meV for a series of temperatures between 5 and 300 K. The INS measurements were carried out at liquid helium temperature in the energy transfer range from 0.78 to 54.6 meV. The THz spectra are dominated by one line, between 7 and 12 meV, at low temperatures for three noble gas atoms studied. The line shifts to higher energy and broadens as the temperature is increased. Using a spherical oscillator model, with a temperature-independent parameterized potential function and an atom-displacement-induced dipole moment, we show that the change of the THz spectrum shape with temperature is caused by the anharmonicity of the potential function. We find good agreement between experimentally determined potential energy functions and functions calculated with Lennard-Jones additive pair-wise potentials with parameters taken from the work of Pang and Brisse, J. Chem. Phys. 97, 8562 (1993).