Inelastic neutron scattering measurements on silver oxide (Ag$_2$O) with the cuprite structure were performed at temperatures from 40 to 400\,K, and Fourier transform far-infrared spectra were measured from 100 to 300\,K. The measured phonon densities of states and the infrared spectra showed unusually large energy shifts with temperature, and large linewidth broadenings. First principles molecular dynamics (MD) calculations were performed at various temperatures, successfully accounting for the negative thermal expansion (NTE) and local dynamics. Using the Fourier-transformed velocity autocorrelation method, the MD calculations reproduced the large anharmonic effects of Ag$_2$O, and were in excellent agreement with the neutron scattering data. The quasiharmonic approximation (QHA) was less successful in accounting for much of the phonon behavior. The QHA could account for some of the NTE below 250 K, although not at higher temperatures. Strong anharmonic effects were found for both phonons and for the NTE. The lifetime broadenings of Ag$_2$O were explained by anharmonic perturbation theory, which showed rich interactions between the Ag-dominated modes and the O-dominated modes in both up- and down-conversion processes.