We report a comprehensive investigation of the lattice dynamics of URu$_2$Si$_2$ as a function of temperature using Raman scattering, optical conductivity and inelastic neutron scattering measurements as well as theoretical {\it ab initio} calculations. The main effects on the optical phonon modes are related to Kondo physics. The B$_{1g}$ ($Γ_3$ symmetry) phonon mode slightly softens below $∼$100~K, in connection with the previously reported softening of the elastic constant, $C_{11}-C_{12}$, of the same symmetry, both observations suggesting a B$_{1g}$ symmetry-breaking instability in the Kondo regime. Through optical conductivity, we detect clear signatures of strong electron-phonon coupling, with temperature dependent spectral weight and Fano line shape of some phonon modes. Surprisingly, the line shapes of two phonon modes, E$_u$(1) and A$_{2u}$(2), show opposite temperature dependencies. The A$_{2u}$(2) mode loses its Fano shape below 150 K, whereas the E$_u$(1) mode acquires it below 100~K, in the Kondo cross-over regime. This may point out to momentum-dependent Kondo physics. By inelastic neutron scattering measurements, we have drawn the full dispersion of the phonon modes between 300~K and 2~K. No remarkable temperature dependence has been obtained including through the hidden order transition. {\it Ab initio} calculations with the spin-orbit coupling are in good agreement with the data except for a few low energy branches with propagation in the (a,b) plane.