We have performed Raman scattering measurements on URu$_2$Si$_2$ single crystals on a large energy range up to $∼$ 1300 cm$^{-1}$ and in all the Raman active symmetries as a function of temperature down to 15 K. A large excitation, active only in the E$_{g}$ symmetry, is reported. It has been assigned to a crystal electric field excitation on the Uranium site. We discuss how this constrains the crystal electric field scheme of the Uranium ions. Furthermore, three excitations in the A$_{1g}$ symmetry are observed. They have been associated to double Raman phonon processes consistently with ab initio calculations of the phonons dispersion.