We report an inelastic neutron scattering study of acoustic phonons in the charge and orbitally ordered bilayer manganite LaSr$_2$Mn$_2$O$_7$. For excitation energies less than $15\,\rm{meV}$, we observe an abrupt increase (decrease) of the phonon energies (linewidths) of a transverse acoustic phonon branch at $𝐪 = (h, h, 0)$, $h\le0.3$, upon entering the low temperature charge and orbital ordered state ($T_{COO} = 225\,\rm{K}$). This indicates a reduced electron-phonon coupling due to a decrease of electronic states at the Fermi level leading to a partial removal of the Fermi surface below $T_{COO}$ and provides direct experimental evidence for a link between electron-phonon coupling and charge order in manganites. ; Comment: accepted in Physical Review Letters