The effects of the dielectric environment on the optical extinction spectra of gold nanorods were quantitatively studied using individual bare and silica-coated nanorods. The dispersion and amplitude of their extinction cross-section, dominated by absorption for the investigated sizes, were measured using spatial modulation spectroscopy (SMS). The experimental results were compared to calculations from a numerical model that included environmental features present in the measurements and the morphology and size of the corresponding nanorods measured by transmission electron microscopy. The combination of these experimental and theoretical tools permits a detailed interpretation of the optical properties of the individual nanorods. The measured optical extinction spectra and the extinction cross-section amplitudes were well reproduced by the numerical model for silica-coated gold nanorods, for which the silica shell provides a controlled environment. In contrast, additional environmental factors had to be assumed in the model for bare nanorods, stressing the importance of controlling and characterizing the experimental conditions when measuring the optical response of bare surface-deposited single metal nanoparticles.