We measure the anomalous spectral diffusion of single colloidal quantum dots over eight temporal decades simultaneously by combining single-molecule spectroscopy and photon-correlation Fourier spectroscopy. Our technique distinguishes between discrete and continuous dynamics and directly reveals that the quasicontinuous spectral diffusion observed using conventional spectroscopy is composed of rapid, discrete spectral jumps. Despite their multiple time scales, these dynamics can be captured by a single mechanism whose parameters vary widely between dots and over time in individual dots.