The evolution of photoluminescence (PL) spectra of single GaAs/AlGaAs quantum dots (QD) is studied as a function of laser excitation power and temperature. At very low powers, where multiexciton occupation of the QD can be excluded, an unexpected and pronounced spectral evolution is observed (large energy shifts and appearance of multiple emission lines). A similar evolution is observed at low excitation powers with increasing temperature. A model, taking into account the influence of the shallow, residual impurities in the environment of each QD, explains the observed spectral evolutions in terms of photo-depletion of the QD and hopping of impurity-bound carriers back into the QD. Theoretical calculations of the PL due to N electrons ‡ 1 hole (Ne ‡ 1h) QD states allow us to attribute the � 2 meV spaced lines in the experimental spectra to the different charge states Ne ‡ 1h, (N ±± 1) e ‡ 1h, . . . of the QD.