Black hole spectroscopy is the program to measure the complex gravitational-wave frequencies of merger remnants, and to quantify their agreement with the characteristic frequencies of black holes computed at linear order in black hole perturbation theory. In a "weaker" (non-agnostic) version of this test, one assumes that the frequencies depend on the mass and spin of the final Kerr black hole as predicted in perturbation theory. Linear perturbation theory is expected to be a good approximation only at late times, when the remnant is close enough to a stationary Kerr black hole. However, it has been claimed that a superposition of overtones with frequencies fixed at their asymptotic values in linear perturbation theory can reproduce the waveform strain even at the peak. Is this overfitting, or are the overtones physically present in the signal? To answer this question, we fit toy models of increasing complexity, waveforms produced within linear perturbation theory, and full numerical relativity waveforms using both agnostic and non-agnostic ringdown models. We find that higher overtones are unphysical: their role is mainly to "fit away" features such as initial data effects, power-law tails, and (when present) nonlinearities. We then identify physical quasinormal modes by fitting numerical waveforms in the original, agnostic spirit of the no-hair test. We find that a physically meaningful ringdown model requires the inclusion of higher multipoles, quasinormal mode frequencies induced by spherical-spheroidal mode mixing, and nonlinear quasinormal modes. Even in this "infinite signal-to-noise ratio" version of the original spectroscopy test, there is convincing evidence for the first overtone of the dominant multipole only well after the peak of the radiation.