We derive a many-body method to evaluate photoelectron spectra of atoms, molecules and clusters from first principles. The excitation energies and the spectroscopic factors are calculated from the linear-response time-dependent density functional theory. The method is applied to noble metal anions, anionic clusters and to neutral small molecules. Our approach shows significant improvement over a simple single-particle treatment and gives an insight into the necessary conditions under which the single-particle picture holds. The consideration of the spectroscopic factor is shown to be crucial for the correct description of inner valence photoelectron peaks.