We demonstrate that complex luminescence spectra from silicon wafers can be accurately modelled using a simple expression for the individual phonon-related components. By using combinations of various phonon emission and absorption events, it is possible to build up the entire luminescence spectra, which we demonstrate for temperatures between 78 to 300 K. At room temperature, the dominant components of the spectra are caused by emission and absorption of momentum-conserving transverse optical and transverse acoustic phonons. Secondary phonon replicas also involve the emission of non-momentum-conserving inter-valley phonons. The model correctly reproduces the changes in sharpness of the measured spectra as a function of temperature, as well as correctly accounting for the relative change in impact of phonon emission and absorption related luminescence. At low temperatures, additional peaks caused by recombination through neutral dopant atoms are also evident, and can be modelled using the same approach. The model may be useful for understanding luminescence spectra from silicon wafers and solar cells.