Context: The study of variations in total solar irradiance (TSI) is important for understanding how the Sun affects the Earth's climate. Aims: Full-disk continuum images and magnetograms are now available for three full solar cycles. We investigate how modelled TSI compares with direct observations by building a consistent modelled TSI dataset. The model, based only on changes in the photospheric magnetic flux can then be tested on rotational, cyclical and secular timescales. Methods: We use Kitt Peak and SoHO/MDI continuum images and magnetograms in the SATIRE-S model to reconstruct TSI over cycles 21-23. To maximise independence from TSI composites, SORCE/TIM TSI data are used to fix the one free parameter of the model. We compare and combine the separate data sources for the model to estimate an uncertainty on the reconstruction and prevent any additional free parameters entering the model. Results: The reconstruction supports the PMOD composite as being the best historical record of TSI observations, although on timescales of the solar rotation the IRMB composite provides somewhat better agreement. Further to this, the model is able to account for 92% of TSI variations from 1978 to 2009 in the PMOD composite and over 96% during cycle 23. The reconstruction also displays an inter-cycle, secular decline of 0.20 (+0.12 / -0.09) Wm-2 between cycle 23 minima, in agreement with the PMOD composite. Conclusions: SATIRE-S is able to recreate TSI observations on all timescales of a day and longer over 31 years from 1978. This is strong evidence that changes in photospheric magnetic flux alone are responsible for almost all solar irradiance variations over the last three solar cycles. ; Comment: 15 pages, 10 figures