Aims: We investigate how well modeled solar irradiances agree with measurements from the SORCE satellite, both for total solar irradiance and broken down into spectral regions on timescales of several years. Methods: We use the SATIRE model and compare modeled total solar irradiance (TSI) with TSI measurements between 2003 and 2009. Spectral solar irradiance over 200-1630nm is compared with the SIM instrument on SORCE between 2004 and 2009 during a period of decline from moderate activity to the recent solar minimum in 10 nm bands and for three spectral regions of significant interest: the UV integrated over 200-300nm, the visible over 400-691nm and the IR between 972-1630 nm. Results: The model captures 97% of observed TSI variation. In the spectral comparison, rotational variability is well reproduced, especially between 400 and 1200 nm. The magnitude of change in the long-term trends is many times larger in SIM at almost all wavelengths while trends in SIM oppose SATIRE in the visible between 500 and 700nm and between 1000 and 1200nm. We discuss the remaining issues with both SIM data and the identified limits of the model, particularly with the way facular contributions are dealt with, the limit of flux identification in MDI magnetograms during solar minimum and the model atmospheres in the IR employed by SATIRE. It is unlikely that improvements in these areas will significantly enhance the agreement in the long-term trends. This disagreement implies that some mechanism other than surface magnetism is causing SSI variations, in particular between 2004 and 2006, if the SIM data are correct. Since SATIRE was able to reproduce UV irradiance between 1991 and 2002 from UARS, either the solar mechanism for SSI variation fundamentally changed around the peak of cycle 23, or there is an inconsistency between UARS and SORCE UV measurements. We favour the second explanation. ; Comment: 14 pages, 13 figures