We present a detailed study of the dust energy balance in the Sombrero galaxy M104. From a full radiative transfer analysis, including scattering, absorption and thermal re-emission, we construct models that can reproduce images at optical/near-infrared (near-IR) wavelengths, the observed stellar spectral energy distribution and the minor axis extinction profiles in the V and R-C band. A standard model, which contains only an old stellar population to heat the dust, underestimates the observations of dust emission at IR wavelengths by a factor of similar to 3. Supplementing this basic model with a young stellar component of low star formation activity in both the inner disc (star formation rate, SFR similar to 0.21 M-circle dot yr(-1)) and dust ring (SFR similar to 0.05 M-circle dot yr(-1)), we are capable of solving the discrepancy in the dust energy budget of the Sombrero galaxy at wavelengths shortwards of 100 mu m. To account for the increased far-IR/submm emission beyond 100 mu m, we propose an additional dust component distributed in quiescent clumps. This model with a clumpy dust structure predicts three-quarters of the total dust content (similar to 2.8 x 10(7) M-circle dot) to reside in compact dust clouds with no associated embedded sources. Although the assumption of a clumpy dust structure in the Sombrero galaxy is supported by high-resolution optical data, we cannot rule out the possibility that dust grains with a higher dust emissivity account for part of the discrepancy in the energy budget at submm wavelengths.