We present an observation-based heat budget analysis for seasonal and interannual variations of ocean heat content (H) in the mixed layer (Hmld) and full depth ocean (Htot). Surface heat flux and ocean heat content estimates are combined using a novel Kalman smoother-based method. Regional contributions from ocean heat transport convergences are inferred as a residual and the dominant drivers of Hmld and Htot are quantified for seasonal and interannual time scales. We find that non-Ekman ocean heat transport processes dominate Hmld variations in the equatorial oceans and regions of strong ocean currents and substantial eddy activity. In these locations, surface temperature anomalies generated by ocean dynamics result in turbulent flux anomalies that drive the overlying atmosphere. In addition, we find large regions of the Atlantic and Pacific oceans where heat transports combine with local air-sea fluxes to generate mixed layer temperature anomalies. In all locations except regions of deep convection and water mass transformation, interannual variations in Htot are dominated by the internal rearrangement of heat by ocean dynamics rather than the loss or addition of heat at the surface. Our analysis suggests that, even in extra-tropical latitudes, initialization of ocean dynamical processes could be an important source of skill for interannual predictability of Hmld and Htot. Furthermore, we expect variations in Htot (and thus thermosteric sea level) to be more predictable than near surface temperature anomalies due to the increased importance of ocean heat transport processes for full-depth heat budgets. This article is protected by copyright. All rights reserved.