Accurate estimation of Energy Expenditure (EE) in ambulatory settings is a key element in determining the causal relation between aspects of human behavior related to physical activity and health. We present a new methodology for activity-specific EE algorithms. The proposed methodology models activity clusters using specific parameters that capture differences in EE within a cluster, and combines these models with Metabolic Equivalents (METs) derived from the compendium of physical activities. We designed a protocol consisting of a wide set of sedentary, household, lifestyle and gym activities, and developed a new activity-specific EE algorithm applying the proposed methodology. The algorithm uses accelerometer (ACC) and heart rate (HR) data acquired by a single monitoring device, together with anthropometric variables, to predict EE. Our model recognizes six clusters of activities independent of the subject in 52.6 hours of recordings from 19 participants. Increases in EE estimation accuracy ranged from 18 to 31% compared to state of the art single and multi-sensor activity-specific methods.