The amount and the distribution of body fat exhibit trajectories that are sex- and human species-specific and both are determinants for health. The enhanced accumulation of fat in the truncal part of the body as a risk factor for cardiovascular and metabolic diseases is well supported by epidemiological studies. In addition, a possible independent protective role of the gluteofemoral fat compartment and of the brown adipose tissue is emerging. The present narrative review summarizes the current knowledge on sexual dimorphism in fat depot amount and repartition and consequences on cardiometabolic and reproductive health. The drivers of the sex differences and fat depot repartition, considered to be the results of complex interactions between sex determination pathways determined by the sex chromosome composition, genetic variability, sex hormones and the environment, are discussed. Finally, the inter- and intra-depot heterogeneity in adipocytes and progenitors, emphasized recently by unbiased large-scale approaches, is highlighted.