Breast adiposity is correlated with body mass index, menopausal status and mammary density. We here wish to establish how these factors influence the cross-talk between breast adipocytes and normal or malignant breast cells. Adipocyte-derived stem cells (ASCs) were obtained from healthy women and classified into six distinct groups based on body mass index, menopausal status and mammary density. The ASCs were induced to differentiate, and the influence of their conditioned media (ACM) was determined. Unexpectedly, there were no detectable differences in adipogenic differentiation and secretion between the six ASC groups, while their corresponding ACMs had no detectable influence on normal breast cells. In clear contrast, all ACMs profoundly influenced the proliferation, migration and invasiveness of malignant breast cells and increased the number of lipid droplets in their cytoplasm via increased expression of the fatty acid receptor CD36, thereby increasing fatty acid uptake. Importantly, inhibition of CD36 reduced lipid droplet accumulation and attenuated the migration and invasion of the breast cancer cells. These findings suggest that breast-associated adipocytes potentiate the invasiveness of breast cancer cells which, at least in part, is mediated by metabolic reprogramming via CD36-mediated fatty acid uptake.