Abstract Background The terminal duct lobular unit (TDLU) is the most dynamic structure in the human breast and the putative site of origin of human breast cancer. Although stromal cells contribute to a specialized microenvironment in many organs, this component remains largely understudied in the human breast. We here demonstrate the impact on epithelium of two lineages of breast stromal fibroblasts, one of which accumulates in the TDLU while the other resides outside the TDLU in the interlobular stroma. Methods The two lineages are prospectively isolated by fluorescence activated cell sorting (FACS) based on different expression levels of CD105 and CD26. The characteristics of the two fibroblast lineages are assessed by immunocytochemical staining and gene expression analysis. The differentiation capacity of the two fibroblast populations is determined by exposure to specific differentiating conditions followed by analysis of adipogenic and osteogenic differentiation. To test whether the two fibroblast lineages are functionally imprinted by their site of origin, single cell sorted CD271 low /MUC1 high normal breast luminal epithelial cells are plated on fibroblast feeders for the observation of morphological development. Epithelial structure formation and polarization is shown by immunofluorescence and digitalized quantification of immunoperoxidase-stained cultures. Results Lobular fibroblasts are CD105 high /CD26 low while interlobular fibroblasts are CD105 low /CD26 high . Once isolated the two lineages remain phenotypically stable and functionally distinct in culture. Lobular fibroblasts have properties in common with bone marrow derived mesenchymal stem cells and they specifically convey growth and branching morphogenesis of epithelial progenitors. Conclusions Two distinct functionally specialized fibroblast lineages exist in the normal human breast, of which the lobular fibroblasts have properties in common with mesenchymal stem cells and support epithelial growth and morphogenesis. We propose that lobular fibroblasts constitute a specialized microenvironment for human breast luminal epithelial progenitors, i.e. the putative precursors of breast cancer.