This study intended to evaluate a contemporary concept of scaffolding in bone tissue engineering in order to mimic functions of the extracellular matrix. The investigated approach considered the effect of the glycosaminoglycan heparin on structural and biological properties of a synthetic biomimetic bone graft material consisting of mineralized collagen. Two strategies for heparin functionalization were explored in order to receive a three-component bone substitute material. Heparin was either incorporated during matrix synthesis by mixing with collagen prior to simultaneous fibril reassembly and mineralization (in situ) or added to the matrix after fabrication (a posteriori). Both methods resulted in an incorporation of comparable amounts of heparin, though its distribution in the matrix varied as indicated by TOF-SIMS analyses, and a similar modulation of their protein binding properties. Differential scanning calorimetry revealed that the thermal stability and thereby the degree of crosslinking of the heparinized matrices was increased. However, in contrast to the a posteriori modification, the in situ integration of heparin led to considerable changes of morphology and composition of the matrix: a more open network of collagen fibers yielding a more porous surface and a reduced mineral content were observed. Cell culture experiments with human mesenchymal stem cells (hMSC) revealed a strong influence of the mode of heparin functionalization on cellular processes, as demonstrated for proliferation and osteogenic differentiation of hMSC. Our results indicate that not only heparin per se but also the way of its incorporation into a collagenous matrix determines the cell response. In conclusion, the a posteriori modification was beneficial to support adhesion, proliferation and differentiation of hMSC.