Penile erection is mediated by the corpora cavernosa, a trabecular-like vascular bed that enlarges upon vasodilation, but its regulation is not completely understood. Here, we show that perivascular fibroblasts in the corpora cavernosa support vasodilation by reducing norepinephrine availability. The effect on penile blood flow depends on the number of fibroblasts, which is regulated by erectile activity. Erection dynamically alters the positional arrangement of fibroblasts, temporarily down-regulating Notch signaling. Inhibition of Notch increases fibroblast numbers and consequently raises penile blood flow. Continuous Notch activation lowers fibroblast numbers and reduces penile blood perfusion. Recurrent erections stimulate fibroblast proliferation and limit vasoconstriction, whereas aging reduces the number of fibroblasts and lowers penile blood flow. Our findings reveal adaptive, erectile activity-dependent modulation of penile blood flow by fibroblasts.