ObjectivesTo assess: (1) the impact of a reproductive health program on modern contraceptive use from baseline to program close; (2) the sustained impact from baseline to follow-up 36 months later; and (3) the exposure-adjusted impact at program close and follow-up.DesignRetrospective, cross-sectional matched control study.SettingKarachi, Pakistan.Participants2561 married women aged 16–49 years.InterventionsThe Willows Program, a community-based family planning counselling and referral program implemented from 2013 to 2015.Primary and secondary outcome measuresThe primary outcome was community-level modern contraceptive prevalence rate (mCPR), measured for January 2013 (baseline), June 2015 (program close) and at follow-up 36 months later. A secondary outcome was exposure-adjusted mCPR (among women reporting a family planning home visit) at program close and at follow-up.ResultsThere was no significant effect on community-level mCPR at program close (2.4 percentage point increase in intervention over comparison; 95% CI −2.2 to 7.0) or at follow-up (1.9 percentage point decrease; 95% CI −6.7 to 2.8). Only 18% of women in the intervention area reported receiving a family planning visit in the preceding 5 years. Among those reporting a visit, we observed a significant 10.3 percentage point increase (95% CI 4.6 to 15.9) from baseline to close, and a non-significant 2.0 percentage point increase (95% CI −3.8 to 7.8) from baseline to follow-up, relative to matched women in the comparison area. The cost per new modern method user was US$1089, while the cost per user-year during the intervention period was US$455.ConclusionsThe program had a positive short-term effect on women who received a family planning visit; however, this effect was not sustained. Program coverage was low and did not significantly increase community-level family planning use. Findings highlight the need to increase community coverage of high-quality counselling and contextually relevant interventions for family planning demand generation.