Abstract Background Delays between breast cancer diagnosis and surgery are associated with worsened survival. Delays are more common in urban-residing patients, although factors specific to surgical delays among rural and urban patients are not well understood. Methods We used a 100% sample of fee-for-service Medicare claims during 2007–2014 to identify 238,491 women diagnosed with early-stage breast cancer undergoing initial surgery and assessed whether they experienced biopsy-to-surgery intervals > 90 days. We employed multilevel regression to identify associations between delays and patient, regional, and surgeon characteristics, both in combined analyses and stratified by rurality of patient residence. Results Delays were more prevalent among urban patients (2.5%) than rural patients (1.9%). Rural patients with medium- or high-volume surgeons had lower odds of delay than patients with low-volume surgeons (odds ratio [OR] = 0.71, 95% confidence interval [CI] = 0.58–0.88; OR = 0.74, 95% CI = 0.61–0.90). Rural patients whose surgeon operated at ≥ 3 hospitals were more likely to experience delays (OR = 1.29, 95% CI = 1.01–1.64, Ref: 1 hospital). Patient driving times ≥ 1 h were associated with delays among urban patients only. Age, black race, Hispanic ethnicity, multimorbidity, and academic/specialty hospital status were associated with delays. Conclusions Sociodemographic, geographic, surgeon, and facility factors have distinct associations with > 90-day delays to initial breast cancer surgery. Interventions to improve timeliness of breast cancer surgery may have disparate impacts on vulnerable populations by rural-urban status.