Abstract Background We evaluated deficit accumulation and how deficits affected cognition and physical activity among breast cancer survivors and non-cancer controls. Methods Newly diagnosed nonmetastatic survivors (n = 353) and matched non-cancer controls (n = 355) ages 60-98 years without neurological impairments were assessed presystemic therapy (or at enrollment for controls) from August 2010 to December 2016 and followed for 36 months. Scores on a 42-item index were analyzed in growth-mixture models to determine deficit accumulation trajectories separately and combined for survivors and controls. Multilevel models tested associations between trajectory and cognition (FACT-Cog and neuropsychological tests) and physical activity (IPAQ-SF) for survivors and controls. Results Deficit accumulation scores were in the robust range, but survivors had higher scores (95% confidence intervals [CI]) than controls at 36 months (0.18, 95% CI = 0.16 to 0.19, vs 0.16, 95% CI = 0.14 to 0.17; P = .001), and averages included diverse deficit trajectories. Survivors who were robust but became frailer (8.8%) had similar baseline characteristics to those remaining robust (76.2%) but experienced a 9.6-point decline self-reported cognition (decline of 9.6 vs 3.2 points; P = .04) and a 769 MET minutes per week decline in physical activity (P < .001). Survivors who started and remained prefrail (15.0%) had self-reported and objective cognitive problems. At baseline, frail controls (9.5%) differed from robust controls (83.7%) on deficits and self-reported cognition (P < .001). Within combined trajectories, frail survivors had more sleep disturbances than frail controls (48.6% [SD = 17.4%] vs 25.0% [SD = 8.2%]; P = .05). Conclusions Most survivors and controls remained robust, and there were similar proportions on a frail trajectory. However, there were differences in deficit patterns between survivors and controls. Survivor deficit accumulation trajectory was associated with patient-reported outcomes. Additional research is needed to understand how breast cancer and its treatments affect deficit accumulation.