Abstract Purpose There is limited high-quality evidence on quality of life, anxiety, and depressive symptoms in breast cancer survivors and women with no history of cancer. We aimed to address this by comparing patient-reported outcomes between breast cancer survivors and women with no history of breast cancer. Methods Breast cancer survivors and women with no prior cancer were selected from the UK Clinical Practice Research Datalink GOLD primary care database, which includes population-based primary care electronic health record data. Breast cancer survivors and controls were frequency matched by age and primary care practice. Outcomes were assessed with validated instruments via postal questionnaire. Linear and logistic regression models were fitted to estimate adjusted associations between breast cancer survivorship and outcomes. Results A total of 356 breast cancer survivors (8.1 years post diagnosis) and 252 women with no prior cancer participated in the study. Compared with non-cancer controls, breast cancer survivors had poorer QoL in the domains of cognitive problems (adjusted β (aβ) = 1.4, p = 0.01), sexual function (aβ = 1.7, p = 0.02) and fatigue (aβ = 1.3, p = 0.01), but no difference in negative feelings, positive feelings, pain, or social avoidance. Breast cancer survivors had higher odds of borderline-probable anxiety (score ≥ 8) (adjusted OR = 1.47, 95%CI:1.15–1.87), but no differences in depression. Advanced stage at diagnosis and chemotherapy treatment were associated with poorer QoL. Conclusions Compared with women with no history of cancer, breast cancer survivors report more problems with cognition, sexual function, fatigue, and anxiety, particularly where their cancer was advanced and/or treated with chemotherapy. Implications for Cancer Survivors Breast cancer survivors with more advanced disease and/or treated with chemotherapy should be closely monitored and, when possible, offered evidence-based intervention for fatigue, cognitive dysfunction, and sexual problems.