Positron emission tomography using [18F]fluorodeoxyglucose (FDG PET) potentially underperforms for staging of patients with grade 1–2 estrogen receptor positive (ER+) breast cancer. The aim of this study was to retrospectively investigate the diagnostic accuracy of FDG PET in this patient population. Suspect tumor lesions detected on conventional imaging and FDG PET were confirmed with pathology or follow up. PET-positive lesions were (semi)quantified with standardized uptake values (SUV) and these were correlated with various pathological features, including the histological subtype. Pre-operative imaging detected 155 pathologically verified lesions (in 74 patients). A total of 115/155 (74.2%) lesions identified on FDG PET were classified as true positive, i.e., malignant (in 67 patients) and 17/155 (10.8%) lesions as false positive, i.e., benign (in 9 patients); 7/155 (4.5%) as false negative (in 7 patients) and 16/155 (10.3%) as true negative (in 14 patients). FDG PET incorrectly staged 16/70 (22.9%) patients. The FDG uptake correlated with histological subtype, showing higher uptake in ductal carcinoma, compared to lobular carcinoma (p < 0.05). Conclusion: Within this study, FDG PET inadequately staged 22.9% of grade 1–2, ER + BC cases. Incorrect staging can lead to inappropriate treatment choices, potentially affecting survival and quality of life. Prospective studies investigating novel radiotracers are urgently needed.