A link has been proposed between the use of nonsteroidal anti-inflammatory drugs (NSAIDs) and the risk of breast cancer. There is, however, insufficient data regarding the subtype and stage of breast cancer, and few studies have assessed the interaction between the use of NSAIDs and breast density or previous breast disorders. There is also a lack of data from population-based studies. We first conducted a nested case-control study within the general female population of Sweden, including 56,480 women with newly diagnosed breast cancer during 2006–2015 and five breast cancer-free women per case as controls, to assess the association of NSAID use with the risk of incident breast cancer, focusing on subtype and stage of breast cancer as well as the interaction between NSAID use and previous breast disorders. We then used the Karolinska Mammography Project for Risk Prediction of Breast Cancer (Karma) cohort to assess the interaction between NSAID use and breast density in relation to the risk of breast cancer. Conditional logistic regression was used to estimate the hazard ratio (HR) and a 95% confidence interval (CI) was used for breast cancer in relation to the use of aspirin and non-aspirin NSAIDs. In the nested case-control study of the general population, exclusive use of aspirin was not associated with the risk of breast cancer, whereas exclusive use of non-aspirin NSAIDs was associated with a modestly higher risk of stage 0–2 breast cancer (HR: 1.05; 95% CI: 1.02–1.08) but a lower risk of stage 3–4 breast cancer (HR 0.80; 95% CI: 0.73–0.88). There was also a statistically significant interaction between the exclusive use of NSAIDs and previous breast disorders (p for interaction: <0.001). In the analysis of Karma participants, the exclusive use of non-aspirin NSAIDs was associated with a lower risk of breast cancer among women with a breast dense area of >40 cm2 (HR: 0.72; 95% CI: 0.59–0.89). However, the possibility of finding this by chance cannot be ruled out. Overall, we did not find strong evidence to support an association between the use of NSAIDs and the risk of breast cancer.