Background: The risk of breast cancer in carriers of BRCA1 and BRCA2 mutations is influenced by factors other than the genetic mutation itself. Modifying factors include a woman’s reproductive history and family history of cancer. Risk factors are more likely to be present in women with breast cancer than in women without breast cancer, and therefore the risk of cancer in the two breasts should not be independent. It is not clear to what extent modifying factors influence the risk of a first primary or a contralateral breast cancer in BRCA carriers. Methods: We conducted a matched case–control study of breast cancer among 3920 BRCA1 or BRCA2 mutation carriers. We asked whether a past history of breast cancer in the contralateral breast was a risk factor for breast cancer. Results: After adjustment for age, country of residence, and cancer treatment, a previous cancer of the right breast was found to be a significant risk factor for cancer of the left breast among BRCA1 or BRCA2 carriers (relative risk: 2.1; 95% confidence interval: 1.4 to 3.0; p < 0.0001). Conclusions: In a woman with a BRCA1 or BRCA2 mutation who is diagnosed with breast cancer, the risk of cancer in the contralateral breast depends on the first diagnosis. That observation supports the hypothesis that there are important genetic or non-genetic modifiers of cancer risk in BRCA carriers. Discovering risk modifiers might lead to greater personalization of risk assessment and management recommendations for BRCA-positive patients.