Male breast cancer (mBC) is associated with a high prevalence of pathogenic variants (PVs) in the BRCA2 gene; however, data regarding other BC predisposition genes are limited. In this retrospective multicenter study, we investigated the prevalence of PVs in BRCA1/2 and 23 non-BRCA1/2 genes using a sample of 614 patients with mBC, recruited through the centers of the German Consortium for Hereditary Breast and Ovarian Cancer. A high proportion of patients with mBC carried PVs in BRCA2 (23.0%, 142/614) and BRCA1 (4.6%, 28/614). The prevalence of BRCA1/2 PVs was 11.0% in patients with mBC without a family history of breast and/or ovarian cancer. Patients with BRCA1/2 PVs did not show an earlier disease onset than those without. The predominant clinical presentation of tumor phenotypes was estrogen receptor (ER)-positive, progesterone receptor (PR)-positive, and HER2-negative (77.7%); further, 10.2% of the tumors were triple-positive, and 1.2% were triple-negative. No association was found between ER/PR/HER2 status and BRCA1/2 PV occurrence. Comparing the prevalence of protein-truncating variants (PTVs) between patients with mBC and control data (ExAC, n = 27,173) revealed significant associations of PTVs in both BRCA1 and BRCA2 with mBC (BRCA1: OR = 17.04, 95% CI = 10.54–26.82, p < 10−5; BRCA2: OR = 77.71, 95% CI = 58.71–102.33, p < 10−5). A case-control investigation of 23 non-BRCA1/2 genes in 340 BRCA1/2-negative patients and ExAC controls revealed significant associations of PTVs in CHEK2, PALB2, and ATM with mBC (CHEK2: OR = 3.78, 95% CI = 1.59–7.71, p = 0.002; PALB2: OR = 14.77, 95% CI = 5.02–36.02, p < 10−5; ATM: OR = 3.36, 95% CI = 0.89–8.96, p = 0.04). Overall, our findings support the benefit of multi-gene panel testing in patients with mBC irrespective of their family history, age at disease onset, and tumor phenotype.