Li-Fraumeni syndrome (LFS) is an autosomal dominant cancer-predisposition disorder. Approximately 70% of individuals who fit the clinical definition of LFS harbor a pathogenic germline variant in the TP53 tumor suppressor gene. However, the remaining 30% of patients lack a TP53 variant and even among variant TP53 carriers, approximately 20% remain cancer-free. Understanding the variable cancer penetrance and phenotypic variability in LFS is critical to developing rational approaches to accurate, early tumor detection and risk-reduction strategies. We leveraged family-based whole-genome sequencing and DNA methylation to evaluate the germline genomes of a large, multi-institutional cohort of patients with LFS (n = 396) with variant (n = 374) or wildtype TP53 (n = 22). We identified alternative cancer-associated genetic aberrations in 8/14 wildtype TP53 carriers who developed cancer. Among variant TP53 carriers, 19/49 who developed cancer harbored a pathogenic variant in another cancer gene. Modifier variants in the WNT signaling pathway were associated with decreased cancer incidence. Furthermore, we leveraged the noncoding genome and methylome to identify inherited epimutations in genes including ASXL1, ETV6, and LEF1 that confer increased cancer risk. Using these epimutations, we built a machine learning model that can predict cancer risk in patients with LFS with an area under the receiver operator characteristic curve (AUROC) of 0.725 (0.633–0.810). Significance: Our study clarifies the genomic basis for the phenotypic variability in LFS and highlights the immense benefits of expanding genetic and epigenetic testing of patients with LFS beyond TP53. More broadly, it necessitates the dissociation of hereditary cancer syndromes as single gene disorders and emphasizes the importance of understanding these diseases in a holistic manner as opposed to through the lens of a single gene.