Abstract Context Pancreatic steatosis leading to beta-cell failure might be involved in type 2 diabetes (T2D) pathogenesis. Objective We hypothesized that the genetic background modulates the effect of pancreatic fat on beta-cell function and investigated genotype × pancreatic fat interactions on insulin secretion. Design Two observational studies. Setting University hospital. Patients or participants A total of 360 nondiabetic individuals with elevated risk for T2D (Tuebingen Family Study [TUEF]), and 64 patients undergoing pancreatectomy (Pancreas Biobank [PB], HbA1c <9%, no insulin therapy). Main Outcome Measures Insulin secretion calculated from 5-point oral glucose tolerance test (TUEF) and fasting blood collection before surgery (PB). A genome-wide polygenic score for T2D was computed from 484,788 genotyped variants. The interaction of magnetic resonance imaging-measured and histologically quantified pancreatic fat with the polygenic score was investigated. Partitioned risk scores using genome-wide significant variants were also computed to gain insight into potential mechanisms. Results Pancreatic steatosis interacted with genome-wide polygenic score on insulin secretion (P = 0.003), which was similar in the replication cohort with histological measurements (P = 0.03). There was a negative association between pancreatic fat and insulin secretion in participants with high genetic risk, whereas individuals with low genetic risk showed a positive correlation between pancreatic fat and insulin secretion. Consistent interactions were found with insulin resistance-specific and a liver/lipid-specific polygenic scores. Conclusions The associations suggest that pancreatic steatosis only impairs beta-cell function in subjects at high genetic risk for diabetes. Genetically determined insulin resistance specifically renders pancreatic fat deleterious for insulin secretion.