Abstract Background Pancreatic neuroendocrine tumors (PANETs) are rare, slow growing cancers that often present with local and distant metastasis upon detection. PANETS contain distinct karyotypes, epigenetic dysregulation, and recurrent mutations in MEN1, ATRX, and DAXX (MAD+); however, the molecular basis of disease progression remains uncharacterized. Methods We evaluated associations between aneuploidy and the MAD+ mutational state of 532 PANETs from 11 published genomic studies and 19 new cases using a combination of exome, targeted panel, shallow WGS, or RNA-seq. We mapped the molecular timing of MAD+ PANET progression using cellular fractions corrected for inferred tumor content. Results In 287 PANETs with mutational data, MAD+ tumors always exhibited a highly recurrent signature of loss of heterozygosity (LOH) and copy-number alterations affecting 11 chromosomes, typically followed by genome doubling upon metastasis. These LOH chromosomes substantially overlap with those that undergo non-random mis-segregation due to ectopic CENP-A localization to flanking centromeric regions in DAXX-depleted cell lines. Using expression data from 122 PANETs, we found decreased gene expression in the regions immediately adjacent to the centromere in MAD+ PANETs. Using 43 PANETs from AACR GENIE, we inferred this signature to be preceded by mutations in MEN1, ATRX, and DAXX. We conducted a meta-analysis on 226 PANETs from 8 CGH studies to show an association of this signature with metastatic incidence. Our study shows that MAD+ tumors are a genetically diverse and aggressive subtype of PANETs that display extensive chromosomal loss after MAD+ mutation, which is followed by genome doubling. Conclusions We propose an evolutionary model for a subset of aggressive PANETs that is initiated by mutation of MEN1, ATRX, and DAXX, resulting in defects in centromere cohesion from ectopic CENP-A deposition that leads to selective loss of chromosomes and the LOH phenotype seen in late-stage metastatic PANETs. These insights aid in disease risk stratification and nominate potential therapeutic vulnerabilities to treat this disease.