Embryonic stem (ES) cells are derived from blastocyst stage embryos and are believed to be functionally equivalent to the inner cell mass, which lacks the ability to produce all extraembryonic tissues. Here we report the identification of a rare transient cell population within mouse ES and induced pluripotent stem (iPS) cell cultures that express high levels of transcripts found in two-cell (2C) embryos in which the blastomeres are totipotent. We genetically tagged these 2C-like ES cells and show that they lack the ICM pluripotency proteins Oct4, Sox2, and Nanog and have acquired the ability to contribute to both embryonic and extraembryonic tissues. We show that nearly all ES cells cycle in and out of this privileged state, which we find is partially controlled by histone modifying enzymes. Transcriptome sequencing and bioinformatic analyses revealed that a significant number of 2C-transcripts are initiated from long terminal repeats derived from murine endogenous retroviruses, suggesting this foreign sequence has helped to drive cell fate regulation in placental mammals.