Giant viruses or nucleocytoplasmic large DNA viruses (NCLDVs) infect a wide range of eukaryotic hosts (including, algae, amoebae and metazoans) and show a very large range in genome size (between 100 kb and 1.2 Mb). Here we review some recent results concerning the extensive lateral gene transfer which appears to have occurred during NCLDV evolution. Current data suggest that giant viruses probably originated from a simple and ancient viral ancestor with a small subset of 30–35 genes encoding replication and structural proteins. A large array of lateral gene transfers from diverse cellular sources, including several families of mobile genetic elements, is probably responsible for the huge diversity of genome size and composition found in extant giant viruses.