West Nile virus (WNV), like the dengue virus (DENV) and yellow fever virus (YFV), are major arboviruses belonging to the Flavivirus genus. WNV is emerging or endemic in many countries around the world, affecting humans and other vertebrates. Since 1999, it has been considered to be a major public and veterinary health problem, causing diverse pathologies, ranging from a mild febrile state to severe neurological damage and death. WNV is transmitted in a bird–mosquito–bird cycle, and can occasionally infect humans and horses, both highly susceptible to the virus but considered dead-end hosts. Many studies have investigated the molecular determinants of WNV virulence, mainly with the ultimate objective of guiding vaccine development. Several vaccines are used in horses in different parts of the world, but there are no licensed WNV vaccines for humans, suggesting the need for greater understanding of the molecular determinants of virulence and antigenicity in different hosts. Owing to technical and economic considerations, WNV virulence factors have essentially been studied in rodent models, and the results cannot always be transported to mosquito vectors or to avian hosts. In this review, the known molecular determinants of WNV virulence, according to invertebrate (mosquitoes) or vertebrate hosts (mammalian and avian), are presented and discussed. This overview will highlight the differences and similarities found between WNV hosts and models, to provide a foundation for the prediction and anticipation of WNV re-emergence and its risk of global spread.