The origins and evolution of higher cognitive functions including complex forms of learning, attention and executive functions are unknown. A potential mechanism driving the evolution of vertebrate cognition early in the vertebrate lineage (550 My ago) was genome duplication and subsequent diversification of postsynaptic genes. Here we report the first genetic analysis of a vertebrate gene family in cognitive functions measured using computerized touchscreens. Comparison of mice carrying mutations in all four Dlg paralogs show simple associative learning required Dlg4, while Dlg2 and Dlg3 diversified to play opposing roles in complex cognitive processes. Exploiting the translational utility of touchscreens in humans and mice, testing Dlg2 mutations in both species showed Dlg2’s role in complex learning, cognitive flexibility and attention has been highly conserved over 100 My. Dlg family mutations underlie psychiatric disorders suggesting genome evolution expanded the complexity of vertebrate cognition at the cost of susceptibility to mental illness.