Genetic diversity provides the raw material for populations to respond to changing environmental conditions. The evolution of diversity within populations is based on the accumulation of mutations and their retention or loss through selection and genetic drift, while migration can also introduce new variation. However, the extent to which population growth and sustained large population size can lead to rapid and significant increases in diversity has not been widely investigated. Here, we assess this empirically by applying approximate Bayesian computation to a novel ancient DNA dataset that spans the life of a southern elephant seal (Mirounga leonina) population, from initial founding approximately 7000 years ago to eventual extinction within the past millennium. We find that rapid population growth and sustained large population size can explain substantial increases in population genetic diversity over a period of several hundred generations, subsequently lost when the population went to extinction. Results suggest that the impact of diversity introduced through migration was relatively minor. We thus demonstrate, by examining genetic diversity across the life of a population, that environmental change could generate the raw material for adaptive evolution over a very short evolutionary time scale through rapid establishment of a large, stable population.