Postzygotic isolation evolves due to an accumulation of substitutions (potentially deleterious alleles in hybrids) in populations that have become geographically isolated. These potentially deleterious alleles might also be maintained in ancestral populations before geographic isolation. We used an individual-based model to examine the effect of the genetic state of an ancestral population on the evolution of postzygotic isolation after geographic isolation of a population. The results showed that the number of loci at which degenerative alleles are fixed in an ancestral population at equilibrium significantly affects the evolutionary rates of postzygotic isolation between descendant allopatric populations. Our results suggest that: (1) a severe decrease in population size (e.g., less than ten individuals) is not necessary for the rapid evolution of postzygotic isolation (e.g., <10,000 generation); (2) rapid speciation can occur when there is a large difference in the equilibrium number of accumulated degenerative alleles between ancestral and descendant populations; and (3) in an ancestral population maintained at a small effective population size for a long period of time, postzygotic isolation rarely evolves if back mutations that restore the function of degenerative alleles are limited.