Abstract
A long-standing goal in evolutionary biology is to identify the conditions that promote the evolution of reproductive isolation and speciation. The factors promoting sympatric speciation have been of particular interest, both because it is notoriously difficult to prove empirically and because theoretical models have generated conflicting results, depending on the assumptions made. Here, we analyze the conditions under which selection favors the evolution of assortative mating, thereby reducing gene flow between sympatric groups, using a general model of selection, which allows fitness to be frequency dependent. Our analytical results are based on a two-locus diploid model, with one locus altering the trait under selection and the other locus controlling the strength of assortment (a "one-allele" model). Examining both equilibrium and nonequilibrium scenarios, we demonstrate that whenever heterozygotes are less fit, on average, than homozygotes at the trait locus, indirect selection for assortative mating is generated. While costs of assortative mating hinder the evolution of reproductive isolation, they do not prevent it unless they are sufficiently great. Assortative mating that arises because individuals mate within groups (formed in time or space) is most conducive to the evolution of complete assortative mating from random mating. Assortative mating based on female preferences is more restrictive, because the resulting sexual selection can lead to loss of the trait polymorphism and cause the relative fitness of heterozygotes to rise above homozygotes, eliminating the force favoring assortment. When assortative mating is already prevalent, however, sexual selection can itself cause low heterozygous fitness, promoting the evolution of complete reproductive isolation (akin to "reinforcement") regardless of the form of natural selection.