Abstract
The effect of selection on patterns of genetic structure within and between populations may be studied by contrasting observed patterns at the genes targeted by selection with those of unlinked neutral marker loci. Local directional selection on target genes will produce stronger population genetic structure than at neutral loci, whereas the reverse is expected for balancing selection. However, theoretical predictions on the intensity of this signal under precise models of balancing selection are still lacking. Using negative frequency-dependent selection acting on self-incompatibility systems in plants as a model of balancing selection, we investigated the effect of such selection on patterns of spatial genetic structure within a continuous population. Using numerical simulations, we tested the effect of the type of self-incompatibility system, the number of alleles at the self-incompatibility locus and the dominance interactions among them, the extent of gene dispersal, and the immigration rate on spatial genetic structure at the selected locus and at unlinked neutral loci. We confirm that frequency-dependent selection is expected to reduce the extent of spatial genetic structure as compared to neutral loci, particularly in situations with low number of alleles at the self-incompatibility locus, high frequency of codominant interactions among alleles, restricted gene dispersal and restricted immigration from outside populations. Hence the signature of selection on spatial genetic structure is expected to vary across species and populations, and we show that empirical data from the literature as well as data reported here on three natural populations of the herb Arabidopsis halleri confirm these theoretical results.