Abstract
We describe a new method of estimating the selfing rate (S) in a mixed mating population based on a population structure approach that accounts for possible intergenerational correlation in selfing rate, giving rise to an estimate of the upper limit for heritability of selfing rate (h2). A correlation between generations in selfing rate is shown to affect one- and two-locus probabilities of identity by descent. Conventional estimates of selfing rate based on a population structure approach are positively biased by intergenerational correlation in selfing. Multilocus genotypes of individuals are used to give maximum-likelihood estimates of S and h2 in the presence of scoring artifacts. Our multilocus estimation of selfing rate and its heritability (MESH) method was tested with simulated data for a range of conditions. Selfing rate estimates from MESH have low bias and root mean squared error, whereas estimates of the heritability of selfing rate have more uncertainty. Increasing the number of individuals in a sample helps to reduce bias and root mean squared error more than increasing the number of loci of sampled individuals. Improved estimates of selfing rate, as well as estimates of its heritability, can be obtained with this method, although a large number of loci and individuals are needed to achieve best results.