Abstract
Mapping quantitative trait loci (QTL) for complex binary traits is more challenging than for normally distributed traits due to the nonlinear relationship between the observed phenotype and unobservable genetic effects, especially when the mapping population contains multiple outbred families. Because the number of alleles of a QTL depends on the number of founders in an outbred population, it is more appropriate to treat the effect of each allele as a random variable so that a single variance rather than individual allelic effects is estimated and tested. Such a method is called the random model approach. In this study, we develop the random model approach of QTL mapping for binary traits in outbred populations. An EM-algorithm with a Fisher-scoring algorithm embedded in each E-step is adopted here to estimate the genetic variances. A simple Monte Carlo integration technique is used here to calculate the likelihood-ratio test statistic. For the first time we show that QTL of complex binary traits in an outbred population can be scanned along a chromosome for their positions, estimated for their explained variances, and tested for their statistical significance. Application of the method is illustrated using a set of simulated data.