Abstract
In a quantitative trait locus (QTL) study, it is usually not feasible to select families with offspring that simultaneously display variability in more than one phenotype. When multiple phenotypes are of interest, the sample will, with high probability, contain 'non-segregating' families, i.e. families with both parents homozygous at the QTL. These families potentially reduce the power of regression-based methods to detect linkage. Moreover, follow-up studies in individual families will be inefficient, and potentially even misleading, if non-segregating families are selected for the study. Our work extends Haseman-Elston regression using a latent class model to account for the mixture of segregating and non-segregating families. We provide theoretical motivation for the method using an additive genetic model with two distinct functions of the phenotypic outcome, squared difference (SqD) and mean-corrected product (MCP). A permutation procedure is developed to test for linkage; simulation shows that the test is valid for both phenotypic functions. For rare alleles, the method provides increased power compared to a 'marginal' approach that ignores the two types of families; for more common alleles, the marginal approach has better power. These results appear to reflect the ability of the algorithm to accurately assign families to the two classes and the relative weights of segregating and non-segregating families to the test of linkage. An application of Bayes rule is used to estimate the family-specific probability of segregating. High predictive value positive values for segregating families, particularly for MCP, suggest that the method has considerable value for identifying segregating families. The method is illustrated for gene expression phenotypes measured on 27 candidate genes previously demonstrated to show linkage in a sample of 14 families.