Abstract
Estimating total narrow-sense heritability in admixed populations remains an open question. In this work, we used extensive simulations to evaluate existing linear mixed-model frameworks for estimating total narrow-sense heritability in two population-based cohorts from Greenland, and compared the results with data from unadmixed individuals from Denmark. When our analysis focused on Greenlandic sib pairs, and under the assumption that shared environment among siblings has a negligible effect, the model with two relationship matrices, one capturing identity by descent and one capturing identity by state, returned heritability estimates close to the true simulated value, while using each of the two matrices alone led to downward biases. When phenotypes correlated with ancestry, heritability estimates were inflated. Based on these observations, we propose a PCA-based adjustment that recovers the true simulated heritability. We use this knowledge to estimate the heritability of ten quantitative traits from the two Greenlandic cohorts, and report differences such as lower heritability for height in Greenlanders compared with Europeans. In conclusion, narrow-sense heritability in admixed populations is best estimated when using a mixture of genetic relationship matrices on individuals with at least one first-degree relative included in the sample.