Abstract
The evolution of reproductive barriers leads to the formation of new species. However, recent research has demonstrated that hybridization has been pervasive across the tree of life even in the presence of strong barriers. Using swordtail fishes (genus Xiphophorus), an emerging model system, we document overlapping mechanisms that act as barriers to gene flow between Xiphophorus birchmanni and Xiphophorus cortezi by combining genomic sequencing from natural hybrid populations, experimental laboratory crosses, behavioural assays, sperm measures and developmental studies. We show that assortative mating plays a role in maintaining subpopulations with distinct ancestry within natural hybrid populations. Using F(2) hybrids we identify several genomic regions that strongly impact hybrid viability. Strikingly, two of these regions underlie genetic incompatibilities in hybrids between X. birchmanni and its sister species Xiphophorus malinche. Our results demonstrate that ancient hybridization has played a role in the origin of this shared genetic incompatibility. Moreover, ancestry mismatch at these incompatible regions has remarkably similar consequences for phenotypes and hybrid survival in X. cortezi × X. birchmanni hybrids as in X. malinche × X. birchmanni hybrids. Our findings identify varied reproductive barriers that shape genetic exchange between naturally hybridizing species and highlight the complex evolutionary outcomes of hybridization.