Abstract
Existing cell type annotation methods face significant hurdles: supervised approaches often fail to differentiate between novel cell types not present in reference data, while unsupervised techniques can suffer from cluster impurity and difficulties in robustly distinguishing multiple distinct unknown cell populations. This critical gap motivated the development of HiCat, a semi-supervised pipeline specifically designed to overcome these limitations. HiCat is a semi-supervised pipeline that integrates both approaches, leveraging reference (labeled) and query (unlabeled) genomic data to simultaneously enhance annotation accuracy for known cell types and improve the discovery and differentiation of novel ones. HiCat follows a structured pipeline: (1) removing batch effects and generate a low-dimensional embedding; (2) nonlinear dimensionality reduction for capturing key patterns; (3) unsupervised clustering for proposing novel cell type candidates; (4) merging multi-resolution features from previous steps into a condensed feature space; (5) training a classifier on reference data for supervised annotation; and (6) resolving inconsistencies between supervised predictions and unsupervised clusters to finalize annotations, particularly for unseen types. Performance was evaluated across 10 public genomic datasets and perform a case study on a molecular cell atlas of the human lung. HiCat demonstrated superior performance in both known cell type classification and novel cell type identification. In benchmark evaluations, HiCat consistently outperformed existing methods, critically excelling in identifying and distinguishing multiple novel cell types. HiCat presents a robust framework for scRNA-seq cell annotation, improving classification accuracy and novel type identification. In addition, it provides a scalable and transferable solution for biomedical research, directly addressing key challenges in automated cell annotation.