Abstract
Single-cell RNA sequencing (scRNA-seq) has become a key technology for characterizing cell-type heterogeneity in complex tissues. However, its utility depends on accurate and reproducible cell-type annotation, which remains a major analytical challenge. Although hundreds of computational tools have been developed for automated annotation, there is currently no systematic framework to evaluate annotation robustness in a dataset-specific manner or within the context of complete analytical pipelines. Here, we present STEVE (Single-cell Transcriptomics Expression Visualization and Evaluation), a quantitative framework designed to assess the accuracy, robustness, and reproducibility of cell-type annotation in scRNA-seq studies. STEVE implements three complementary in silico evaluation modules: (i) Subsampling Evaluation to quantify annotation stability under varying reference sizes and data partitions; (ii) Novel Cell Evaluation to assess the ability to detect previously unseen cell types; and (iii) Annotation Benchmarking to compare alternative annotation tools against ground-truth labels. In addition, STEVE includes a Reference Transfer Annotation module that enables cross-dataset cell-type mapping using external reference datasets. All modules are built upon a unified probabilistic framework that provides consistent confidence estimation across evaluation scenarios. We evaluated STEVE across four independent scRNA-seq datasets with experimentally defined or expert-curated cell-type labels. Our results show that annotation robustness is strongly influenced by the annotation method, biological separability, dataset complexity, and batch effects. STEVE provides a practical framework for quantifying annotation uncertainty and improving reproducibility in single-cell transcriptomic analyses. STEVE is freely available at GitHub (https://github.com/XiaoDongLab/STEVE).