Abstract
MOTIVATION: Clustering single-cell RNA sequencing (scRNA-seq) data plays a vital role in the study of cellular heterogeneity. Many algorithms have been developed to cluster scRNA-seq data. However, traditional clustering algorithms often fail to capture local consistency, whereas biclustering algorithms suffer from issues such as cell loss, poor adaptability to high-dimensional data, and iterative selection challenges. RESULTS: In this paper, we introduce scDBic, a novel deep learning-based biclustering algorithm specialized for scRNA-seq data. It comprises three main steps: cell clustering with a deep autoencoder, gene clustering, and identification of key gene clusters using the reverse strategy. The key idea is that the deep autoencoder captures the main information of gene expression and the reverse strategy identifies the key genes of cell groups. Therefore, cell clustering performance can be improved. The results demonstrate that our algorithm not only discovers cell groups in scRNA-seq data but also identifies the key genes of the cell groups. Furthermore, the clustering performance of our algorithm is better than that of traditional clustering and biclustering algorithms. This novel technique can be directly applied to discover cell groups and identify key genes in cell groups. AVAILABILITY AND IMPLEMENTATION: The source code and test data are freely available at GitHub (https://github.com/Xiaoqi-Tang/scDBic) and archived on Zenodo (DOI: 10.5281/zenodo.18676401).