Abstract
Spatial transcriptomics enables high-resolution gene expression measurements while preserving the two-dimensional spatial organization of the biological sample. A common objective in spatial transcriptomics data analysis is to identify spatially variable genes within predefined cell types or regions within the tissue. However, these regions are often implicitly one-dimensional, making standard two-dimensional coordinate-based methods less effective as they overlook the underlying tissue organization. Here we introduce a methodology grounded in spectral graph theory to elucidate a one-dimensional curve that effectively approximates the spatial coordinates of the examined sample. This curve is then used to establish a new coordinate system that reflects tissue morphology. We then develop a generalized additive model (GAM) to estimate spatial patterns which permits the detection of genes with variable expression in the new morphologically relevant coordinate system. Our approach directly models gene counts, thereby eliminating the need for normalization or transformations to satisfy normality assumptions. A second important advantage over existing hypothesis-testing approaches is that our method not only improves performance but also accurately estimates gene expression patterns and precisely pinpoints spatial loci where deviations from constant expression occur. We validate our approach through extensive simulation and by analyzing experimental data from multiple platforms such as Slide-seq and MERFISH. As an example of its ability to enable biological discovery, we demonstrate how our methodology enables the identification of novel interferon-related subpopulations in the mouse mucosa, as well as markers of inflammation-associated fibroblasts in a multi-sample spatial transcriptomic dataset.