Spatial transcriptomic technologies allow researchers to explore the diversity and specificity of gene expression within an original tissue structure. Accurately identifying regions that are spatially coherent in both gene expression and physical tissue structures is an emerging topic; however, it is challenging because of the lack of ground truth labels which complicates the validation of clustering consistency and reproducibility. This highlights a need for a computational evaluation framework to rigorously and unbiasedly assess the clustering performance. To address this gap, we propose Spatial Transcriptomics Evaluation Algorithm and Metric (STEAM), a user-friendly computational pipeline to evaluate the consistency and reliability of clustering results by leveraging machine learning classification and prediction methods, with the goal of maintaining spatial proximity and gene expression patterns within clusters. In addition, it enables iterative correction of misclassified cells, providing actionable guidance for cluster refinement. We benchmarked STEAM on various public datasets, spanning multicell to single-cell resolution, normal and diseased tissues, as well as spatial transcriptomics and proteomics. The results highlighted its robustness and generalizability through comprehensive statistical evaluation metrics, such as Kappa score, F1 score, accuracy, adjusted rand index, normalized mutual information, and percentage of abnormal spots. Notably, STEAM supports multisample training, enabling cross-replicate clustering consistency assessments. Moreover, STEAM provides practical guidance by comparing clustering results across multiple approaches; here, we evaluated four representative methods encompassing both spatial-aware and spatial-ignorant strategies. In summary, STEAM is a promising tool for evaluating clustering robustness and benchmarking clustering performance for spatial omics data, offering valuable insights to drive reproducible discoveries in spatial biology.

BACKGROUND: A vital step in analyzing single-cell data is ascertaining which cell types are present in a dataset, and at what abundance. In many diseases, the proportions of varying cell types can have important implications for health and prognosis. Most approaches for cell type annotation have centered around cell typing for single-cell RNA-sequencing (scRNA-seq) and have had promising success. However, reliable methods are lacking for many other single-cell modalities such as single-cell sequencing assay for transposase-accessible chromatin (scATAC-seq), which quantifies the extent to which genes of interest in each cell are epigenetically "open" for expression. RESULTS: To leverage the informative potential of scATAC-seq data, we developed CAMML with the integration of chromatin accessibility (CAraCAl), a bioinformatic method that performs cell typing on scATAC-seq data. CAraCAl performs cell typing by scoring each cell for its enrichment of cell type-specific gene sets. These gene sets are composed of the most upregulated or downregulated genes present in each cell type according to projected gene activity. CONCLUSIONS: We found that CAraCAl does not improve performance beyond CAMML when scRNA-seq is present, but if only scATAC-seq is available, CAraCAl performs cell typing relatively successfully. As such, we also discuss best practices for cell typing and the strengths and weaknesses of various cell annotation options.

MOTIVATION: Cell typing is a critical task in the analysis of single-cell data, particularly when studying complex diseased tissues. Unfortunately, the sparsity and noise of single-cell data make accurate cell typing of individual cells difficult. To address these challenges, we previously developed the CAMML method for multi-label cell typing of single-cell RNA-sequencing (scRNA-seq) data. CAMML uses weighted gene sets to score each profiled cell for multiple potential cell types. While CAMML outperforms other scRNA-seq cell typing techniques, it only leverages transcriptomic data so cannot take advantage of newer multi-omic single-cell assays that jointly profile gene expression and protein abundance (e.g. joint scRNA-seq/CITE-seq). RESULTS: We developed the CAMML with the Integration of Marker Proteins (ChIMP) method to support multi-label cell typing of individual cells jointly profiled via scRNA-seq and CITE-seq. ChIMP combines cell type scores computed on scRNA-seq data via the CAMML approach with discretized CITE-seq measurements for cell type marker proteins. The multi-omic cell type scores generated by ChIMP allow researchers to more precisely and conservatively cell type joint scRNA-seq/CITE-seq data. AVAILABILITY AND IMPLEMENTATION: An implementation of this work is available on CRAN at https://cran.r-project.org/web/packages/CAMML/. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.