It's challenging work to identify disease-causing genes from the next-generation sequencing (NGS) data of patients with Mendelian disorders. To improve this situation, researchers have developed many phenotype-driven gene prioritization methods using a patient's genotype and phenotype information, or phenotype information only as input to rank the candidate's pathogenic genes. Evaluations of these ranking methods provide practitioners with convenience for choosing an appropriate tool for their workflows, but retrospective benchmarks are underpowered to provide statistically significant results in their attempt to differentiate. In this research, the performance of ten recognized causal-gene prioritization methods was benchmarked using 305 cases from the Deciphering Developmental Disorders (DDD) project and 209 in-house cases via a relatively unbiased methodology. The evaluation results show that methods using Human Phenotype Ontology (HPO) terms and Variant Call Format (VCF) files as input achieved better overall performance than those using phenotypic data alone. Besides, LIRICAL and AMELIE, two of the best methods in our benchmark experiments, complement each other in cases with the causal genes ranked highly, suggesting a possible integrative approach to further enhance the diagnostic efficiency. Our benchmarking provides valuable reference information to the computer-assisted rapid diagnosis in Mendelian diseases and sheds some light on the potential direction of future improvement on disease-causing gene prioritization methods.

Evaluation of phenotype-driven gene prioritization approaches for Mendelian diseases could facilitate the software development and method selection for the workflow configuration and clinical practice. In our original article, the performance of 10 well-recognized causal-gene prioritization methods was benchmarked using 305 cases from the deciphering developmental disorders (DDD) project and 209 in-house cases via a relatively unbiased methodology. The evaluation results showed that LIRICAL and AMELIE were two of the best methods in our benchmark experiments, and the possible integrative approach of these two methods may enhance the diagnostic efficiency. However, some methodological critiques were raised by the authors of Exomiser and PhenIX, so we revisited our benchmarking studies to answer their comments in this letter.

The authors regret that the published work cited in reference 14 that benchmarked the cell type prediction methods based on machine learning models for scRNA-seq datasets was not properly acknowledged in this work. In the Abstract section, “Single-cell transcriptomics is rapidly advancing our understanding of the cellular composition of complex tissues and organisms. A major limitation in most analysis pipelines is the reliance on manual annotations to determine cell identities, which is time-consuming, irreproducible, and sometimes lack canonical markers for certain cell types” should read: “Single-cell transcriptomics technologies have vast potential in advancing our understanding of cellular heterogeneity in complex tissues. While methods to interpret single-cell transcriptomics data are developing rapidly, challenges in most analysis pipeline still remain, and the major limitation is a reliance on manual annotations for cell-type identification that is time-consuming, irreproducible, and sometimes lack canonical markers for certain cell types” In the Introduction section, “The manual annotations, which are often not based on standardized ontologies of cell labels, are not reproducible across different experiments within and across research groups. These caveats become even more pronounced as the number of cells and samples increases, preventing fast and reproducible annotations” should read: “The manual annotations of cell types usually suffer from limited standardized ontologies and reproducibility. And the replicability of manual strategies for cell-type identification could be further reduced with an increased number of cells, clusters, and multiple selections of cluster marker genes” “Machine learning-based scRNA-seq classification methods predict the identity of each cell by learning these identities from annotated training data (e.g., a reference atlas). In contrast to the extensive evaluations of clustering, differential expression and trajectory inference methods, there is currently one single attempt comparing methods to assign cell-type labels to cell clusters. The lack of a comprehensive comparison of scRNA-seq classification models across different machine learning algorithms leaves users without indications as to which classification method best fits their problem. More importantly, a proper assessment of the classic machine learning approaches in comparison with the baseline methods can greatly benefit new developments in the field and prevent unnecessary complexity” should read: “Machine learning-based scRNAseq cell type identification models predict the phenotype of the cells by using the large and well-annotated reference cell atlas as the training sets. Because of the limited number of comprehensive benchmarking work on the cell type classification methods based on different machine learning algorithms for scRNAseq data leaves users without indications as to which automated prediction model could achieve optimal performance[14]” “Here, we benchmarked 10 classic machine learning models to automatically assign cell identities, that were evaluated by using 20 publicly available single-cell RNA sequencing datasets of different sizes, technologies, species, and complexity. The performance of the methods was evaluated based on their classification accuracy and computation time. We performed several experiments to cover different levels of challenge in the classification task and to test specific features or tasks. We evaluated the classification performance through two experimental setups: (1) intra-dataset in which we applied 5-fold cross-validation within each dataset and (2) inter-dataset involving across datasets comparisons. The inter-dataset comparison is more realistic and more practical, where a reference dataset (e.g., atlas) is used to train a classifier which can then be applied to identify cells in new unannotated datasets. In general, most classifiers perform well across all datasets in both experimental setups (inter- and intra-dataset), while large variation across different methods in the computation time and classification performance was observed in response to changing the input features and the complexity of the datasets” should read: “Here, we benchmarked 10 classic machine learning models to automatically assign cell identities that were evaluated based on the classification accuracy and computation time by using 20 publicly accessible scRNA-seq datasets with different technologies, species, data sizes, and complexity. Two experimental designs were employed to measure the prediction accuracy: (1) benchmarking each machine learning model for intra-dataset (within the dataset), in which the 5-fold cross-validation was applied; and (2) benchmarking each machine learning model for inter-dataset (across different datasets), the scenario in which is more realistic and practical. On the whole, most classifiers perform well in both the intra-dataset and inter-dataset setups, while the significant difference for machine learning models in the classification performance and computation time was observed in response to different numbers of input features and the complexity of the datasets” In the Materials and Methods section, “A total of 6 groups (PBMC, Pancreas, Immune cells within the tumor microenvironment, Tumor-Infiltrating T cells, Allen Mouse Brain, and Tabula Muris) consist of 20 scRNA-seq datasets (Table 1) were used to evaluate and benchmark all machine learning-based classification methods, from which all the datasets were used for intra-dataset evaluation using a cross-validation scheme, and 16 datasets were used for inter-dataset evaluation” should read: “A total of 6 scRNAseq dataset groups (PBMC, Pancreas, Immune cells within the tumor microenvironment, Tumor-Infiltrating T cells, Allen Mouse Brain, and Tabula Muris) consist of 20 individual datasets (Table 1) were employed to benchmark the classification performance of the machine learning-based prediction models. All the datasets were used for intra-dataset benchmark design, and 16 of them were used for inter-dataset classification (Table 1)” “The performance of intra-dataset classification experiments for each machine learning-based models was evaluated by applying 5-fold cross-validation (repeat 10 times for mean value) across each dataset after genes, cells, and small cell populations filtering. The folds were divided in a stratified manner to keep equal proportions of each cell population in each fold. The training and testing folds were the same for all classifiers” should read: “The performance of intra-dataset classification experiments for each machine learning-based models was evaluated by applying 5-fold cross-validation (repeat 10 times for mean value). The folds were stratified to balance the proportions of each cell cluster in each fold, and the classification was performed on the same folds for all the classifiers” In the Results section, “A major aspect affecting the classification performance is the complexity of the dataset. We defined the complexity of each dataset in terms of the pairwise similarity between cell clusters (see the “Methods” section) and compared the complexity to the performance of the classifiers and the number of cell clusters in a dataset” should read: “Previous studies have shown the complexity and variability of the dataset could affect the performance of automatic cell type classification[14]. In this study, we also investigate the effect of data complexity (measure by the pairwise similarity across cell clusters) of the benchmarking datasets on the prediction accuracy of machine learning models” “The performance of all classifiers is relatively low on the TIT dataset, which can be explained by the high pairwise correlations between the mean expression profiles of each cell population” should read: “Because of high pairwise correlations among the overall expression profiles of different infiltrating T cell clusters in the tumor microenvironment, the classification models show general low-level performance on all the TIT datasets” In the Discussion section, “In this study, we evaluated the performance of 10 different machine learning models for automatic cell identification by using 20 scRNAseq datasets. We performed several experiments to cover different levels of challenges in the classification task and to test specific aspects of the classifiers. Our evaluation results could be used as a reference for bioinformatics or experimental researchers to choose the most appropriate machine learning-based classifier models depending on the dataset setup at hand. The relatively lower performance was observed for all classification models for the inter-dataset experiments, likely due to the technical and biological differences between the datasets, when compared with the intra-dataset setup. Furthermore, the tested models vary considerably in their computation time” should read: “In this study, we benchmarked the cell type classification performance of classic machine learning models (N=10) based on different scRNA-seq datasets (N=20). The various experimental designs were applied to cover different levels of challenges in the automatic prediction task, and to test the specific aspects of machine learning-based models. Our evaluation results could be used as a reference for bioinformatics or experimental researchers to choose the most appropriate machine learning-based classifier models depending on the dataset setup at hand.” “which calculate the similarity between query cells and one representative gene expression per cell type in reference data sets. This type of tool usually has a preset model with tunable parameters, and the neighbors could either be on the cell level or the cluster level. The other is developed with supervised learning algorithms, that predict cell type labels of a new data set based on a reference data set” should read: “that calculate the similarity between query cells and representative cell type in reference data sets. The other tools are developed by applying machine learning approaches to predict cell identifies of query dataset on the basis of a well-annotated reference dataset” “It is still an open question how cell cluster annotation performance will be affected when using these eventual comprehensive cell-type gene expression marker set databases, as we observed that many methods are highly sensitive in precision-recall analysis when used with larger cell type marker gene set databases that contain additional cell types not represented in a given scRNA-seq dataset. Future work will need to study confusion matrices of all methods and better quantify precision scores” should read: “It is still an open question how the comprehensive reference atlas with annotated cell-type as the training dataset would affect automatic cell identifies prediction, and several studies have revealed that many methods are highly sensitive in precision-recall analysis when used with the larger cell type marker gene set databases that contain additional cell types[14,20]”

Based on clinical outcomes in colorectal cancer, high microsatellite instability (MSI-H) has recently been approved by the Food and Drug Administration (FDA) as a genetic test to select patients for immunotherapy targeting PD-1 and/or CTLA-4 without limitation to cancer type. However, it is unclear whether the MSI-H would broadly alter the tumor microenvironment to confer the therapeutic response of different cancer types to immunotherapy. To fill in this gap, we performed an in silico analysis of tumor immunity among different MSI statuses in five cancer types. We found that consistent with clinical responses to immunotherapy, MSI-H and non-MSI-H samples from colorectal cancer (COAD-READ) exhibited distinct infiltration levels and immune phenotypes. Surprisingly, the immunological difference between MSI-H and non-MSI-H samples was diminished in stomach adenocarcinoma and esophageal carcinoma (STAD-ESCA) and completely disappeared in uterine corpus endometrial carcinoma (UCEC). Regardless of cancer types, the abundance of tumor-infiltrating immune cells, rather than MSI status, strongly associated with the clinical outcome. Since preexisting antitumor immune response in the tumor (hot cancer) is accepted as a prerequisite to the therapeutic response to anti-PD-1/CTLA-4 immunotherapy, our data demonstrate that the impact of MSI varied on immune contexture will lead to the further evaluation of predictive immunotherapy responsiveness based on the universal biomarker of MSI status.

UNLABELLED: We here present BioCircos.js, an interactive and lightweight JavaScript library especially for biological data interactive visualization. BioCircos.js facilitates the development of web-based applications for circular visualization of various biological data, such as genomic features, genetic variations, gene expression and biomolecular interactions. AVAILABILITY AND IMPLEMENTATION: BioCircos.js and its manual are freely available online at http://bioinfo.ibp.ac.cn/biocircos/ CONTACT: [email protected] SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.