Decoding the linguistic intricacies of the genome is a crucial problem in biology, and pre-trained foundational models such as DNABERT and Nucleotide Transformer have made significant strides in this area. Existing works have largely hinged on k-mer, fixed-length permutations of A, T, C, and G, as the token of the genome language due to its simplicity. However, we argue that the computation and sample inefficiencies introduced by k-mer tokenization are primary obstacles in developing large genome foundational models. We provide conceptual and empirical insights into genome tokenization, building on which we propose to replace k-mer tokenization with Byte Pair Encoding (BPE), a statistics-based data compression algorithm that constructs tokens by iteratively merging the most frequent co-occurring genome segment in the corpus. We demonstrate that BPE not only overcomes the limitations of k-mer tokenization but also benefits from the computational efficiency of non-overlapping tokenization. Based on these insights, we introduce DNABERT-2, a refined genome foundation model that adapts an efficient tokenizer and employs multiple strategies to overcome input length constraints, reduce time and memory expenditure, and enhance model capability. Furthermore, we identify the absence of a comprehensive and standardized benchmark for genome understanding as another significant impediment to fair comparative analysis. In response, we propose the Genome Understanding Evaluation (GUE), a comprehensive multi-species genome classification dataset that amalgamates $36$ distinct datasets across $9$ tasks, with input lengths ranging from $70$ to $10000$. Through comprehensive experiments on the GUE benchmark, we demonstrate that DNABERT-2 achieves comparable performance to the state-of-the-art model with $21 \times$ fewer parameters and approximately $92 \times$ less GPU time in pre-training. Compared to DNABERT, while being $3 \times$ more efficient, DNABERT-2 outperforms it on $23$ out of $28$ datasets, with an average improvement of $6$ absolute scores on GUE. The code, data, and pre-trained model are available at \url{https://github.com/MAGICS-LAB/DNABERT_2}.

MOTIVATION: Although human tissues carry out common molecular processes, gene expression patterns can distinguish different tissues. Traditional informatics methods, primarily at the gene level, overlook the complexity of alternative transcript variants and protein isoforms produced by most genes, changes in which are linked to disease prognosis and drug resistance. RESULTS: We developed TransTEx (Transcript-level Tissue Expression), a novel tissue-specificity scoring method, for grouping transcripts into four expression groups. TransTEx applies sequential cut-offs to tissue-wise transcript probability estimates, subsampling-based P-values and fold-change estimates. Application of TransTEx on GTEx mRNA-seq data divided 199 166 human transcripts into different groups as 17 999 tissue-specific (TSp), 7436 tissue-enhanced, 36 783 widely expressed (Wide), 79 191 lowly expressed (Low), and 57 757 no expression (Null) transcripts. Testis has the most (13 466) TSp isoforms followed by liver (890), brain (701), pituitary (435), and muscle (420). We found that the tissue specificity of alternative transcripts of a gene is predominantly influenced by alternate promoter usage. By overlapping brain-specific transcripts with the cell-type gene-markers in scBrainMap database, we found that 63% of the brain-specific transcripts were enriched in nonneuronal cell types, predominantly astrocytes followed by endothelial cells and oligodendrocytes. In addition, we found 61 brain cell-type marker genes encoding a total of 176 alternative transcripts as brain-specific and 22 alternative transcripts as testis-specific, highlighting the complex TSp and cell-type specific gene regulation and expression at isoform-level. TransTEx can be adopted to the analysis of bulk RNA-seq or scRNA-seq datasets to find tissue- and/or cell-type specific isoform-level gene markers. AVAILABILITY AND IMPLEMENTATION: TransTEx database: https://bmi.cewit.stonybrook.edu/transtexdb/ and the R package is available via GitHub: https://github.com/pallavisurana1/TransTEx.

An in-depth exploration of gene prognosis using different methodologies aids in understanding various biological regulations of genes in disease pathobiology and molecular functions. Interpreting gene functions at biological and molecular levels remains a daunting yet crucial task in domains such as drug design, personalized medicine, and next-generation diagnostics. Recent advancements in omics technologies have produced diverse heterogeneous genomic datasets like micro-array gene expression, miRNA expression, DNA sequence, 3D structures, which are significant resources for understanding the gene functions. In this paper, we propose a novel self-attention based deep multi-modal model, named DeePROG, for the prognosis of disease affected genes based on heterogeneous omics data. We use three NCBI datasets covering three modalities, namely gene expression profile, the underlying DNA sequence, and the 3D protein structures. To extract useful features from each modality, we develop several context-specific deep learning models. Besides, we develop three attention-based deep bi-modal architectures along with DeePROG to leverage the prognosis of the underlying biomedical data. We assess the performance of the models' in terms of computational assessment of function annotation (CAFA2) metrics. Moreover, we analyze the results in terms of receiver operating characteristics (ROC) curve in high-class imbalance data setting and perform statistical significance tests in terms of Welch's t-test. Experiment results show that DeePROG significantly outperforms baseline models across in terms of performance metrics. The source code and all preprocessed datasets used in this study are available at https://github.com/duttaprat/DeePROG.

The pandemic disease COVID-19, originated from the SARS-CoV-2 virus has spread globally. Researchers are working tirelessly on areas including studying the transmission of COVID-19, promoting its identification, designing new vaccines and therapies, and recognizing its socio-economic consequences. This extensive research leads to the exploration of thousands of scientific papers related to biology, chemistry, genetics, health, and economy. Therefore, it is essential to develop an intelligent text mining technique for segregating this rich source of data to perform easy access, information retrieval, and interpretation within minimum time and resources. We propose a multi-objective optimization-based document clustering approach for the CORD-19 (COVID-19 Open Research Dataset) dataset in this paper. Here, a new technique utilizing BioBERT has been proposed, which benefits from the abstract and the document text, rather than only the brief abstract, to perceive a concise understanding of the text to generate clusters with better definitions. The main contributions of the proposed work are two-fold: in the first step, we have used BioBERT to generate the sentence embedding which is further used for the document representation. In the next step, we have developed a multi-objective optimization (MOO) based clustering algorithm for grouping the generated document vector representations. In this MOO-based clustering, we have used Non-dominated Sorting Genetic Algorithm-II and Fuzzy c-means algorithm as the underlying MOO and clustering technique, respectively. This model is evaluated using the Silhouette Score (Silhouette score) and Calinski-Harabasz index (CH index), and the clustering solutions are visualized using word clouds. The clustering results exhibit significant improvements over various other existing clustering models.

Protein is an essential macro-nutrient for perceiving a wide range of biochemical activities and biological regulations in living cells. In this work, we have presented a novel multi-modal approach, named MultiPredGO, for predicting protein functions by utilizing two different kinds of information, namely protein sequence and the protein secondary structure. Here, our contributions are threefold; firstly, along with the protein sequence, we learn the feature representation from the protein structure. Secondly, we develop two different deep learning models after considering the characteristics of the underlying data patterns of the protein sequence and protein 3D structures. Finally, along with these two modalities, we have also utilized protein interaction information for expediting the efficiency of the proposed model in predicting the protein functions. For extracting features from different modalities, we have utilized various variations of the convolutional neural network. As the protein function classes are dependent on each other, we have used a neuro-symbolic hierarchical classification model, which resembles the structure of Gene Ontology (GO), for effectively predicting the dependent protein functions. Finally, to validate the goodness of our proposed method (MultiPredGO), we have compared our results with various uni-modal along with two well-known multi-modal protein function prediction approaches, namely, INGA and DeepGO. Results show that the overall performance of the proposed approach in terms of accuracy, F-measure, precision, and recall metrics are better than those by the state-of-the-art methods. MultiPredGO attains an average 13.05% and 30.87% improvements over the best existing comparing approach (DeepGO) for cellular component and molecular functions, respectively.

An in-depth exploration of protein-protein interactions (PPI) is essential to understand the metabolism in addition to the regulations of biological entities like proteins, carbohydrates, and many more.Most of the recent PPI tasks in BioNLP domain have been carried out solely using textual data.In this paper, we argue that incorporation of multimodal cues can improve the automatic identification of PPI.As a first step towards enabling the development of multimodal approaches for PPI identification, we have developed two multimodal datasets which are extensions and multimodal versions of two popular benchmark PPI corpora (BioInfer and HRPD50).Besides, existing textual modalities, two new modalities, 3D protein structure and underlying genomic sequence, are also added to each instance.Further, a novel deep multi-modal architecture is also implemented to efficiently predict the protein interactions from the developed datasets.A detailed experimental analysis reveals the superiority of the multi-modal approach in comparison to the strong baselines including uni-modal approaches and state-of the-art methods over both the generated multimodal datasets.The developed multi-modal datasets are available for use at https:// github.com/sduttap16/MM_PPI_NLP.

Cluster ensemble techniques aim to combine the outputs of multiple clustering algorithms to obtain a single consensus partitioning. The current paper reports about the development of a cluster ensemble based technique combining the concepts of multiobjective optimization and deep-learning models for gene clustering where some additional protein-protein interaction information are utilized for generating the consensus partitioning. The proposed ensemble based framework works in four phases: (i) filtering out the irrelevant genes from the microarray dataset: only the statistically significant genes are considered for further data analysis; (ii) generation of diverse base partitionings: a multi-objective optimization-based clustering technique is proposed which simultaneously optimizes three different cluster quality measures and generates a set of partitioning solutions on the Pareto optimal front; (iii) generation of a consensus partitioning: mentha scores, calculated by accessing a highly enriched protein-protein interaction archive named mentha, of different clustering solutions are considered for generating a weighted incidence matrix; (iv) finally, two approaches are used to generate a consensus partitioning from the obtained incidence matrix. The first approach is based on a traditional machine learning method, and another approach exploits the graph partitioning algorithm and two deep neural models to generate the final clustering. To validate the efficacy of the proposed ensemble framework, it is applied on five gene expression datasets. We present a comparative analysis of the proposed technique over different clustering algorithms in terms of biological homogeneity index (BHI) and biological stability index (BSI). The traditional approach attains an average 3 and 2 percent improvements over the best non-dominated solution with respect to BHI and BSI, respectively, whereas deep learning models illustrate an average 6.8 and 1.5 percent improvements over the proposed traditional approach with respect to BHI and BSI, respectively. Subsequently, Welch's t-test is executed to prove that the results obtained by the proposed methods are statistically significant. Availability of data and materials: https://github.com/sduttap16/DeepEnsm.

In the field of computational bioinformatics, for tasks such as medical diagnosis or disease gene classification, acquiring class labels is very much essential and costly. Now a days, labelling biomedical data became one of the crucial bottlenecks for developing supervised machine learning systems, as well as deep learning systems. On the contrary, weak supervision sources that generate diverse, semi-accurate or programmatically-generated labels are comparatively cheaper and less time consuming. In this paper, we have proposed a framework for integrating weak supervision sources with a generative model to estimate probabilistic class labels of the gene expression data. Here as the weak supervision source, a multi-objective optimization (MOO) based clustering technique is utilized. The non-dominated solutions of Pareto optimal front are viewed as the weak supervised labels. These weak supervised labels are fed to a generative model, which finally assigns probabilistic class labels to different genes. In this work, we have shown that our proposed generative model based clustering technique attains better Silhouette score (some quality measure of clustering) for three real-life NCBI gene expression data sets than other stateof-the-art methods, without using any labeled data. Finally, the superiority of the proposed method is validated by using a statistical and a biological significance test.

Classification of samples of gene expression profile plays a significant role in prediction and diagnosis of diseases. In the task of sample classification, a robust feature selection algorithm is very much essential to identify the important genes from the high dimensional gene expression data. This paper explores the information of protein-protein interaction with a graph mining technique for finding a proper subset of features (genes), which further takes part in sample classification. Here, our contribution for feature selection is three-fold: first, all the genes are grouped into different clusters based on the integrated information of the gene expression values and their protein interactions using a multi-objective optimization based clustering approach. Second, the confidence scores of the protein interactions are incorporated in a popular graph mining algorithm namely Goldberg algorithm to find out the relevant features. These features are the topologically and functionally significant genes, named as hub genes. Finally, these hub genes are identified varying the degrees of the nodes, and those are utilized for the sample classification task. Different machine learning classifiers are exploited for this purpose, and the classification performance is measured with respect to various performance metrics namely accuracy, sensitivity, specificity, precision, F-measure, and Mathews coefficient correlation. Comparative analysis with respect to two baselines and several existing approaches proves the efficiency of the proposed approach. Furthermore, the robustness of the identified hub-gene modules is endorsed using some strong biological significance analysis.