RGB and infrared images has shown remarkable robustness for object detection based on unmanned aerial vehicles (UAV). However, the primitive RGB and infrared (IR) images are inevitably misaligned due to the device gap between RGB and infrared cameras. Most existing methods rely on manually filtered and aligned images, and thus are limited in real-world application. Some recent methods tend to directly learn from misaligned images, which only weakly benefit from the multi-modality and may be misled by dramatically misaligned IR images. Considering that the manually aligned images are available during training while unavailable in inference, we explore a new learning paradigm using the IR modality as privileged information. In the training stage, our model learns to hallucinate the complementary knowledge in IR modality based on RGB modality. In inference, our model could hallucinate the complementary IR modality to facilitate UAV detection. Specifically, we propose to quantize the IR features and hallucinate the codebook-indices based on RGB features, which is more effective and robust than directly hallucinating features. In addition, we propose to hierarchically hallucinate multi-scale codebook-indices, which could further improve the hallucinating quality. Experiments on DroneVehicle and VisDrone datasets demonstrate the effectiveness of our method.

Keypoint estimation is a fundamental task in computer vision, but generally requires large-scale annotated data for training. Few-shot and unsupervised keypoint estimation are prevalent economical paradigms, but the former still requires annotations for extensive novel classes while the latter only supports for single class. In this paper, we focus on the task of weak-shot keypoint estimation, where multiple novel classes are learned from unlabeled images with the help of labeled base classes. The key problem is what to transfer from base classes to novel classes, and we propose to transfer keyness and correspondence, which essentially belong to comparing entities and thus are class-agnostic and class-wise transferable. The keyness compares which pixel in the local region is more key, which can guide the keypoints of novel classes to move towards the local maximum (i.e., obtaining keypoints). The correspondence compares whether the two pixels belongs to the same semantic part, which can activate the keypoints of novel classes by reinforcing the consistency between corresponding points on two paired images. By transferring keyness and correspondence, our framework achieves favourable performance for weak-shot keypoint estimation. Extensive experiments and analyses on large-scale benchmark MP-100 demonstrate our effectiveness.

The prediction of binary protein-protein interactions (PPIs) is essential for protein engineering, but a major challenge in deep learning-based methods is the unknown decision-making process of the model. To address this challenge, we propose the ESM2_AMP framework, which utilizes the ESM2 protein language model for extracting segment features from actual amino acid sequences and integrates the Transformer model for feature fusion in binary PPIs prediction. Further, the two distinct models, ESM2_AMPS and ESM2_AMP_CSE are developed to systematically explore the contributions of segment features and combine with special tokens features in the decision-making process. The experimental results reveal that the model relying on segment features demonstrates strong correlations between segments with high attention weights and known functional regions of amino acid sequences. This insight suggests that attention to these segments helps capture biologically relevant functional and interaction-related information. By analyzing the coverage relationship between high-attention sequence fragments and functional regions, we validated the model's ability to capture key segment features of PPIs and revealed the critical role of functional domains in PPIs. This finding not only enhances the interpretability methods for sequence-based prediction models but also provides biological evidence supporting the important regulatory role of functional sequences in protein-protein interactions. It offers cross-disciplinary insights for algorithm optimization and experimental validation research in the field of computational biology.

Transfer learning has been widely applied to drug sensitivity prediction based on single-cell RNA sequencing, leveraging knowledge from large datasets of cancer cell lines or other sources to improve the prediction of drug responses. However, previous studies require model fine-tuning for different patient single-cell datasets, limiting their ability to meet the clinical need for high-throughput rapid prediction. In this research, we introduce single-cell Adaptive Transfer and Distillation model (scATD), a transfer learning framework leveraging large language models for high-throughput drug sensitivity prediction. Based on different large language models (scFoundation and Geneformer) and transfer strategies, scATD includes three distinct sub-models: scATD-sf, scATD-gf, and scATD-sf-dist. scATD-sf and scATD-gf employs an important bidirectional style transfer to enable predictions for new patients without model parameter training. Additionally, scATD-sf-dist uses knowledge distillation from large models to enhance prediction performance, improve efficiency, and reduce resource requirements. Benchmarking across more diverse datasets demonstrates scATD's superior accuracy, generalization and efficiency. Besides, by rigorously selecting reference background samples for feature attribution algorithms, scATD also provides more meaningful insights into the relationship between gene expression and drug resistance mechanisms. Making scATD more interpretability for addressing critical challenges in precision oncology.

MOTIVATION: Protein function prediction is important for drug development and disease treatment. Recently, deep learning methods have leveraged protein sequence and structural information, achieving remarkable progress in the field of protein function prediction. However, existing methods ignore the complex multimodal interaction information between sequence and structural features. Since protein sequence and structural information reveal the functional characteristics of proteins from different perspectives, it is challenging to effectively fuse the information from these two modalities to portray protein functions more comprehensively. In addition, current methods have difficulty in effectively capturing long-range dependencies and global contextual information in protein sequences during feature extraction, thus limiting the ability of the model to recognize critical functional residues. RESULTS: In this study, we propose a novel framework termed Multi-stage Attention-based Extraction and Fusion model for GO prediction (MAEF-GO) based on a multistage attention mechanism to predict protein functions. MAEF-GO innovatively integrates the graph convolutional network and the graph attention network to extract protein structural features. To address the issue of modeling long-range dependencies within protein sequences, we introduce a frequency-domain attention mechanism capable of extracting global contextual relationships. Additionally, a cross-attention module is implemented to facilitate interactive fusion between protein sequence and structural modalities. Experimental evaluations demonstrate that MAEF-GO achieves superior performance compared to several state-of-the-art baseline models across standard benchmarks. Furthermore, analysis of the cross-attention weight distributions demonstrates MAEF-GO's interpretability. It can effectively identify critical functional residues of proteins. AVAILABILITY AND IMPLEMENTATION: The MAEF-GO source code can be found at https://github.com/nebstudio/MAEF-GO, an archived snapshot of the code used in this study is also available via Zenodo at https://doi.org/10.5281/zenodo.15422392.

Recently, numerous large language models (LLMs) have emerged as foundational models, reshaping biological data modeling and achieving remarkable breakthroughs in both discriminative and generative tasks. The success of these models is largely attributed to the inherent similarities between natural language and biological data, such as DNA, RNA, and amino acid sequences. Through pre-training and fine-tuning phases, LLMs have demonstrated their ability to effectively model these biological datasets. Additionally, while protein structures and RNA-seq expression data are not inherently sequential, they can still be modeled and predicted effectively by LLMs based on the Transformer architecture. Previous research has predominantly focused on architectural innovations in LLMs and their applications to sequential data across various domains. However, there is a notable lack of systematic reviews addressing the reasons and methods behind LLM modifications for fitting biological omics data, particularly for non-sequential data types. Furthermore, comprehensive analyses of LLM applications in synthetic biology remain limited. We first systematically review representative LLMs in the biological domain. Next, we delve into their applications across the genome, transcriptome, and proteome fields, detailing the goals, processes, datasets, and methodologies involved. Finally, we discuss the challenges of applying LLMs to biological omics data and fundamental scientific research. In summary, we aim to provide a comprehensive overview of the technical and conceptual advances in this field, as well as an essential resource for researchers exploring the diverse applications of LLMs across various biological disciplines.

Ubiquitination is critical in biomedical research. Predicting ubiquitination sites based on deep learning model have advanced the study of ubiquitination. However, traditional supervised model limits in the scenarios where labels are scarcity across species. To address this issue, we introduce EUP, an online webserver for ubiquitination prediction and model interpretation for multi-species. EUP is constructed by extracting lysine site-dependent features from pretrained language model ESM2. Then, utilizing conditional variational inference to reduce the ESM2 features to a lower-dimensional latent representation. By constructing downstream models built on this latent feature representation, EUP exhibited superior performance in predicting ubiquitination sites across species, while maintaining low inference latency. Furthermore, key features for predicting ubiquitination sites were identified across animals, plants, and microbes. The identification of shared key features that capture evolutionarily conserved traits enhances the interpretability of the EUP model for ubiquitination prediction. EUP is free and available at (https://eup.aibtit.com/).

Tabular data in scientific papers provides valuable structured information for knowledge discovery and validation. Although the language models such as BERT and ChatGPT have significantly advanced the research on general domain tables, challenges remain in scientific tables. Specifically, such models have limitations in understanding scientific entities, as well as lacks numerical representation and computation capabilities. Previous studies have focused on scientific tables, but they are limited to individual modules or tasks and lack a comprehensive framework. To address these issues, we introduce a reading comprehension framework for scientific tables, named NRTR, which uses a multi-task learning approach that shares a common encoder, achieves reasoning across various tasks, including question answering, cloze testing, and fact verification. It has the following characteristics: (1) utilizing entity linking and named entity recognition to extract key information from papers, which enhances the models' understanding of scientific entities; (2) injecting numerical representation capabilities into language models and promoting the model's understanding of the relative magnitude of numbers to better reason about maximum and difference values. Notably, the existing scientific corpus lacks tabular contexts or does not integrate computational reasoning, which hinders the evaluation of reasoning models in scientific tables. To this end, we release SciTab, a multi-task dataset that merges high-quality scientific tables with contextual information to provide a benchmark for future research. Our experimental results show that NRTR outperforms existing models on SciTab.

As the application of large language models (LLMs) has broadened into the realm of biological predictions, leveraging their capacity for self-supervised learning to create feature representations of amino acid sequences, these models have set a new benchmark in tackling downstream challenges, such as subcellular localization. However, previous studies have primarily focused on either the structural design of models or differing strategies for fine-tuning, largely overlooking investigations into the nature of the features derived from LLMs. In this research, we propose different ESM2 representation extraction strategies, considering both the character type and position within the ESM2 input sequence. Using model dimensionality reduction, predictive analysis and interpretability techniques, we have illuminated potential associations between diverse feature types and specific subcellular localizations. Particularly, the prediction of Mitochondrion and Golgi apparatus prefer segments feature closer to the N-terminal, and phosphorylation site-based features could mirror phosphorylation properties. We also evaluate the prediction performance and interpretability robustness of Random Forest and Deep Neural Networks with varied feature inputs. This work offers novel insights into maximizing LLMs' utility, understanding their mechanisms, and extracting biological domain knowledge. Furthermore, we have made the code, feature extraction API, and all relevant materials available at https://github.com/yujuan-zhang/feature-representation-for-LLMs.