Exploring the functions of genes and gene products is crucial to a wide range of fields, including medical research, evolutionary biology, and environmental science. However, discovering new functions largely relies on expensive and exhaustive wet lab experiments. Existing methods of automatic function annotation or prediction mainly focus on protein function prediction with sequence, 3D-structures or protein family information. In this study, we propose to tackle the gene function prediction problem by exploring Gene Ontology graph and annotation with BERT (GoBERT) to decipher the underlying relationships among gene functions. Our proposed novel function prediction task utilizes existing functions as inputs and generalizes the function prediction to gene and gene products. Specifically, two pre-train tasks are designed to jointly train GoBERT to capture both explicit and implicit relations of functions. Neighborhood prediction is a self-supervised multi-label classification task that captures the explicit function relations. Specified masking and recovering task helps GoBERT in finding implicit patterns among functions. The pre-trained GoBERT possess the ability to predict novel functions for various gene and gene products based on known functional annotations. Extensive experiments, biological case studies, and ablation studies are conducted to demonstrate the superiority of our proposed GoBERT.

Accurate prediction of RNA properties, such as stability and interactions, is crucial for advancing our understanding of biological processes and developing RNA-based therapeutics. RNA structures can be represented as 1D sequences, 2D topological graphs, or 3D all-atom models, each offering different insights into its function. Existing works predominantly focus on 1D sequence-based models, which overlook the geometric context provided by 2D and 3D geometries. This study presents the first systematic evaluation of incorporating explicit 2D and 3D geometric information into RNA property prediction, considering not only performance but also real-world challenges such as limited data availability, partial labeling, sequencing noise, and computational efficiency. To this end, we introduce a newly curated set of RNA datasets with enhanced 2D and 3D structural annotations, providing a resource for model evaluation on RNA data. Our findings reveal that models with explicit geometry encoding generally outperform sequence-based models, with an average prediction RMSE reduction of around 12% across all various RNA tasks and excelling in low-data and partial labeling regimes, underscoring the value of explicitly incorporating geometric context. On the other hand, geometry-unaware sequence-based models are more robust under sequencing noise but often require around 2-5x training data to match the performance of geometry-aware models. Our study offers further insights into the trade-offs between different RNA representations in practical applications and addresses a significant gap in evaluating deep learning models for RNA tasks.

Messenger RNA (mRNA) plays a crucial role in protein synthesis, with its codon structure directly impacting biological properties. While Language Models (LMs) have shown promise in analyzing biological sequences, existing approaches fail to account for the hierarchical nature of mRNA's codon structure. We introduce Hierarchical Encoding for mRNA Language Modeling (HELM), a novel pre-training strategy that incorporates codon-level hierarchical structure into language model training. HELM modulates the loss function based on codon synonymity, aligning the model's learning process with the biological reality of mRNA sequences. We evaluate HELM on diverse mRNA datasets and tasks, demonstrating that HELM outperforms standard language model pre-training as well as existing foundation model baselines on six diverse downstream property prediction tasks and an antibody region annotation tasks on average by around 8%. Additionally, HELM enhances the generative capabilities of language model, producing diverse mRNA sequences that better align with the underlying true data distribution compared to non-hierarchical baselines.

Inferring Gene Regulatory Networks (GRNs) from gene expression data is crucial for understanding biological processes. While supervised models are reported to achieve high performance for this task, they rely on costly ground truth (GT) labels and risk learning gene-specific biases—such as class imbalances of GT interactions—rather than true regulatory mechanisms. To address these issues, we introduce InfoSEM, an unsupervised generative model that leverages textual gene embeddings as informative priors, improving GRN inference without GT labels. InfoSEM can also integrate GT labels as an additional prior when available, avoiding biases and further enhancing performance. Additionally, we propose a biologically motivated benchmarking framework that better reflects real-world applications such as biomarker discovery and reveals learned biases of existing supervised methods. InfoSEM outperforms existing models by 38.5% across four datasets using textual embeddings prior and further boosts performance by 11.1% when integrating labeled data as priors.

Processing global geometric context while preserving equivariance is crucial when modeling biological, chemical, and physical systems. Yet, this is challenging due to the computational demands of equivariance and global context at scale. Standard methods such as equivariant self-attention suffer from quadratic complexity, while local methods such as distance-based message passing sacrifice global information. Inspired by the recent success of state-space and long-convolutional models, we introduce Geometric Hyena, the first equivariant long-convolutional model for geometric systems. Geometric Hyena captures global geometric context at sub-quadratic complexity while maintaining equivariance to rotations and translations. Evaluated on all-atom property prediction of large RNA molecules and full protein molecular dynamics, Geometric Hyena outperforms existing equivariant models while requiring significantly less memory and compute that equivariant self-attention. Notably, our model processes the geometric context of $30k$ tokens $20 \times$ faster than the equivariant transformer and allows $72 \times$ longer context within the same budget.

Molecular property prediction aims to learn representations that map chemical structures to functional properties. While multimodal learning has emerged as a powerful paradigm to learn molecular representations, prior works have largely overlooked textual and taxonomic information of molecules for representation learning. We introduce TRIDENT, a novel framework that integrates molecular SMILES, textual descriptions, and taxonomic functional annotations to learn rich molecular representations. To achieve this, we curate a comprehensive dataset of molecule-text pairs with structured, multi-level functional annotations. Instead of relying on conventional contrastive loss, TRIDENT employs a volume-based alignment objective to jointly align tri-modal features at the global level, enabling soft, geometry-aware alignment across modalities. Additionally, TRIDENT introduces a novel local alignment objective that captures detailed relationships between molecular substructures and their corresponding sub-textual descriptions. A momentum-based mechanism dynamically balances global and local alignment, enabling the model to learn both broad functional semantics and fine-grained structure-function mappings. TRIDENT achieves state-of-the-art performance on 18 downstream tasks, demonstrating the value of combining SMILES, textual, and taxonomic functional annotations for molecular property prediction. Our code and data are available at https://github.com/uta-smile/TRIDENT.

2D/3D image registration to align a 3D volume and 2D X-ray images is a challenging problem due to its ill-posed nature and various artifacts presented in 2D X-ray images. In this paper, we propose a multi-agent system with an auto attention mechanism for robust and efficient 2D/3D image registration. Specifically, an individual agent is trained with dilated Fully Convolutional Network (FCN) to perform registration in a Markov Decision Process (MDP) by observing a local region, and the final action is then taken based on the proposals from multiple agents and weighted by their corresponding confidence levels. The contributions of this paper are threefold. First, we formulate 2D/3D registration as a MDP with observations, actions, and rewards properly defined with respect to X-ray imaging systems. Second, to handle various artifacts in 2D X-ray images, multiple local agents are employed efficiently via FCN-based structures, and an auto attention mechanism is proposed to favor the proposals from regions with more reliable visual cues. Third, a dilated FCN-based training mechanism is proposed to significantly reduce the Degree of Freedom in the simulation of registration environment, and drastically improve training efficiency by an order of magnitude compared to standard CNN-based training method. We demonstrate that the proposed method achieves high robustness on both spine cone beam Computed Tomography data with a low signal-to-noise ratio and data from minimally invasive spine surgery where severe image artifacts and occlusions are presented due to metal screws and guide wires, outperforming other state-of-the-art methods (single agent-based and optimization-based) by a large margin.

Domain adaptation aims to transfer knowledge across different data domains (distributions). In this paper, we propose a novel model to enforce the distributional and structural similarities during the adaptation. Specifically, we embed the data from both domain into a latent space where the distributions are matched via minimization of the maximum mean discrepancy metric. Then we characterize the transformed manifold by means of graphs and maximize the similarities between the graphical structure in the embedding space. This is achieved by minimizing the spectrum distance of the graph Laplacians between the embeddings. The spectrum maintains intrinsic information of the manifold and is invariant under data permutations and unitary transformations of the eigenspace. A two stage optimization algorithm is proposed to minimize the distributional and structural differences iteratively. Extensive numerical experiments have been done and demonstrated the superiority of our proposed model over previous arts.

3-D image registration, which involves aligning two or more images, is a critical step in a variety of medical applications from diagnosis to therapy. Image registration is commonly performed by optimizing an image matching metric as a cost function. However this task is challenging due to the non-convex nature of the matching metric over the plausible registration parameter space and insufficient approches for a robust optimization. As a result, current approaches are often customized to a specific problem and sensitive to image quality and artifacts. In this paper, we propose a completely different approach to image registration, inspired by how experts perform the task. We first cast the image registration problem as a "strategic learning" process, where the goal is to find the best sequence of motion actions (e.g. up, down, etc) that yields image alignment. Within this approach, an artificial agent is learned, modeled using deep convolutional neural networks, with 3D raw image data as the input, and the next optimal action as the output. To copy with the dimensionality of the problem, we propose a greedy supervised approach for an end-to-end training, coupled with attention-driven hierarchical strategy. The resulting registration approach inherently encodes both a data-driven matching metric and an optimal registration strategy (policy). We demonstrate on two 3-D/3-D medical image registration examples with drastically different nature of challenges, that the artificial agent outperforms several state-of-the-art registration methods by a large margin in terms of both accuracy and robustness.

In this paper, we present a Convolutional Neural Network (CNN) regression approach to address the two major limitations of existing intensity-based 2-D/3-D registration technology: 1) slow computation and 2) small capture range. Different from optimization-based methods, which iteratively optimize the transformation parameters over a scalar-valued metric function representing the quality of the registration, the proposed method exploits the information embedded in the appearances of the digitally reconstructed radiograph and X-ray images, and employs CNN regressors to directly estimate the transformation parameters. An automatic feature extraction step is introduced to calculate 3-D pose-indexed features that are sensitive to the variables to be regressed while robust to other factors. The CNN regressors are then trained for local zones and applied in a hierarchical manner to break down the complex regression task into multiple simpler sub-tasks that can be learned separately. Weight sharing is furthermore employed in the CNN regression model to reduce the memory footprint. The proposed approach has been quantitatively evaluated on 3 potential clinical applications, demonstrating its significant advantage in providing highly accurate real-time 2-D/3-D registration with a significantly enlarged capture range when compared to intensity-based methods.

2-D X-ray image navigation during an electrophysiology ablation procedure can be enhanced via the overlay of images derived from pre-operative 3-D data, to provide anatomical details that otherwise are not visible under X-ray. However, accurate registration of 3-D data and 2-D X-ray during electrophysiology ablation is a challenging problem, due to the relatively low image quality and the fact that the contrast medium typically fills only a small part of the left atrium in X-ray images. In this paper, we propose a robust 2-D/3-D registration method tailored for electrophysiology ablation procedures. In particular, a weighted similarity measure is utilized to handle the partial data problem, and a sequential subspace optimization is proposed to take advantage of the available bi-plane X-ray images, for a robust and efficient registration. Since the contrast medium washes out quickly and thus visible in only a small number of frames in the X-ray sequence, we furthermore streamline the workflow by automatically detecting the frame that is optimal for registration purpose. Experimental results on seven clinical data sets demonstrate the effectiveness of the proposed method.

Minimally invasive and less invasive procedure is becoming more and more common in medical therapy. Image guidance is an indispensable component in minimally invasive procedures by providing critical information about the position of the target sites and the optimal manipulation of the devices, while the field of view is limited to naked eyes due to the small incision. Registration is one of the enabling technologies for computer-aided image guidance, which brings high-resolution pre-operative data into the operating room to provide more realistic information about the patient's anatomy. In this paper, we survey the recent advances in registration techniques applied to minimally and/or less invasive therapy, including a wide variety of therapies in surgery, endoscopy, interventional cardiology, interventional radiology, and hybrid procedures. The registration approaches are categorized into several groups, including projection-to-volume, slice-to-volume, video-to-volume, and volume-to-volume registration. The focus is on recent advances in registration techniques that are specifically developed for minimally and/or less invasive procedures in the following medical specialties: neuroradiology and neurosurgery, cardiac applications, and thoracic-abdominal interventions.

Fluoroscopic overlay images rendered from preoperative volumetric data can provide additional anatomical details to guide physicians during catheter ablation procedures for treatment of atrial fibrillation (AFib). As these overlay images are often compromised by cardiac and respiratory motion, motion compensation methods are needed to keep the overlay images in sync with the fluoroscopic images. So far, these approaches have either required simultaneous biplane imaging for 3-D motion compensation, or in case of monoplane X-ray imaging, provided only a limited 2-D functionality. To overcome the downsides of the previously suggested methods, we propose an approach that facilitates a full 3-D motion compensation even if only monoplane X-ray images are available. To this end, we use a training phase that employs a biplane sequence to establish a patient specific motion model. Afterwards, a constrained model-based 2-D/3-D registration method is used to track a circumferential mapping catheter. This device is commonly used for AFib catheter ablation procedures. Based on the experiments on real patient data, we found that our constrained monoplane 2-D/3-D registration outperformed the unconstrained counterpart and yielded an average 2-D tracking error of 0.6 mm and an average 3-D tracking error of 1.6 mm. The unconstrained 2-D/3-D registration technique yielded a similar 2-D performance, but the 3-D tracking error increased to 3.2 mm mostly due to wrongly estimated 3-D motion components in X-ray view direction. Compared to the conventional 2-D monoplane method, the proposed method provides a more seamless workflow by removing the need for catheter model re-initialization otherwise required when the C-arm view orientation changes. In addition, the proposed method can be straightforwardly combined with the previously introduced biplane motion compensation technique to obtain a good trade-off between accuracy and radiation dose reduction.

Transcatheter aortic valve implantation (TAVI) is a minimally invasive procedure to treat severe aortic valve stenosis. As an emerging imaging technique, C-arm computed tomography (CT) plays a more and more important role in TAVI on both pre-operative surgical planning (e.g., providing 3-D valve measurements) and intra-operative guidance (e.g., determining a proper C-arm angulation). Automatic aorta segmentation and aortic valve landmark detection in a C-arm CT volume facilitate the seamless integration of C-arm CT into the TAVI workflow and improve the patient care. In this paper, we present a part-based aorta segmentation approach, which can handle structural variation of the aorta in case that the aortic arch and descending aorta are missing in the volume. The whole aorta model is split into four parts: aortic root, ascending aorta, aortic arch, and descending aorta. Discriminative learning is applied to train a detector for each part separately to exploit the rich domain knowledge embedded in an expert-annotated dataset. Eight important aortic valve landmarks (three hinges, three commissures, and two coronary ostia) are also detected automatically with an efficient hierarchical approach. Our approach is robust under all kinds of variations observed in a real clinical setting, including changes in the field-of-view, contrast agent injection, scan timing, and aortic valve regurgitation. Taking about 1.1 s to process a volume, it is also computationally efficient. Under the guidance of the automatically extracted patient-specific aorta model, the physicians can properly determine the C-arm angulation and deploy the prosthetic valve. Promising outcomes have been achieved in real clinical applications.

In this paper we propose and validate a PCA-based respiratory motion model for motion compensation during image-guided cardiac interventions. In a preparatory training phase, a preoperative 3-D segmentation of the coronary arteries is automatically registered with a cardiac gated biplane cineangiogram, and used to build a respiratory motion model. This motion model is subsequently used as a prior within the intraoperative registration process for motion compensation to restrict the search space. Our hypothesis is that the use of this model-constrained registration increases the robustness and registration accuracy, especially for weak data constraints such as low signal-to-noise ratio, the lack of contrast information, or an intraoperative monoplane setting. This allows for reducing radiation exposure without compromising on registration accuracy. Synthetic data as well as phantom and clinical datasets have been used to validate the model-based registration in terms of registration accuracy, robustness and speed. We were able to significantly accelerate the intraoperative registration with a 3-D TRE of less than 2 mm for both monoplane images and intraprocedure settings with missing contrast information based on 2-D guidewire tracking, which makes it feasible for motion correction in clinical procedures.

A three-dimensional image registration method for motion correction of functional magnetic resonance imaging (fMRI) time-series, based on independent component analysis (ICA), is described. We argue that movement during fMRI data acquisition results in a simultaneous increase in the joint entropy of the observed time-series and a decrease in the joint entropy of a nonlinear function of the derived spatially independent components calculated by ICA. We propose this entropy difference as a reliable criterion for motion correction and refer to a method that maximizes this as motion-corrected ICA (MCICA). Specifically, a given motion-corrupted volume may be corrected by determining the linear combination of spatial transformations of the motion-corrupted volume that maximizes the proposed criterion. In essence, MCICA consists of designing an adaptive spatial resampling filter which maintains maximum temporal independence among the recovered components. In contrast with conventional registration methods, MCICA does not require registration of motion-corrupted volumes to a single reference volume which can introduce artifacts because corrections are applied without accounting for variability due to the task-related activation. Simulations demonstrate that MCICA is robust to activation level, additive noise, random motion in the reference volumes and the exact number of independent components extracted. When the method was applied to real data with minimal estimated motion, the method had little effect and, hence, did not introduce spurious changes in the data. However, in a data series from a motor fMRI experiment with larger motion, preprocessing the data with the proposed method resulted in the emergence of activation in primary motor and supplementary motor cortices. Although mutual information (MI) was not explicitly optimized, the MI between all subsequent volumes and the first one was consistently increased for all volumes after preprocessing the data with MCICA. We suggest MCICA represents a robust and reliable method for preprocessing of fMRI time-series corrupted with motion.

Head movement during fMRI data collection can result in confusing artifacts when estimating task-related brain activity. We propose an improved version of motion-corrected independent component analysis (MCICA), which mitigates the motion effects of the fMRI time-series by maximizing the entropy difference between the observed fMRI data and a nonlinear function of the derived ICA components. Specifically, the improved MCICA algorithm operates on all timepoints, removing the requirement for the existence of enough motionless timepoints in the time-series and the need to detect motion-corrupted timepoints. Simulations demonstrate that MCICA is robust to activity level and the results are more accurate than cubic interpolation, even when the displacement is known. In real data from a motor fMRI experiment, preprocessing the data with MCICA resulted in the emergence of activity in the primary motor and supplementary motor cortices, and the mutual information between all subsequent volumes and the first one was increased.

An automatic face recognition system based on multiple facial features is described. Each facial feature is represented by a Gabor-based complex vector and is localized by an automatic facial feature detection scheme. Two face recognition approaches, named two-layer nearest neighbor (TLNN) and modular nearest feature line (MNFL) respectively, are proposed. Both TLNN and MNFL are based on the multiple facial features detected for each image and their superiority in face recognition is demonstrated.