Large language models (LLMs) exhibit logically inconsistent hallucinations that appear coherent yet violate reasoning principles, with recent research suggesting an inverse relationship between causal reasoning capabilities and such hallucinations. However, existing reasoning approaches in LLMs, such as Chain-of-Thought (CoT) and its graph-based variants, operate at the linguistic token level rather than modeling the underlying causal relationships between variables, lacking the ability to represent conditional independencies or satisfy causal identification assumptions. To bridge this gap, we introduce causal-DAG construction and reasoning (CDCR-SFT), a supervised fine-tuning framework that trains LLMs to explicitly construct variable-level directed acyclic graph (DAG) and then perform reasoning over it. Moreover, we present a dataset comprising 25,368 samples (CausalDR), where each sample includes an input question, explicit causal DAG, graph-based reasoning trace, and validated answer. Experiments on four LLMs across eight tasks show that CDCR-SFT improves the causal reasoning capability with the state-of-the-art 95.33% accuracy on CLADDER (surpassing human performance of 94.8% for the first time) and reduces the hallucination on HaluEval with 10% improvements. It demonstrates that explicit causal structure modeling in LLMs can effectively mitigate logical inconsistencies in LLM outputs.

Reasoning Segmentation (RS) is a multimodal vision-text task that requires segmenting objects based on implicit text queries, demanding both precise visual perception and vision-text reasoning capabilities. Current RS approaches rely on fine-tuning vision-language models (VLMs) for both perception and reasoning, but their tokenization of images fundamentally disrupts continuous spatial relationships between objects. We introduce DTwinSeger, a novel RS approach that leverages Digital Twin (DT) representation as an intermediate layer to decouple perception from reasoning. Innovatively, DTwinSeger reformulates RS as a two-stage process, where the first transforms the image into a structured DT representation that preserves spatial relationships and semantic properties and then employs a Large Language Model (LLM) to perform explicit reasoning over this representation to identify target objects. We propose a supervised fine-tuning method specifically for LLM with DT representation, together with a corresponding fine-tuning dataset Seg-DT, to enhance the LLM’s reasoning capabilities with DT representations. Experiments show that our method can achieve state-of-the-art performance on two image RS benchmarks and three image referring segmentation benchmarks.

Medical image analysis increasingly relies on the integration of multiple imaging modalities to capture complementary anatomical and functional information, enabling more accurate diagnosis and treatment planning. Achieving aligned feature representations across these diverse modalities is therefore important for effective multimodal analysis. While contrastive language-image pre-training (CLIP) and its variant have enabled image-text alignments, they require explicitly paired data between arbitrary two modalities, which is difficult to acquire in medical contexts. To address the gap, we present Multimodal Medical Image Binding with Text (M3Bind), a novel pre-training framework that enables seamless alignment of multiple medical imaging modalities through a shared text representation space without requiring explicit paired data between any two medical image modalities. Specifically, based on the insight that different images can naturally bind with text, M3Bind first fine-tunes pre-trained CLIP-like image-text models, which are derived from different medical modalities, to align their modality-specific text embedding space while preserving their original image-text alignments. Subsequently, we distill these modality-specific text encoders into a unified model, creating a shared text embedding space. Notably, M3Bind is a flexible framework in which the selection of CLIP-like models is not fixed and can be adapted according to the requirements of the task. Experiments on X-ray, CT, retina, ECG, and pathological images on multiple downstream tasks demonstrate that M3Bind achieves competitive or even superior performance in zero-shot, few-shot classification and cross-modal retrieval tasks compared to its CLIP-like counterparts. These results validate M3Bind's effectiveness in achieving cross-image-modal alignment for medical analysis.

Autonomous multi-agent trajectory prediction in open-world scenarios presents persistent challenges, including high behavioral uncertainty, long-horizon dependencies, and the lack of structured guidance during generation. Existing generative approaches often compromise behavioral fidelity in favor of accuracy or diversity, resulting in predictions that are either unrealistic or difficult to control. We propose M2Traj, a unified framework that couples a closed-loop conditional diffusion model with structured trajectory reasoning and behavior-driven constraints. M2Traj features a history-guided encoder that captures long-range cross-agent dependencies and scene semantics, and a dynamic closed-loop rollout mechanism that refines predictions through goal-conditioned denoising with iterative feedback. To enable fine-grained control, we introduce a learnable behavior guidance module that softly enforces constraints on velocity, collision risk, comfort, and traffic rule adherence. By jointly modeling agent interactions, future constraints, and uncertainty within a structured generative process, M2Traj delivers controllable and reliable predictions across diverse urban scenarios. Extensive experiments on three large-scale benchmarks—Waymo, HighD, and MoCAD—demonstrate that M2Traj achieves balanced and reliable performance across standard accuracy, diversity, and behavior-sensitive metrics, highlighting its potential as a generalizable solution for controllable, structure-aware trajectory prediction in complex multi-agent environments.

Examining thyroid fine needle aspiration (FNA) can grade cancer risks, derive prognostic information, and guide follow-up care or surgery decision-making. However, thyroid cytology's diagnostic cues are more dispersed compared with pathology images in other disciplines, making standard annotation strategy for AI diagnosis labor-intensive. Inspired by how cytologists diagnose under the microscope, we propose an innovative cellularity-based active learning framework, namely Cyto-AL, to correlate cellularity with diagnostic categories for the active learning query. We also improve the Whole Slide Image (WSI) category of The Bethesda System for Reporting Thyroid Cytology (TBSRTC) prediction by proposing severe-stage pinpointed Multiple Instance Learning (MIL). Additionally, we introduce a lightweight score model to optimize the query in human in the loop (HITL) annotation strategy. Given scarce public thyroid cytology datasets, we release our collected and labeled images as benchmarks. The benchmark comprises 138 WSIs (27,496 valid image patches) collected from 2021-2023 across six classes, annotated by three pathologists using TBSRTC. At patch-level verification, Cyto-AL achieves a 2.2% average classification accuracy improvement over state-of-the-art methods with an equally labeled dataset, and its lightweight ranking-aware model reduces training time by around 65%. Moreover, the WSI-level MIL approach improves average accuracy by 10.7% and Macro-F1 score by 3.5%, outperforming standard sampling methods such as Monte Carlo sampling. The source code and dataset are available at https://github.com/Junchao-Zhu/Cyto-AL.

Foundation Models (FMs) have shown great promise for multimodal medical image analysis such as Magnetic Resonance Imaging (MRI). However, certain MRI sequences may be unavailable due to various constraints, such as limited scanning time, patient discomfort, or scanner limitations. The absence of certain modalities can hinder the performance of FMs in clinical applications, making effective missing modality imputation crucial for ensuring their applicability. Previous approaches, including generative adversarial networks (GANs), have been employed to synthesize missing modalities in either a one-to-one or many-to-one manner. However, these methods have limitations, as they require training a new model for different missing scenarios and are prone to mode collapse, generating limited diversity in the synthesized images. To address these challenges, we propose DiffM4RI, a diffusion model for many-to-many missing modality imputation in MRI. DiffM4RI innovatively formulates the missing modality imputation as a modality-level inpainting task, enabling it to handle arbitrary missing modality situations without the need for training multiple networks. Experiments on the BraTs datasets demonstrate DiffM4RI can achieve an average SSIM improvement of 0.15 over MustGAN, 0.1 over SynDiff, and 0.02 over VQ-VAE-2. These results highlight the potential of DiffM4RI in enhancing the reliability of FMs in clinical applications. The code is available athttps://github.com/27yw/DiffM4RI.

Automated Machine Learning (AutoML) simplifies ML pipeline development but most existing approaches focus only on discriminative tasks, neglecting generative models and multimodal applications. We introduce UniAutoML, a human-centered multimodal AutoML framework that uses large language models (LLMs) to unify automation across both discriminative (e.g., CNNs, Transformers) and generative models (e.g., diffusion models, LLMs) for images, audio, video, and text. UniAutoML enables natural-language interaction and real-time feedback, improving interpretability and user control without programming expertise. A built-in safety guardline filters harmful inputs/outputs. Experiments across multiple modalities and user studies (25 participants) show UniAutoML achieves superior performance and usability. Code: https://github.com/richarf22/UniAutoML.

Personalising large text-to-image diffusion models with only a handful of images unlocks various applications but exposes a fidelity–forgetting dilemma: models learn new concepts slowly and often erase semantically distant knowledge. We tackle this by injecting discriminative signals into the fine-tuning loop. Our framework, DiffDisReg, couples a Segmentation-Guided Attention Regularizer with a lightweight feature-level head and schedules both through a Semantic-Similarity regularization Scheduler that targets concepts most at risk of being forgotten. Experiments on DreamBooth benchmarks show that DiffDisReg reaches baseline geometric fidelity in under half the training steps while improving final image quality and reducing out-of-domain degradation. Ablation studies confirm the contribution of each component, and qualitative timelines reveal cleaner geometry and color from the earliest iterations. The code is available at: https://github.com/yj373/DiffDisReg.

In recent years, researchers have employed search agents to perform real-time retrieval of biomedical information or leveraged biomedical Retrieval-Augmented Generation (RAG) frameworks to integrate authoritative biomedical knowledge bases into the retrieval and generation process, aiming to mitigate the hallucination issues of large language models (LLMs) in biomedical contexts. However, search agents face limitations in multi-source retrieval, such as the lack of integration with protein databases, which can lead to protein information mismatches and erroneous feedback of 3D structural data. Moreover, existing biomedical RAG frameworks lack real-time retrieval capabilities, making it difficult to access the latest scientific findings in a timely manner. To effectively address the limitations of existing methods, we propose BioMedSearch, a real-time multi-source biomedical information retrieval framework based on LLMs. The framework consists of three core modules: Biomedical Search Planner, Biomedical Retrieval Executor, and Summary Report Generator. These modules decompose complex biomedical queries, such as protein-related queries, to extract keywords and perform real-time multi-source retrieval across literature, web, and protein databases, generating high-quality, contextually accurate answers through iterative filtering. To evaluate the accuracy of question answering, we constructed a multi-level dataset, BioMedMCQs, consisting of 3,000 questions. The dataset covers three levels of reasoning: mechanistic identification, non-adjacent semantic integration, and temporal causal reasoning, and is used to assess the performance of BioMedSearch and other methods on complex QA tasks. Experimental results show that BioMedSearch outperforms all existing comparison methods at level 1–3. Specifically, compared to search agents, BioMedSearch surpasses the best-performing method, PaSa, with an average accuracy improvement of 13.2 % at level 3; in comparison with biomedical RAG frameworks, it outperforms the strongest method, MedRAG, achieving a 15.6% average accuracy gain at level 3. The code and BioMedMCQs are available at: https://github.com/CyL-ucas/BioMed_Search.

Virtual staining can visualize transparent tissue and cellular structures in pathological images. However, existing virtual staining methods focus on emulating existing real-world stains. Furthermore, these methods are based on learning-based approaches, which can be computationally expensive and timeconsuming. To address these challenges, we propose a novel non-learning-based virtual staining framework that leverages physicsinspired particle swarm optimization to discover enhanced staining templates. Our method formulates virtual staining as an optimization problem, efficiently exploring the vast staining template space using a modified update strategy inspired by Newton's laws of motion. To guide the search process, we introduce an innovative composite staining quality metric that integrates colorfulness, tissue component differentiation, and perceptual quality. Our approach aims to balance visual vibrancy with perceptual fidelity. We evaluated our method on four diverse pathological image datasets, demonstrating consistent improvements in segmentation and classification performance compared to traditional staining methods, with performance gains ranging from 1.38 % to 2.03 %. The optimized virtual stains identified by our method expand beyond the limitations of existing real-world stains, offering new possibilities for tissue visualization and analysis.

Reasoning segmentation (RS) aims to identify and segment objects of interest based on implicit text queries. As such, RS is a catalyst for embodied AI agents, enabling them to interpret high-level commands without requiring explicit step-by-step guidance. However, current RS approaches rely heavily on the visual perception capabilities of multimodal large language models (LLMs), leading to several major limitations. First, they struggle with queries that require multiple steps of reasoning or those that involve complex spatial/temporal relationships. Second, they necessitate LLM fine-tuning, which may require frequent updates to maintain compatibility with contemporary LLMs and may increase risks of catastrophic forgetting during fine-tuning. Finally, being primarily designed for static images or offline video processing, they scale poorly to online video data. To address these limitations, we propose an agent framework that disentangles perception and reasoning for online video RS without LLM fine-tuning. Our innovation is the introduction of a just-in-time digital twin concept, where -- given an implicit query -- a LLM plans the construction of a low-level scene representation from high-level video using specialist vision models. We refer to this approach to creating a digital twin as "just-in-time" because the LLM planner will anticipate the need for specific information and only request this limited subset instead of always evaluating every specialist model. The LLM then performs reasoning on this digital twin representation to identify target objects. To evaluate our approach, we introduce a new comprehensive video reasoning segmentation benchmark comprising 200 videos with 895 implicit text queries. The benchmark spans three reasoning categories (semantic, spatial, and temporal) with three different reasoning chain complexity.

The increasing volume of radiology examinations has created an urgent need for automated report generation that is comparable to those written by radiologists. The major challenge is achieving precise semantic alignment between images and text, ensuring that generated reports accurately capture and describe the visual findings in medical images. Current approaches often struggle with this alignment, compromising diagnostic accuracy and clinical utility. To address these challenges, we present Adaptive Semantic Alignment Method (ASAM), a novel framework that enhances cross-modal semantic alignment through two innovations. First, we introduce a gated disease knowledge base by memory matrix that provides structured medical context to guide the mapping between image and report modalities. Second, we develop a cross-modal pre-trained visual encoder that enriches feature representation through improved understanding of medical imaging characteristics. Extensive experiments demonstrate that ASAM achieves state-of-the-art performance on two public chest X-ray datasets in BLEU-n metrics.

Diffusion models have rapidly advanced the field of data synthesis as powerful generative models. However, efficiently evaluating their generative capability while reflecting human preferences remains challenging. In contrast, evaluation metrics for discriminative tasks are based on objective truth and can scale across various image domains with negligible cost. On the other hand, recent studies have expanded the application of diffusion models to discriminative tasks, highlighting a promising intersection between generative and discriminative capabilities. Inspired by these, we propose DiffDisEval, a novel benchmark method that assesses the generative ability of diffusion models through the lens of the discriminative segmentation task. DiffDisEval utilizes semantic masks generated by the denoising network in diffusion models, which fuses self-attention and cross-attention maps conditioned on carefully designed textual prompts using the BLIP model, enabling an unbiased evaluation of the model's ability to align images with texts. Additionally, based on DiffDisEval, we introduce an automatic dataset construction pipeline that can mitigate the constraints caused by the predefined label set of an existing image segmentation dataset and generate a dataset to fully characterize the segmentation capability of diffusion models. We formulate a general-purpose evaluation dataset, consisting of 1,000 images from four domains: natural images, medical scans, urban scenes, and aerial views. Evaluating prevalent diffusion models, including Stable Diffusion, Openjourney, and SDXL, reveals a strong correlation between the evaluation results and human preferences, demonstrating the efficacy of assessing generative models through discriminative tasks.

Spatio-temporal neural networks have achieved remarkable success in traffic forecasting by modeling complex temporal dynamics and spatial correlations. These advances are largely due to the combination of essential linear transformations and nonlinear activations. While prior networks have considered the heterogeneous information across nodes and time steps in their linear transformations, they typically apply the same nonlinear activation function to all data. This approach treats representations from different nodes and times as i.i.d. random variables, filtering irrelevant information uniformly and ignoring spatio-temporal heterogeneity. To address this limitation, we propose a novel Spatio-Temporal-aware Rectified Linear Unit (STReLU) as a nonlinear activation function for traffic forecasting. STReLU adaptively constructs the parameters of the activation function by incorporating spatio-temporal information, enabling dynamic filtering of irrelevant information for different inputs. The parameters of STReLU are generated by a hyperfunction that leverages spatial attention to capture node-wise correlations and a timestamp lookup to extract temporal patterns. Extensive experiments on four real-world traffic datasets demonstrate that our STReLU achieves state-of-the-art performance. Specifically, STReLU achieves an improvement of 2.3% in MAE, 1.9% in RMSE, and 1.1% in MAPE compared to the best-performing baselines on PeMSD3, PeMSD4, PeMSD7, and PeMSD8 datasets.

Current financial large language models (FinLLMs) exhibit two major limitations: the absence of standardized evaluation metrics for stock analysis quality and insufficient analytical depth. We address these limitations with two contributions. First, we introduce AnalyScore, a systematic framework for evaluating the quality of stock analysis. Second, we construct Stocksis, an expert-curated dataset designed to enhance the financial analysis capabilities of large language models (LLMs). Building on Stocksis, together with a novel integration framework and quantitative tools, we develop FinSphere, an artificial intelligence (AI) agent that generates professional-grade stock analysis reports. Evaluations with AnalyScore show that FinSphere consistently surpasses general-purpose LLMs, domain-specific FinLLMs, and existing agent-based systems, even when the latter are enhanced with real-time data access and few-shot guidance. The findings highlight FinSphere’s significant advantages in analytical quality and real-world applicability.

Foundation models like the Segment Anything Model (SAM) have shown promising performance in general image segmentation tasks. However, their effectiveness is limited when applied to pathology images due to the inherent multi-scale structural complexity and staining heterogeneity. To address these challenges, we introduce SegAnyPath, a foundational model specifically designed for pathology image segmentation. SegAnyPath is trained on an extensive public pathology dataset comprising over 1.5 million images and 3.5 million masks. We propose a multi-scale proxy task to handle the diverse resolutions in pathology images, complementing the reconstruction objective in the supervised learning stage. To enhance segmentation performance across stain variations, we introduce a novel self-distillation scheme based on stain augmentations. Furthermore, we propose an innovative task-guided Mixture of Experts (MoE) architecture in the decoder of SegAnyPath for efficient management of distinct pathology segmentation tasks, including cell, tissue, and tumor segmentation. Experimental results demonstrate SegAnyPath's zero-shot generalization capability, achieving a Dice score of 0.6797 across multiple datasets and organs while maintaining consistent performance across varying staining styles and resolutions. In comparison, the fine-tuned SAM achieves a Dice score of only 0.5258 on the same external test sets, indicating a substantial 29.27% improvement by SegAnyPath. SegAnyPath has the potential to advance the field of pathology analysis and improve diagnostic accuracy in clinical settings. The code is available at https://github.com/wagnchogn/SegAnyPath.

Existing volumetric medical image segmentation models are typically task-specific, excelling at specific targets but struggling to generalize across anatomical structures or modalities. This limitation restricts their broader clinical use. In this article, we introduce segment anything model (SAM)-Med3D, a vision foundation model (VFM) for general-purpose segmentation on volumetric medical images. Given only a few 3-D prompt points, SAM-Med3D can accurately segment diverse anatomical structures and lesions across various modalities. To achieve this, we gather and preprocess a large-scale 3-D medical image segmentation dataset, SA-Med3D-140K, from 70 public datasets and 8K licensed private cases from hospitals. This dataset includes 22K 3-D images and 143K corresponding masks. SAM-Med3D, a promptable segmentation model characterized by its fully learnable 3-D structure, is trained on this dataset using a two-stage procedure and exhibits impressive performance on both seen and unseen segmentation targets. We comprehensively evaluate SAM-Med3D on 16 datasets covering diverse medical scenarios, including different anatomical structures, modalities, targets, and zero-shot transferability to new/unseen tasks. The evaluation demonstrates the efficiency and efficacy of SAM-Med3D, as well as its promising application to diverse downstream tasks as a pretrained model. Our approach illustrates that substantial medical resources can be harnessed to develop a general-purpose medical AI for various potential applications. Our dataset, code, and models are available at: https://github.com/uni-medical/SAM-Med3D.

The high similarities between protein sequences and natural language, particularly in their sequential data structures, have driven parallel advancements in deep learning models for both domains. In natural language processing (NLP), large language models (LLMs) have achieved remarkable success in tasks such as text generation, translation, and conversational agents, owing to their extensive training on diverse datasets that enable them to capture complex language patterns and generate human-like text. Inspired by these advancements, researchers have attempted to adapt LLMs for protein understanding by integrating a protein sequence encoder with a pre-trained LLM, following designs like LLaVa. However, this adaptation raises a fundamental question: "Can LLMs, originally designed for NLP, effectively comprehend protein sequences as a form of language?" Current datasets fall short in addressing this question due to the lack of a direct correlation between protein sequences and corresponding text descriptions, limiting the ability to train and evaluate LLMs for protein understanding effectively. To bridge this gap, we introduce ProteinLMDataset, a dataset specifically designed for further self-supervised pretraining and supervised fine-tuning (SFT) of LLMs to enhance their capability for protein sequence comprehension. Specifically, ProteinLMDataset includes 17.46 billion tokens for pretraining and 893K instructions for SFT. Additionally, we present ProteinLMBench, the first benchmark dataset consisting of 944 manually verified multiple-choice questions for assessing the protein understanding capabilities of LLMs. ProteinLMBench incorporates protein-related details and sequences in multiple languages, establishing a new standard for evaluating LLMs’ abilities in protein comprehension. The large language model InternLM2-7B, pretrained and fine-tuned on the ProteinLMDataset, outperforms GPT-4 on ProteinLMBench, achieving the highest accuracy score. The dataset and the benchmark are available at https://huggingface. co/datasets/tsynbio/ProteinLMDataset/ and https://huggingface.co/datasets/tsynbio/ProteinLMBench. The code is available at https://github.com/tsynbio/ProteinLMDataset/.

The structural similarities between protein sequences and natural languages have led to parallel advancements in deep learning across both domains. While large language models (LLMs) have achieved much progress in the domain of natural language processing, their potential in protein engineering remains largely unexplored. Previous approaches have equipped LLMs with protein understanding capabilities by incorporating external protein encoders, but this fails to fully leverage the inherent similarities between protein sequences and natural languages, resulting in sub-optimal performance and increased model complexity. To address this gap, we present TourSynbio-7B, the first multi-modal large model specifically designed for protein engineering tasks without external protein encoders. TourSynbio-7B demonstrates that LLMs can inherently learn to understand proteins as language. The model is post-trained and instruction fine-tuned on InternLM2-7B using ProteinLM-Dataset, a dataset comprising 17.46 billion tokens of text and protein sequence for self-supervised pretraining and 893K instructions for supervised fine-tuning. TourSynbio7B outperforms GPT-4 on the ProteinLMBench, a benchmark of 944 manually verified multiple-choice questions, with 62.18% accuracy. Leveraging TourSynbio-7B’s enhanced protein sequence understanding capability, we introduce TourSynbioAgent, an innovative framework capable of performing various protein engineering tasks, including mutation analysis, inverse folding, protein folding, and visualization. TourSynbio-Agent integrates previously disconnected deep learning models in the protein engineering domain, offering a unified conversational user interface for improved usability. Finally, we demonstrate the efficacy of TourSynbio-7B and TourSynbio-Agent through two wet lab case studies on vanilla key enzyme modification and steroid compound catalysis. Our results show that this combination facilitates protein engineering tasks in wet labs, leading to higher positive rates, improved mutations, shorter delivery times, and increased automation. The model weights are available at https://huggingface.co/tsynbio/Toursynbio and codes at https://github.com/tsynbio/TourSynbio.

The apparition of multimodal solutions helped closing this breach by teaching AI solutions how to use multiple inputs to infer a given task. Elevating AI algorithms to a higher level by increasing robustness and accuracy. This approach also made models more human, by learning from multiple sources just as how clinicians do. However, such models are still far from being flexible due to mostly be constrained into one singular task, vastly differing from real clinicians who are in charge of broad fields with multiple tasks to perform. This niche is perfect for AI algorithms to grow. However, existing AI models lack flexibility to act similarly to doctors. Thus, some innovations are required to overcome this rigidity. One possible solution would be the emergence of Medical Artificial Intelligence (GMAI) seeks to evolve the current paradigm by aspiring to emulate more humanlike intelligence in medical applications. The emergence of these "Virtual Doctors" has the potential to significantly transform the healthcare paradigm, offering convenient healthcare solutions with just a few clicks at any time of the day. The focal point of this paper is to delineate the prospective applications of GMAI within the field of telemedicine, positioned as ‘teledoctors’, and to outline the existing technological gap between these innovative models and their conventional counterparts. More specifically, we envision GMAI as a ‘Virtual Doctor,’ functioning as an adaptable telemedicine system capable of addressing a broad array of inquiries and providing diagnoses for a diverse range of medical conditions by utilizing patient- or doctor-provided data inputs. While a general AI model for medical imaging may offer some benefits, it is vital to consider such a model’s potential drawbacks and limitations. It is crucial to balance the benefits of AI and the need for human expertise and oversight in medical imaging.

Recent advancements in Large Language Models (LLMs) have enhanced efficiency across various domains, including protein engineering, where they offer promising opportunities for dry lab and wet lab experiment workflow automation. Previous work, namely TourSynbio-Agent, integrates a protein-specialized multimodal LLM (i.e. TourSynbio-7B) with domain-specific deep learning (DL) models to streamline both computational and experimental protein engineering tasks. While initial validation demonstrated TourSynbio-7B’s fundamental protein property understanding, the practical effectiveness of the complete TourSynbio-Agent framework in real-world applications remained unexplored. This study presents a comprehensive validation of TourSynbio-Agent through five diverse case studies spanning both computational (dry lab) and experimental (wet lab) protein engineering. In three computational case studies, we evaluate the TourSynbio-Agent’s capabilities in mutation prediction, protein folding, and protein design. Additionally, two wet-lab validations demonstrate TourSynbio-Agent’s practical utility: engineering P450 proteins with up to 70% improved selectivity for steroid 19-hydroxylation, and developing reductases with 3.7× enhanced catalytic efficiency for alcohol conversion. Our findings from the five case studies establish that TourSynbioAgent can effectively automate complex protein engineering workflows through an intuitive conversational interface, potentially accelerating scientific discovery in protein engineering.

Medical image segmentation is crucial for diagnosis and treatment planning. While recent advancements in deep learning, particularly UNet variants, have improved segmentation performance, they often result in increased model complexity, which limits their real-time applicability on resource-constrained devices in clinical settings. To address this challenge, we present the Tiny Efficient U-Net (TE-UNet), a novel lightweight model balancing efficiency and accuracy. TE-UNet uses a U-shaped encoder-decoder framework with a gated linear attention mechanism to process low-level and high-level features, preserving details and reducing complexity. It employs depth-wise separable convolutions for higher-level processing, enhancing efficiency without losing performance. Additionally, skip connections improve multi-scale feature extraction and information flow. Experiments on two public datasets across different modalities demonstrate that TE-UNet outperforms ten state-of-the-art methods, maintaining a parameter size under 40KB and low computational cost. TE-UNet makes real-time segmentation more accessible for various clinical applications.

The exponential growth in protein-related databases and scientific literature, combined with increasing demands for efficient biological information retrieval, has created an urgent need for unified and accessible search methods in protein engineering research. We present TourSynbio-Search, a novel bioinformatics search agent framework powered by the TourSynbio7B protein multimodal large language model (LLM), designed to address the growing challenges of information retrieval across rapidly expanding protein databases and corresponding online research literature. The agent’s dual-module architecture consists of PaperSearch and ProteinSearch components, enabling comprehensive exploration of both scientific literature and protein data across multiple biological databases. At its core, TourSynbioSearch employs an intelligent agent system that interprets natural language queries, optimizes search parameters, and executes search operations across major platforms including UniProt, PDB, ArXiv, and BioRxiv The agent’s ability to process intuitive natural language queries reduces technical barriers, allowing researchers to efficiently access and analyze complex biological data without requiring extensive bioinformatics expertise. Through detailed case studies in literature retrieval and protein structure visualization, we demonstrate TourSynbio-Search’s effectiveness in streamlining biological information retrieval and enhancing research productivity. This framework represents an advancement in bridging the accessibility gap between complex biological databases and researchers, potentially accelerating progress in protein engineering applications. Our codes are available at https://github.com/tsynbio/Toursynbio-Search.

Accurate cell nuclei segmentation is crucial for characterizing cell morphology and elucidating disease types in medical image analysis. However, fully-supervised segmentation models often have limited generalization to unseen classes or domains due to being trained on specific modalities or cell types. The recent Segment Anything Model (SAM) has shown promise for interactive instance segmentation and zero-shot generalization. However, its effectiveness is hindered by the sparse and distributed nature of nuclei. It also requires a large number of user prompts that scale with the number of nuclei in the image. To overcome these challenges, we introduce Segment Any Nuclei (SAN), a novel foundation model tailored specifically for nuclei segmentation. SAN is trained on an extensive multi-modal dataset containing diverse nuclei instances from various imaging modalities, staining methods, and tissue types. Unlike previous SAM approaches that rely on manual prompts, SAN incorporates an innovative auto-prompting auxiliary segmentation network, which enables the model to make predictions without manual intervention while still allowing for manual interaction when needed. We evaluate SAN on a large dataset of 9,244 images and demonstrate state-of-the-art nuclei segmentation performance, surpassing both fully-supervised approaches and other SAM-like models. SAN represents a step towards more generalizable, efficient and interactive nuclei segmentation. Our code is available at https://github.com/Mrliujunwen/SAN.

Protein inverse folding aims to identify viable amino acid sequences that can fold into given protein structures, enabling the design of novel proteins with desired functions for applications in drug discovery, enzyme engineering, and biomaterial development. Diffusion probabilistic models have emerged as a promising approach in inverse folding, offering both feasible and diverse solutions compared to traditional energy-based methods and more recent protein language models. However, existing diffusion models for protein inverse folding operate in discrete data spaces, necessitating prior distributions for transition matrices and limiting smooth transitions and gradients inherent to continuous spaces, leading to suboptimal performance. Drawing inspiration from the success of diffusion models in continuous domains, we introduce the Latent Graph Diffusion Model for Protein Inverse Folding (LaGDif). LaGDif bridges discrete and continuous realms through an encoder-decoder architecture, transforming protein graph data distributions into random noise within a continuous latent space. Our model then reconstructs protein sequences by considering spatial configurations, biochemical attributes, and environmental factors of each node. Additionally, we propose a novel inverse folding self-ensemble method that stabilizes prediction results and further enhances performance by aggregating multiple denoised output protein sequence. Empirical results on the CATH dataset demonstrate that LaGDif outperforms existing state-of-the-art techniques, achieving up to 45.55% improvement in sequence recovery rate for single-chain proteins and maintaining an average RMSD of 1.96 Å between generated and native structures. These advancements of LaGDif in protein inverse folding have the potential to accelerate the development of novel proteins for therapeutic and industrial applications. The code is public available at https://github.com/TaoyuW/LaGDif.

Accurate nuclei segmentation is fundamental in histology image analysis, playing an essential role in cancer grading and diagnosis. However, this task remains challenging due to variations in staining protocols, heterogeneity among nuclei types, and the densely clustered nature of nuclei. While SAM exhibits zero-shot generalization capabilities in natural image segmentation, its performance degrades when applied to nuclei segmentation in histology images. Existing adaptations of SAM for medical imaging primarily focus on organ or lesion segmentation, which differs substantially from nuclei segmentation due to the unique characteristics of nuclei—specifically, their sparse distribution combined with dense clustering. To address these challenges, we propose Edge-SAN (Segment Any Nuclei with Edge Prompting), an interactive segmentation foundation model specifically designed for nuclei segmentation. Edge-SAN introduces a novel edge prompting method that enhances the delineation of nuclei boundaries, particularly among densely clustered nuclei, by leveraging edge information to improve segmentation accuracy. We evaluate Edge-SAN on 12 diverse datasets in both few-shot and zero-shot scenarios, demonstrating its effectiveness as a foundation model for nuclei segmentation, achieving 66.81% AJI and 73.13% DSC—improvements of 16.33% and 15.48% over SAM-Med2D, respectively. The code is available at https://github.com/deep-geo/Edge-SAN.

Large Language Models (LLMs) have demonstrated zero-shot generalization capabilities in analyzing and predicting protein properties through natural language interactions. However, existing protein-focused datasets for LLM fine-tuning, such as ProteinLMDataset with its 17.46 billion tokens for pre-training and 893,000 instructions for fine-tuning, face limitations. Specifically, they include insufficient coverage of protein functional properties, inadequate protein-protein interaction data, and limited integration of contextual information from biomedical literature. To overcome these challenges, we present ProteinPFAIDataset, which integrates data from UniProt and PubMed, comprising 72.8 million tokens for Supervised Fine-Tuning (SFT). ProteinPFAIDataset encompasses important protein characteristics including enzyme activities, molecular functions, pH dependence, tissue specificity, temperature sensitivity, subunit structure, and disease associations. Additionally, we propose a novel knowledge graph-based approach that incorporates over 300,000 biomedical literature entries, providing rich contextual information about protein functions and interactions. To validate the effectiveness of our dataset, we fine-tuned Falcon2-11B LLM, resulting in a model we call Falcon2-11B-PFAI. The fine-tuned model achieved state-of-the-art performance on ProteinLMBench, improving accuracy from 47.10% to 58.37%. The dataset is available at https://huggingface.co/datasets/xiaorui1/PFAI. The fine-tuned model is available at https://huggingface.co/xiaorui1/PFAI/tree/main.

Enzymes are biological catalysts that can accelerate chemical reactions compared to uncatalyzed reactions in aqueous environments. Their catalytic efficiency is quantified by the turnover number (kcat), a parameter in enzyme kinetics. Enhancing enzyme activity is important for optimizing slow chemical reactions, with far-reaching implications for both research and industrial applications. However, traditional wet-lab methods for measuring and optimizing enzyme activity are often resource-intensive and time-consuming. To address these limitations, we introduce kcatDiffuser, a novel regressor-guided diffusion model designed to predict and improve enzyme turnover numbers. Our approach innovatively reformulates enzyme mutation prediction as a protein inverse folding task, thereby establishing a direct link between structural prediction and functional optimization. kcatDiffuser is a graph diffusion model guided by a regressor, enabling the prediction of amino acid mutations at multiple random positions simultaneously. Evaluations on BERENDA dataset shows that kcatDiffuser can achieve a ∆logkcatof 0.209, outperforming state-of-the-art methods like ProteinMPNN, PiFold, GraDe-IF in improving enzyme turnover numbers. Additionally, kcatDiffuser maintains high structural fidelity with a recovery rate of 0.716, pLDDT score of 92.515, RMSD of 3.764, and TM-score of 0.934, demonstrating its ability to generate enzyme variants with enhanced activity while preserving essential structural properties. Overall, kcatDiffuser represents a more efficient and targeted approach to enhancing enzyme activity. The code is available at https://github.com/xz32yu/KcatDiffuser.

Microsatellite instability (MSI), characterized by alterations in short tandem repeat sequences throughout the genome, is an important biomarker for cancer diagnosis, prognosis, and treatment planning. Traditional MSI detection methods, such as immunohistochemistry (IHC) and polymerase chain reaction (PCR), are expensive, time-consuming, and resource-intensive. Recent advances in deep learning have enabled direct MSI prediction from hematoxylin and eosin (H&E) stained whole slide images (WSIs), offering a potentially faster and more cost-effective approach. However, existing convolutional neural network (CNN)-based methods for MSI prediction often focus on patch-level prediction and therefore struggle to capture the complex, multi-scale nature of histological features and their spatial relationships. To address these limitations, we propose a novel three-level multiscale graph neural network (GNN) method that integrates tissue, patch, and cellular-level analyses within a unified framework, namely TPC-GNN. The TPC-GNN extracts comprehensive biological information from WSIs by simulating structural interactions across various scales. At the tissue level, it captures global morphological patterns and tissue organization. The patch level focuses on local texture and architectural features, while the cellular level analyzes individual cell morphology and distribution. This multi-scale approach enables the model to capture complex biomarkers from micro to macro levels, providing a more holistic representation of the histological image. To enhance the integration of multi-scale information, we investigate different learnable fusion schemes based on multi-layer perceptrons (MLP), Transformers, and Mamba architectures. Experimental results demonstrate the superiority of our approach, achieving performance improvements of over 2% and 4% in Area Under the Curve (AUC) on the CRC-MSI and STAD-MSI datasets, respectively, across different network backbones compared to existing methods. These findings not only validate the effectiveness of our multiscale GNN approach but also highlight its application to complex histopathological-level image analysis. Codes are available at https://github.com/zhillusion/TPC-GNN.

Tabular data plays a crucial role in various domains but often suffers from missing values, thereby curtailing its potential utility. Traditional imputation techniques frequently yield suboptimal results and impose substantial computational burdens, leading to inaccuracies in subsequent modeling tasks. To address these challenges, we propose DiffImpute, a novel Denoising Diffusion Probabilistic Model (DDPM). Specifically, DiffImpute is trained on complete tabular datasets, ensuring that it can produce credible imputations for missing entries without undermining the authenticity of the existing data. Innovatively, it can be applied to various settings of Missing Completely At Random (MCAR) and Missing At Random (MAR). To effectively handle the tabular features in DDPM, we tailor four tabular denoising networks, spanning MLP, ResNet, Transformer, and U-Net. We also propose Harmonization to enhance coherence between observed and imputed data by infusing the data back and de-noising them multiple times during the sampling stage. To enable efficient inference while maintaining imputation performance, we propose a refined non-Markovian sampling process that works along with Harmonization. Empirical evaluations on seven diverse datasets underscore the prowess of DiffImpute. Specifically, when paired with the Transformer as the denoising network, it consistently outperforms its competitors, boasting an average ranking of 1.7 and the most minimal standard deviation. In contrast, the next best method lags with a ranking of 2.8 and a standard deviation of 0.9. The code is available at https://github.com/Dendiiiii/DiffImpute.

Images captured by robotics under low-light conditions are often plagued by several challenges, including diminished contrast, increased noise, loss of fine details, and unnatural color reproduction. These factors can significantly hinder the performance of computer vision tasks such as object detection and image segmentation. As a result, improving the quality of low-light images is of paramount importance for practical applications in the computer vision domain. To effectively address these challenges, we present a novel low-light image enhancement model, termed Spatial Consistency Retinex Network (SCRNet), which leverages the Retinex-based structure and is guided by the principle of spatial consistency. Specifically, our proposed model incorporates three levels of consistency: channel level, semantic level, and texture level, inspired by the principle of spatial consistency. These levels of consistency enable our model to adaptively enhance image features, ensuring more accurate and visually pleasing results. Extensive experimental evaluations on various low-light image datasets demonstrate that our proposed SCRNet outshines existing state-of-the-art methods, highlighting the potential of SCRNet as an effective solution for enhancing low-light images.

Knowledge distillation has been successfully applied to various pattern recognition and computer vision tasks, enabling the transfer of knowledge from a pre-trained teacher model to a student model or mutual learning between peer student models. However, most existing works focus on either offline or online distillation independently, despite the potential benefits of combining both approaches to address the complexity of real-world applications. To bridge the gap between offline and online distillation, we propose a novel unified framework called Knowledge Inheritance and Evolution (KnowledgeIE), inspired by human learning behaviors. KnowledgeIE seamlessly integrates offline and online distillation modes, allowing the student model to learn from both a pre-trained teacher and its peers during training. We evaluate the effectiveness of KnowledgeIE on several image classification benchmarks and demonstrate consistent improvements over baseline methods. Our comprehensive experiments on three datasets show that the unified architecture outperforms both standalone offline and online distillation, highlighting the strong compatibility and synergy achieved by combining the two approaches. KnowledgeIE presents a promising direction for knowledge distillation, offering a more flexible and effective framework for transferring knowledge between models in various application scenarios.

Automated Machine Learning (AutoML) offers a promising approach to streamline the training of machine learning models. However, existing AutoML frameworks are often limited to unimodal scenarios and require extensive manual configuration. Recent advancements in Large Language Models (LLMs) have showcased their exceptional abilities in reasoning, interaction, and code generation, presenting an opportunity to develop a more automated and user-friendly framework. To this end, we introduce AutoM3L, an innovative Automated Multimodal Machine Learning framework that leverages LLMs as controllers to automatically construct multimodal training pipelines. AutoM3L comprehends data modalities and selects appropriate models based on user requirements, providing automation and interactivity. By eliminating the need for manual feature engineering and hyperparameter optimization, our framework simplifies user engagement and enables customization through directives, addressing the limitations of previous rule-based AutoML approaches. We evaluate the performance of AutoM3L on six diverse multimodal datasets spanning classification, regression, and retrieval tasks, as well as a comprehensive set of unimodal datasets. The results demonstrate that AutoM3L achieves competitive or superior performance compared to traditional rule-based AutoML methods. Furthermore, a user study highlights the user-friendliness and usability of our framework, compared to the rule-based AutoML methods.

The wide prevalence of staining variations in digital pathology presents a significant obstacle, often undermining the effectiveness of diagnosis and analysis. The current strategies to counteract this issue primarily revolve around Stain Normalization (SN) and Stain Augmentation (SA). Nonetheless, these methodologies come with inherent limitations. They struggle to adapt to the vast array of staining styles, tend to presuppose linear associations between color spaces, and often lead to unrealistic color transformations. In response to these challenges, we introduce RandStainNA++, a novel method seamlessly integrating SN and SA. This method exploits the versatility of random SN and SA within randomly selected color spaces, effectively managing variations for the foreground and background independently. By refining the transformations of staining styles for the foreground and background within a realistic scope, this strategy promotes the generation of more practical staining transformations during the training phase. Further enhancing our approach, we propose a unique self-distillation method. This technique incorporates prior knowledge of stain variation, substantially augmenting the generalization capability of the network. The striking results yield that, compared to conventional classification models, our method boosts performance by a significant margin of 16-25%. Furthermore, when juxtaposed with baseline segmentation models, the Dice score registers an increase of 0.06.

Histopathology plays an indispensable role in cancer diagnosis. However, the frequent staining variations observed in tissue slides pose significant challenges to the effectiveness of computer-aided diagnostic systems. Stain Normalization (SN) and Stain Augmentation (SA) have been proposed to mitigate these issues. Traditional SN techniques focused on minimizing variations between slides, whereas SA sought to enhance data distribution without changing the inherent morphology, mainly within the image domain. Notably, many medical signals originate from frequency information, often processed in specialized facilities. Motivated by this, we introduce a novel approach: harnessing the frequency domain for SN and SA. Our proposed Fourier Stain Normalization (F-SN) and Fourier Stain Augmentation (FSA) methods innovatively mark a significant departure from conventional techniques by leveraging the focus on the frequency domain. Through the application of Fourier transformation, we transmute time-domain signals into the frequency domain, resulting in a streamlined and more efficient preprocessing phase compared to time-domain-based methodologies. Enhancing this approach, we incorporate a learnable mask prediction mechanism, adeptly merging source and target images, which bolsters its adaptability for a spectrum of downstream tasks. Comprehensive evaluations spanning medical image classification and segmentation using diverse network architectures, affirm the superiority of our F-SN and F-SA techniques over traditional time-domain methods, underscoring the frequency domain’s richer informational content. The code is available at https://github.com/Windyskys/FSNFSA.

Gigapixel medical images are a rich source of information containing both morphological textures and spatial information. However, existing deep learning solutions primarily rely on convolutional neural networks (CNNs) for global pixel-level analysis, ignoring the underlying local geometric structure. Since the topological structure in medical images is closely related to tumor evolution, graphs can be utilized to characterize it. To obtain a more comprehensive representation for downstream analysis, a fusion framework is proposed to enhance the global image-level representation captured by CNNs with the geometry of cell-level spatial information learned by graph neural networks (GNN). Two fusion strategies have been developed: one with MLP, which is simple but efficient through fine-tuning, and the other with TRANSFORMER, which excels in fusing multiple networks. The proposed fusion strategies have been evaluated on histology datasets from large patient cohorts of colorectal and gastric cancers for three biomarker prediction tasks. Both models outperform plain CNNs or GNNs, achieving a consistent AUC improvement of more than 5% on various network backbones. Importantly, the experimental results demonstrate the necessity of combining image-level morphological features with cell spatial relations in medical image analysis. Codes are available at https://github.com/yiqings/HEGnnEnhanceCnn.

To address the challenge of diagnosing histology images accurately, we propose a novel Transformer-based model, HistDeblur, to detect and eliminate blur in digitized microscope images. HistDeblur adopts a trustworthy deblurring pyramid architecture with a shared-weight structure for artifact restoration, ensuring image quality control. To the best of our knowledge, HistDeblur is the first attempt at a Transformer based model to detect and eliminate the blur in histology images. Uniquely, it incorporates an auxiliary branch in the decoder for explainable blurriness estimation, diverging from traditional pyramid models. A method is also introduced to synthesize non-uniform blurry images, simulating real-world artifacts occurring during specimen preparation and digitization. Experiments on a representative subset of CRC-VAL-HE-7K [1] show the proposed framework is superior to the state-of-the-art approaches in the context of histology deblurring by a margin of 1.97 PSNR and 0.03 SSIM respectively. HistDeblur, being fully automated, optimizes histology image classification, quality assessment, and restoration, thereby mimicking the histologist’s examination experience proficiently. The approach facilitates reliable medical diagnoses, allowing histologists to make informed decisions on accepting deblurred images or opting for costly resampling and re-scanning. The source code is available at https://github.com/KKK-Liu/HistDeblur

Mixed Sample Regularization (MSR), such as MixUp or CutMix, is a powerful data augmentation strategy to generalize convolutional neural networks. Previous empirical analysis has illustrated an orthogonal performance gain between MSR and conventional offline Knowledge Distillation (KD). To be more specific, student networks can be enhanced with the involvement of MSR in the training stage of sequential distillation. Yet, the interplay between MSR and online knowledge distillation, where an ensemble of peer students learn mutually from each other, remains unexplored. To bridge the gap, we make the first attempt at incorporating CutMix into online distillation, where we empirically observe a significant improvement. Encouraged by this fact, we propose an even stronger MSR specifically for online distillation, named as CutnMix. Furthermore, a novel online distillation framework is designed upon CutnMix, to enhance the distillation with feature level mutual learning and a self-ensemble teacher. Comprehensive evaluations on CIFAR10 and CIFAR100 with six network architectures show that our approach can consistently outperform state-of-the-art distillation methods.

Inverse protein folding is challenging due to its inherent one-to-many mapping characteristic, where numerous possible amino acid sequences can fold into a single, identical protein backbone. This task involves not only identifying viable sequences but also representing the sheer diversity of potential solutions. However, existing discriminative models, such as transformer-based auto-regressive models, struggle to encapsulate the diverse range of plausible solutions. In contrast, diffusion probabilistic models, as an emerging genre of generative approaches, offer the potential to generate a diverse set of sequence candidates for determined protein backbones. We propose a novel graph denoising diffusion model for inverse protein folding, where a given protein backbone guides the diffusion process on the corresponding amino acid residue types. The model infers the joint distribution of amino acids conditioned on the nodes' physiochemical properties and local environment. Moreover, we utilize amino acid replacement matrices for the diffusion forward process, encoding the biologically-meaningful prior knowledge of amino acids from their spatial and sequential neighbors as well as themselves, which reduces the sampling space of the generative process. Our model achieves state-of-the-art performance over a set of popular baseline methods in sequence recovery and exhibits great potential in generating diverse protein sequences for a determined protein backbone structure.

Currently, data-driven based machine learning is considered one of the best choices in clinical pathology analysis, and its success is subject to the sufficiency of digitized slides, particularly those with deep annotations. Although centralized training on a large data set may be more reliable and more generalized, the slides to the examination are more often than not collected from many distributed medical institutes. This brings its own challenges, and the most important is the assurance of privacy and security of incoming data samples. In the discipline of histopathology image, the universal stain-variation issue adds to the difficulty of an automatic system as different clinical institutions provide distinct stain styles. To address these two important challenges in AI-based histopathology diagnoses, this work proposes a novel conditional Generative Adversarial Network (GAN) with one orchestration generator and multiple distributed discriminators, to cope with multiple-client based stain-style normalization. Implemented within a Federated Learning (FL) paradigm, this framework well preserves data privacy and security. Additionally, the training consistency and stability of the distributed system are further enhanced by a novel temporal self-distillation regularization scheme. Empirically, on large cohorts of histopathology datasets as a benchmark, the proposed model matches the performance of conventional centralized learning very closely. It also outperforms state-of-the-art stain-style transfer methods on the downstream Federated Learning image classification task, with an accuracy increase of over 20.0% in comparison to the baseline classification model.

The artifacts in histology images may encumber the accurate interpretation of medical information and cause misdiagnosis. Accordingly, prepending manual quality control of artifacts considerably decreases the degree of automation. To close this gap, we propose a methodical pre-processing framework to detect and restore artifacts, which minimizes their impact on downstream AI diagnostic tasks. First, the artifact recognition network AR-Classifier first differentiates common artifacts from normal tissues, e.g., tissue folds, marking dye, tattoo pigment, spot, and out-of-focus, and also catalogs artifact patches by their restorability. Then, the succeeding artifact restoration network AR-CycleGAN performs de-artifact processing where stain styles and tissue structures can be maximally retained. We construct a benchmark for performance evaluation, curated from both clinically collected WSIs and public datasets of colorectal and breast cancer. The functional structures are compared with state-of-the-art methods, and also comprehensively evaluated by multiple metrics across multiple tasks, including artifact classification, artifact restoration, downstream diagnostic tasks of tumor classification and nuclei segmentation. The proposed system allows full automation of deep learning based histology image analysis without human intervention. Moreover, the structure-independent characteristic enables its processing with various artifact subtypes. The source code and data in this research are available at https://github.com/yunboer/AR-classifier-and-AR-CycleGAN.

Numerous patch-based methods have recently been proposed for histological image based breast cancer classification. However, their performance could be highly affected by ignoring spatial contextual information in the whole slide image (WSI). To address this issue, we propose a novel hierarchical Graph V-Net by integrating 1) patch-level pre-training and 2) context-based fine-tuning, with a hierarchical graph network. Specifically, a semi-supervised framework based on knowledge distillation is first developed to pre-train a patch encoder for extracting disease-relevant features. Then, a hierarchical Graph V-Net is designed to construct a hierarchical graph representation from neighboring/similar individual patches for coarse-to-fine classification, where each graph node (corresponding to one patch) is attached with extracted disease-relevant features and its target label during training is the average label of all pixels in the corresponding patch. To evaluate the performance of our proposed hierarchical Graph V-Net, we collect a large WSI dataset of 560 WSIs, with 30 labeled WSIs from the BACH dataset (through our further refinement), 30 labeled WSIs and 500 unlabeled WSIs from Yunnan Cancer Hospital. Those 500 unlabeled WSIs are employed for patch-level pre-training to improve feature representation, while 60 labeled WSIs are used to train and test our proposed hierarchical Graph V-Net. Both comparative assessment and ablation studies demonstrate the superiority of our proposed hierarchical Graph V-Net over state-of-the-art methods in classifying breast cancer from WSIs. The source code and our annotations for the BACH dataset have been released at https://github.com/lyhkevin/Graph-V-Net.

Chest X-ray (CXR) is crucial to the early diagnosis and treatment of pneumonia in children while still facing the viral and bacterial pneumonia identification challenge. Although convolutional neural networks (CNNs) can identify the concealed features between viral and bacterial pneumonia, the influence on the network accuracy led by morphological lesion size variation and confounded visual imaging findings is not explicitly addressed. To address these issues, we propose a novel framework MFA-DSLNet to classify viral and bacterial CXR in childhood pneumonia. It comprises two functional blocks. First, a Multiscale Fusion Attention (MFA) mechanism fuses low-level and high-level features to enrich the information in feature maps. Then, two attention modules, namely Context Attention (CA) and Spatial Position Attention (SPA), attempt to extract effective latent information from the channel and spatial dimension respectively to tackle the redundancy problem in feature maps. To facilitate the identification of hard cases, Dual-threshold Self-paced Learning (DSL) is proposed to dynamically adapt the training sequence of samples. Empirically, MFA-DSLNet achieves a state-of-the-art 93.33% accuracy, with a 2.33% improvement to the previous works on the ChestXray2017 dataset.

Chronic Obstructive Pulmonary Disease (COPD) is the fourth leading cause of death worldwide. Yet, COPD diagnosis heavily relies on spirometric examination as well as functional airway limitation, which may cause a considerable portion of COPD patients underdiagnosed especially at the early stage. Recent advance in deep learning (DL) has shown their promising potential in COPD identification from CT images. However, with heterogeneous syndromes and distinct phenotypes, DL models trained with CTs from one data center fail to generalize on images from another center. Due to privacy regularizations, a collaboration of distributed CT images into one centralized center is not feasible. Federated learning (FL) approaches enable us to train with distributed private data. Yet, routine FL solutions suffer from performance degradation in the case where COPD CTs are not independent and identically distributed (Non-IID). To address this issue, we propose a novel personalized federated learning (PFL) method based on vision transformer (ViT) for distributed and heterogeneous COPD CTs. To be more specific, we partially personalize some heads in multiheaded self-attention layers to learn the personalized attention for local data and retain the other heads shared to extract the common attention. To the best of our knowledge, this is the first proposal of a PFL framework specifically for ViT to identify COPD. Our evaluation of a dataset set curated from six medical centers shows our method outperforms the PFL approaches for convolutional neural networks.

Knowledge distillation (KD) shows a bright promise as a powerful regularization strategy to boost generalization ability by leveraging learned sample-level soft targets. Yet, employing a complex pre-trained teacher network or an ensemble of peer students in existing KD is both timeconsuming and computationally costly. Various self KD methods have been proposed to achieve higher distillation efficiency. However, they either require extra network architecture modification or are difficult to parallelize. To cope with these challenges, we propose an efficient and reliable self-distillation framework, named Self-Distillation from Last Mini-Batch (DLB). Specifically, we rearrange the sequential sampling by constraining half of each mini-batch coinciding with the previous iteration. Meanwhile, the rest half will coincide with the upcoming iteration. Afterwards, the former half mini-batch distills on-the-fly soft targets generated in the previous iteration. Our proposed mechanism guides the training stability and consistency, resulting in robustness to label noise. Moreover, our method is easy to implement, without taking up extra run-time memory or requiring model structure modification. Experimental results on three classification benchmarks illustrate that our approach can consistently outperform state-of-the-art self-distillation approaches with different network architectures. Additionally, our method shows strong compatibility with augmentation strategies by gaining additional performance improvement. The code is available at https://github.com/Meta-knowledge-Lab/DLB.

Federated learning (FL) is a distributed learning paradigm that enables multiple clients to collaboratively learn a shared global model. Despite the recent progress, it remains challenging to deal with heterogeneous data clients, as the discrepant data distributions usually prevent the global model from delivering good generalization ability on each participating client. In this paper, we propose CD2-pFed, a novel Cyclic Distillation-guided Channel Decoupling framework, to personalize the global model in FL, under various settings of data heterogeneity. Different from previous works which establish layer-wise personalization to overcome the non-IID data across different clients, we make the first attempt at channel-wise assignment for model personalization, referred to as channel decoupling. To further facilitate the collaboration between private and shared weights, we propose a novel cyclic distillation scheme to impose a consistent regularization between the local and global model representations during the federation. Guided by the cyclical distillation, our channel decoupling framework can deliver more accurate and generalized results for different kinds of heterogeneity, such as feature skew, label distribution skew, and concept shift. Comprehensive experiments on four benchmarks, including natural image and medical image analysis tasks, demonstrate the consistent effectiveness of our method on both local and external validations.

Histopathological identification of tumor tissue is one of the routine pathological diagnoses for pathologists. Recently, computational pathology has been successfully interpreted by a variety of deep learning-based applications. Nevertheless, the high-efficient and spatial-correlated processing of individual patches have always attracted attention in whole-slide image (WSI) analysis. In this paper, we propose a high-throughput system to detect tumor regions in colorectal cancer histology slides precisely. We train a deep convolutional neural network (CNN) model and design a Monte Carlo (MC) adaptive sampling method to estimate the most representative patches in a WSI. Two conditional random field (CRF) models are designed, namely the correction CRF and the prediction CRF are integrated for spatial dependencies of patches. We use three datasets of colorectal cancer from The Cancer Genome Atlas (TCGA) to evaluate the performance of the system. The overall diagnostic time can be reduced from 56.7 percent to 71.7 percent on the slides of a varying tumor distribution, with an increase in classification accuracy.

Pathology analysis is crucial to precise cancer diagnoses and the succeeding treatment plan as well. To detect abnormality in histopathology images with prevailing patch-based convolutional neural networks (CNNs), contextual information often serves as a powerful cue. However, as whole-slide images (WSIs) are characterized by intense morphological heterogeneity and extensive tissue scale, a straightforward visual span to a larger context may not well capture the information closely associated with the focal patch. In this paper, we propose a novel pixel-offset based patch-location method to identify high-representative tissues, with a CNN backbone. Pathology Deformable Conditional Random Field (PDCRF) is proposed to learn the offsets and weights of neighboring contexts in a spatial-adaptive manner, to search for high-representative patches. A CNN structure with the localized patches as training input is then capable of consistently reaching superior classification outcomes for histology images. Overall, the proposed method has achieved state-of-the-art performance, in terms of the test classification accuracy improvement to the baseline by 1.15-2.60%, 0.78-1.78%, and 1.47-2.18% on TCGA public datasets of TCGA-STAD, TCGA-COAD, and TCGA-READ respectively. It also achieves 88.95% test accuracy and 0.920 test AUC on Camelyon 16. To show the effectiveness of the proposed framework on downstream tasks, we take a further step by incorporating an active learning model, which noticeably reduces the number of manual annotations by PDCRF to reach a parallel patch-based histology classifier.

The massive annotation workload has always hindered the progress towards an automatic analysis of gigapixel whole-slide images. Histologically, individual patches from a constrained spatial region may share rich phenotypic information, where the morphological correlations have the potential to be mined for a grouping or clustering task. In this paper, we propose a clustering technique to extract multiple mutual information from histology images without prior domain knowledge. Specifically, our framework automatically localizes morphologically homogeneous patches within an extended solution space. Our novelty is an expanse and the pattern with which invariant information can be learnt, in contrast to the current literature of feature generation or parametric transformation within an individual patch. Additionally, structure-independent, the model may be applicable to any backbone convolutional neural network architectures. The empirical validation on The Cancer Genome Atlas (TCGA) datasets illustrates an observable margin of patch-level classification accuracy in comparison with state-of-the-art unsupervised approaches.

Expensive annotation cost has always been a critical obstacle in deep learning systems, in particular for the applications requiring domain experts’ knowledge, such as medical image analysis. Active learning has attracted widespread attention by decreasing the quantity and difficulty in annotation with query strategies. To make a query for labeling, contextual features of candidates’ relation are often considered essential. In this paper, we propose an innovative method that incorporating spatial distribution approximation in the uncertainty sampling for whole-slide histopathology image annotation. The active query selection combines the measure of spatial representativeness and histopathological tissue informativeness. With the assumption that the labeling cost of individual instances is non-biased, we use three independent features, namely loss prediction query, Monte Carlo distribution query, and loss estimation enhanced by spatial information for the active learning task. The experiments were validated on large cohorts of The Cancer Genome Atlas (TCGA) and the ”100,000 histological images of human colorectal cancer and healthy tissue” dataset. Empirically, the proposed method can outperform the other annotation strategies on the histopathology datasets. The annotation cost is reduced by an obvious margin of 40% to retain an accurate classifier. This novel annotation strategy provides the potential to efficiently label and classify histopathology images with a patch-based convolutional neural network.

Microsatellite instability (MSI) is the result of a defective DNA mismatch repair (MMR) system, and its presence occurs in a variety of cancers. The determination of MSI in colorectal cancer (CRC) will have a better prognosis and management of cancer patients. As the routine MSI identification via molecular testing is expensive, time-consuming, and region-restricted, novel methods to detect MSI are of great interest. In this work, we propose a multi-stage convolutional neural network (CNN) based framework to identify MSI status in colorectal cancer patients from histopathological images. A mislabel-aware module is designed to deal with the uncertainty problem in global-local labelling. An auto-grading model is proposed to discriminate patches by the degree of their histopathological correlation with recognizable MSI status, and subsequently aggregate the weights to make slide-level predictions. Our proposed methodology outperforms the existing models in the classification accuracy, and explicitly sorts out patches with representative features. The research outcome has the potential to assist in the interpretation of histopathology as a surrogate for MSI testing and also in the study of recognizable morphology of MSI-H/MSS tumors. Furthermore, this approach can be extended and applied to other cancer types.

In the last decade, computational pathology has attracted notable attention in the deep learning domain. However, even on the state-of-the-art deep learning computing platforms, a high-resolution scanned whole slide image (WSI) still requires reducing into massive patches to be processed, which is very time consuming in real-time diagnosis. In this paper, we propose a high-throughput tumor location system with Monte Carlo adaptive sampling to accelerate WSI analysis. Additionally, we design a dynamic programming framework to incorporate spatial correlation, which can iteratively eliminate false positives or false negatives in the identification or tumor tissues. We use three datasets of colorectal cancer from The Cancer Genome Atlas (TCGA) for performance evaluation. The designed computer-aided system can reduce more than 50% of the diagnostic time on average in the tumor location task, along with a slight increase in accuracy.