The gut–brain axis (GBA) represents a complex, bidirectional communication network between the gut microbiome and the central nervous system, influencing both neurological health and the pathogenesis of various diseases. This review explores the integrative role of neuroimaging and machine learning (ML) in advancing our understanding of microbiota–brain interactions, emphasizing their combined potential to uncover novel biomarkers and therapeutic targets. Neuroimaging techniques, including functional magnetic resonance imaging (MRI), diffusion tensor imaging, and structural MRI, have revealed how gut microbiota imbalances, or dysbiosis, affect key brain networks and structural connectivity, contributing to cognitive dysfunction, emotional disturbances, and neurodegenerative conditions. ML methodologies, including deep learning (DL) and multimodal data fusion, are proving indispensable in extracting meaningful insights from high-dimensional neuroimaging and microbiome datasets. Supervised approaches, such as random forests and deep neural networks, have achieved high accuracy in predicting neurological outcomes based on microbial signatures, while unsupervised learning identifies distinct microbiota–brain connectivity patterns associated with disorders such as autism spectrum disorder and depression. Additionally, explainable AI (XAI) techniques are being increasingly applied to enhance the interpretability of ML-driven biomarker discovery, shedding light on the neuroprotective effects of butyrate-producing bacteria (e.g., Faecalibacterium , Roseburia ) and the potential for neuroinflammation linked to an overabundance of Proteobacteria . These findings point to the transformative potential of combining neuroimaging and ML in precision medicine, offering a new paradigm for the diagnosis and treatment of neurological disorders ranging from irritable bowel syndrome to Alzheimer’s disease. However, challenges related to data harmonization, generalizability across populations, and establishing causal relationships remain, necessitating further research to realize the full clinical potential of this approach.

The convergence of large language models (LLMs) and virtual reality (VR) technologies has led to significant breakthroughs across multiple domains, particularly in healthcare and medicine. Owing to its immersive and interactive capabilities, VR technology has demonstrated exceptional utility in surgical simulation, rehabilitation, physical therapy, mental health, and psychological treatment. By creating highly realistic and precisely controlled environments, VR not only enhances the efficiency of medical training but also enables personalized therapeutic approaches for patients. The convergence of LLMs and VR extends the potential of both technologies. LLM-empowered VR can transform medical education through interactive learning platforms and address complex healthcare challenges using comprehensive solutions. This convergence enhances the quality of training, decision-making, and patient engagement, paving the way for innovative healthcare delivery. This study aims to comprehensively review the current applications, research advancements, and challenges associated with these two technologies in healthcare and medicine. The rapid evolution of these technologies is driving the healthcare industry toward greater intelligence and precision, establishing them as critical forces in the transformation of modern medicine.

Automatic recognition of 3-D objects in a 3-D model by convolutional neural network (CNN) methods has been successfully applied to various tasks, e.g., robotics and augmented reality. Three-dimensional object recognition is mainly performed by analyzing the object using multi-view images, depth images, graphs, or volumetric data. In some cases, using volumetric data provides the most promising results. However, existing recognition techniques on volumetric data have many drawbacks, such as losing object details on converting points to voxels and the large size of the input volume data that leads to substantial 3-D CNNs. Using point clouds could also provide very promising results; however, point-cloud-based methods typically need sparse data entry and time-consuming training stages. Thus, using volumetric could be a more efficient and flexible recognizer for our special case in the School of Medicine, Shanghai Jiao Tong University. In this article, we propose a novel solution to 3-D object recognition from volumetric data using a combination of three compact CNN models, low-cost SparseNet, and feature representation technique. We achieve an optimized network by estimating extra geometrical information comprising the surface normal and curvature into two separated neural networks. These two models provide supplementary information to each voxel data that consequently improve the results. The primary network model takes advantage of all the predicted features and uses these features in Random Forest (RF) for recognition purposes. Our method outperforms other methods in training speed in our experiments and provides an accurate result as good as the state-of-the-art.

Rare diseases, which are severely underrepresented in basic and clinical research, can particularly benefit from machine learning techniques. However, current learning-based approaches usually focus on either mono-modal image data or matched multi-modal data, whereas the diagnosis of rare diseases necessitates the aggregation of unstructured and unmatched multi-modal image data due to their rare and diverse nature. In this study, we therefore propose diagnosis-guided multi-to-mono modal generation networks (TMM-Nets) along with training and testing procedures. TMM-Nets can transfer data from multiple sources to a single modality for diagnostic data structurization. To demonstrate their potential in the context of rare diseases, TMM-Nets were deployed to diagnose the lupus retinopathy (LR-SLE), leveraging unmatched regular and ultra-wide-field fundus images for transfer learning. The TMM-Nets encoded the transfer learning from diabetic retinopathy to LR-SLE based on the similarity of the fundus lesions. In addition, a lesion-aware multi-scale attention mechanism was developed for clinical alerts, enabling TMM-Nets not only to inform patient care, but also to provide insights consistent with those of clinicians. An adversarial strategy was also developed to refine multi- to mono-modal image generation based on diagnostic results and the data distribution to enhance the data augmentation performance. Compared to the baseline model, the TMM-Nets showed 35.19% and 33.56% F1 score improvements on the test and external validation sets, respectively. In addition, the TMM-Nets can be used to develop diagnostic models for other rare diseases.

Abstract Angle closure glaucoma leads to fluid deposition in eye, and intraocular pressure occurs that damage the optic nerve, causes blindness and vision loss. Anterior chamber (AC) evaluation is imperative for determining the risk of angle‐closure. Previously, techniques were dependent on either Pentacam–Scheimpflug that interprets poor visual information, anterior segment optical coherence tomography is injurious to intercede opaque optical structures. Therefore, in this paper, an experimental protocol is designed for detailed disease analysis based on IBM SPSS statistics via ultrasound biomicroscopy which is superior in evaluating deep structures; first, the affected parameter for AC is analysed, and afterwards the direction that needs laser surgery is explored. Experiments are conducted on large‐scale clinical studies from an affiliated hospital in Shanghai, China. The dataset comprised 600 AC images in five directions of 60 subjects. The mean with standard deviation for anterior open distance is 0.158790.096779 mm, 0.158630.081435 mm, and anterior chamber angle is 18.74908.0315, 18.74108.3889 for left and right eye respectively. It is found that anterior chamber angle in the downside of the AC is wider than the upside. However, this decision is partly based on the narrowest part of the angle to widen the depth of the direction and eliminate pupil block.

Automatic vertebra segmentation from computed tomography (CT) image is the very first and a decisive stage in vertebra analysis for computer-based spinal diagnosis and therapy support system. However, automatic segmentation of vertebra remains challenging due to several reasons, including anatomic complexity of spine, unclear boundaries of the vertebrae associated with spongy and soft bones. Based on 2D U-Net, we have proposed an Embedded Clustering Sliced U-Net (ECSU-Net). ECSU-Net comprises of three modules named segmentation, intervertebral disc extraction (IDE) and fusion. The segmentation module follows an instance embedding clustering approach, where our three sliced sub-nets use axis of CT images to generate a coarse 2D segmentation along with embedding space with the same size of the input slices. Our IDE module is designed to classify vertebra and find the inter-space between two slices of segmented spine. Our fusion module takes the coarse segmentation (2D) and outputs the refined 3D results of vertebra. A novel adaptive discriminative loss (ADL) function is introduced to train the embedding space for clustering. In the fusion strategy, three modules are integrated via a learnable weight control component, which adaptively sets their contribution. We have evaluated classical and deep learning methods on Spineweb dataset-2. ECSU-Net has provided comparable performance to previous neural network based algorithms achieving the best segmentation dice score of 95.60% and classification accuracy of 96.20%, while taking less time and computation resources.

Glaucoma is an ocular disease that causes permanent blindness if not cured at an early stage. Cup-to-disk ratio (CDR), obtained by dividing the height of optic cup (OC) with the height of optic disk (OD), is a widely adopted metric used for glaucoma screening. Therefore, accurately segmenting OD and OC is crucial for calculating a CDR. Most methods have employed deep learning methods for the segmentation of OD and OC. However, these methods are very time consuming. In this article, we present a new fuzzy broad learning system-based technique for OD and OC segmentation with glaucoma screening. We comprehensively integrated extracting a region of interest from RGB images, data augmentation, extracting red and green channel images, and inputting them to the two separate fuzzy broad learning system-based neural networks for segmenting the OD and OC, respectively, and then calculated CDR. Experiments show that our fuzzy broad learning system-based technique outperforms many state-of-the-art methods.

This article substantially advances upon state-of-the-art to enhance liver vessels segmentation accuracy by leveraging advantages of synthetic PET-CT (SPET-CT) images in addition to computed tomography angiography (CTA) volumes. Our setup makes a hybrid solution of modified generative adversarial network-convolutional autoencoder (GAN-cAED) combining synthetic ability of GAN to deliver SPET-CT images with generative ability of cAED network in terms of latent learning to more refined segmentation of major liver vessels. We improve time complexity through a novel concept of controlled segmentation by introducing a threshold metric to stop segmentation up to a desired level. The innovative concept of controlled vessel segmentation with a stopping criterion via variant threshold levels will help surgeons to avoid unintentional major blood vessels cutting, reducing the risk of excessive blood loss. Clinically, such solutions offer computer-aided liver surgeries and drug treatment evaluation in a CTA-only environment, shorten the requirement of radioactive and expensive fused PET-CT images.

Ultrasound is a widely used technology for diagnosing developmental dysplasia of the hip (DDH) because it does not use radiation. Due to its low cost and convenience, 2-D ultrasound is still the most common examination in DDH diagnosis. In clinical usage, the complexity of both ultrasound image standardization and measurement leads to a high error rate for sonographers. The automatic segmentation results of key structures in the hip joint can be used to develop a standard plane detection method that helps sonographers decrease the error rate. However, current automatic segmentation methods still face challenges in robustness and accuracy. Thus, we propose a neonatal hip bone segmentation network (NHBS-Net) for the first time for the segmentation of seven key structures. We design three improvements, an enhanced dual attention module, a two-class feature fusion module, and a coordinate convolution output head, to help segment different structures. Compared with current state-of-the-art networks, NHBS-Net gains outstanding performance accuracy and generalizability, as shown in the experiments. Additionally, image standardization is a common need in ultrasonography. The ability of segmentation-based standard plane detection is tested on a 50-image standard dataset. The experiments show that our method can help healthcare workers decrease their error rate from 6%-10% to 2%. In addition, the segmentation performance in another ultrasound dataset (fetal heart) demonstrates the ability of our network.

Although significant progress has been made in generating images from the text by using generative adversarial networks (GANs), it is still challenging to deal with long text, which contains complex semantic information like recipes. This paper focuses on generating images with high visual realism and semantic consistency from the complex text of recipes. To achieve this, we propose a GANs based method termed ChefGAN. The critical concept of ChefGAN is that a joint image-recipe embedding model is used before the generation task to provide high-quality representations of recipes, and it acts as an extra regularization during the generation to improve semantic consistency. Two modules are designed for this image text embedding module (ITEM) and a cascaded image generation module (CIGM). The generation process is carried out in 3 steps: (1) Two encoders in ITEM are trained simultaneously to generate similar representations for each image-recipe pair. (2) CIGM generates images according to the representations from ITEM's text encoder. (3) The generated image is fed into ITEM's image encoder to calculate the similarity with the given recipe. This process can provide additional regularization effect other than the impact of a discriminator. To facilitate convergence, we applied a two-stage training strategy, which generates an image with low resolution and then one with high resolution in the CIGM module. Compared with other representative state-of-the-art methods, ChefGAN demonstrates better performance both in visual realism and semantic consistency.

Detection of lung cancer at early stages is critical, in most of the cases radiologists read computed tomography (CT) images to prescribe follow-up treatment. The conventional method for detecting nodule presence in CT images is tedious. In this article, we propose an enhanced multidimensional region-based fully convolutional network (mRFCN) based automated decision support system for lung nodule detection and classification. The mRFCN is used as an image classifier backbone for feature extraction along with the novel multilayer fusion region proposal network (mLRPN) with position-sensitive score maps being explored. We applied a median intensity projection to leverage three-dimensional information from CT scans and introduced deconvolutional layer to adopt proposed mLRPN in our architecture to automatically select the potential region of interest. Our system has been trained and evaluated using LIDC dataset, and the experimental results showed promising detection performance in comparison to the state-of-the-art nodule detection/classification methods, achieving a sensitivity of 98.1% and classification accuracy of 97.91%.

Intracranial blood vessels segmentation from computed tomography angiography (CTA) volumes is a promising biomarker for diagnosis and therapeutic treatment in cerebrovascular diseases. These segmentation outputs are a fundamental requirement in the development of automated decision support systems for preoperative assessment or intraoperative guidance in neuropathology. The state-of-the-art in medical image segmentation methods are reliant on deep learning architectures based on convolutional neural networks. However, despite their popularity, there is a research gap in the current deep learning architectures optimized to address the technical challenges in blood vessel segmentation. These challenges include: (i) the extraction of concrete brain vessels close to the skull; and (ii) the precise marking of the vessel locations. We propose an Optimally Fused Fully end-to-end Network (OFF-eNET) for automatic segmentation of the volumetric 3D intracranial vascular structures. OFF-eNET comprises of three modules. In the first module, we exploit the up-skip connections to enhance information flow, and dilated convolution for detailed preservation of spatial feature map that are designed for thin blood vessels. In the second module, we employ residual mapping along with inception module for speedy network convergence and richer visual representation. For the third module, we make use of the transferred knowledge in the form of cascaded training strategy to gradually optimize the three segmentation stages (basic, complete, and enhanced) to segment thin vessels located close to the skull. All these modules are designed to be computationally efficient. Our OFF-eNET, evaluated using 70 CTA image volumes, resulted in 90.75% performance in the segmentation of intracranial blood vessels and outperformed the state-of-the-art counterparts.

The retinal vessel is one of the determining factors in an ophthalmic examination. Automatic extraction of retinal vessels from low-quality retinal images still remains a challenging problem. In this paper, we propose a robust and effective approach that qualitatively improves the detection of low-contrast and narrow vessels. Rather than using the pixel grid, we use a superpixel as the elementary unit of our vessel segmentation scheme. We regularize this scheme by combining the geometrical structure, texture, color, and space information in the superpixel graph. And the segmentation results are then refined by employing the efficient minimum spanning superpixel tree to detect and capture both global and local structure of the retinal images. Such an effective and structure-aware tree detector significantly improves the detection around the pathologic area. Experimental results have shown that the proposed technique achieves advantageous connectivity-area-length (CAL) scores of 80.92% and 69.06% on two public datasets, namely, DRIVE and STARE, thereby outperforming state-of-the-art segmentation methods. In addition, the tests on the challenging retinal image database have further demonstrated the effectiveness of our method. Our approach achieves satisfactory segmentation performance in comparison with state-of-the-art methods. Our technique provides an automated method for effectively extracting the vessel from fundus images.

Notice of Violation of IEEE Publication Principles"Clinical Report Guided Retinal Microaneurysm Detection With Multi-Sieving Deep Learning,"by Ling Dai, Ruogu Fang, Huating Li, Xuhong Hou, Bin Sheng, Qiang Wu, and Weiping Jiain the IEEE Transactions on Medical Imaging, vol. 37, no. 5, May 2018, pp. 1149-1161After careful and considered review of the content and authorship of this paper by a duly constituted expert committee, this paper has been found to be in violation of IEEE's Publication Principles.This paper contains significant portions of original text from the paper cited below. The original text was copied without attribution (including appropriate references to the original author(s) and/or paper title) and without permission."Mapping Visual Features to Semantic Profiles for Retrieval in Medical Imaging,"by Johannes Hofmanninger ; Georg Langsin the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015, pp. 457-465.Timely detection and treatment of microaneurysms is a critical step to prevent the development of vision-threatening eye diseases such as diabetic retinopathy. However, detecting microaneurysms in fundus images is a highly challenging task due to the low image contrast, misleading cues of other red lesions, and the large variation of imaging conditions. Existing methods tend to fail in face of the large intra-class variation and small inter-class variations for microaneurysm detection in fundus images. Recently, hybrid text/image mining computer-aided diagnosis systems have emerged to offer a promise of bridging the semantic gap between images and diagnostic information. In this paper, we focus on developing an interleaved deep mining technique to cope intelligently with the unbalanced microaneurysm detection problem. Specifically, we present a clinical report guided multi-sieving convolutional neural network, which leverages a small amount of supervised information in clinical reports to identify the potential microaneurysm regions via the image-to-text mapping in the feature space. These potential microaneurysm regions are then interleaved with fundus image information for multi-sieving deep mining in a highly unbalanced classification problem. Critically, the clinical reports are employed to bridge the semantic gap between low-level image features and high-level diagnostic information. We build an efficient microaneurysm detection framework based on the hybrid text/image interleaving and validate its performance on challenging clinical data sets acquired from diabetic retinopathy patients. Extensive evaluations are carried out in terms of fundus detection and classification. Experimental results show that our framework achieves 99.7% precision and 87.8% recall, comparing favorably with the state-of-the-art algorithms. Integration of expert domain knowledge and image information demonstrates the feasibility of reducing the difficulty of training classifiers under extremely unbalanced data distributions.