Effective representation of Regions of Interest (ROI) and independent alignment of these ROIs can significantly enhance the performance of deformable medical image registration (DMIR). However, current learning-based DMIR methods have limitations. Unsupervised techniques disregard ROI representation and proceed directly with aligning pairs of images, while weakly-supervised methods heavily depend on label constraints to facilitate registration. To address these issues, we introduce a weakly-supervised ROI-based registration approach named EASR-DCN. Our method represents medical images through effective ROIs and achieves independent alignment of these ROIs without requiring labels. Specifically, we first used a Gaussian mixture model for intensity analysis to represent images using multiple effective ROIs with distinct intensities. Furthermore, we propose a novel Divide-and-Conquer Network (DCN) that processes ROIs through separate channels to independently align their features. The resulting sub-deformation fields are seamlessly integrated to generate a comprehensive displacement vector field. Extensive experiments were performed on three MRI and one CT datasets to showcase the superior accuracy and deformation reduction efficacy of our EASR-DCN. Compared to VoxelMorph, our EASR-DCN achieved improvements of 10.31% in the Dice score for brain MRI, 13.01% for cardiac MRI, and 5.75% for hippocampus MRI, highlighting its promising potential for clinical applications.

Although recent years have witnessed significant advancements in medical image segmentation, the pervasive issue of domain shift among medical images from diverse centres hinders the effective deployment of pre-trained models. Many Test-time Adaptation (TTA) methods have been proposed to address this issue by fine-tuning pre-trained models with test data during inference. These methods, however, often suffer from less-satisfactory optimization due to suboptimal optimization direction (dictated by the gradient) and fixed step-size (predicated on the learning rate). In this paper, we propose the Gradient alignment-based Test-time adaptation (GraTa) method to improve both the gradient direction and learning rate in the optimization procedure. Unlike conventional TTA methods, which primarily optimize the pseudo gradient derived from a self-supervised objective, our method incorporates an auxiliary gradient with the pseudo one to facilitate gradient alignment. Such gradient alignment enables the model to excavate the similarities between different gradients and correct the gradient direction to approximate the empirical gradient related to the current segmentation task. Additionally, we design a dynamic learning rate based on the cosine similarity between the pseudo and auxiliary gradients, thereby empowering the adaptive fine-tuning of pre-trained models on diverse test data. Extensive experiments establish the effectiveness of the proposed gradient alignment and dynamic learning rate and substantiate the superiority of our GraTa method over other state-of-the-art TTA methods on a benchmark medical image segmentation task.

Icons map computational concepts through pictorial representations, assisting users in learning and using interfaces. However, constructing effective mapping structures and graphically expressing icon semantics remain challenges. This study explores how pictogram-semantic mapping structures impact icon understanding, employing a 3 (Pictogram-semantic mapping structure: Direct mapping, Metaphorical mapping, Metonymical mapping) × 2 (Semantic congruity: Semantic congruence, Semantic incongruence) within-subject design with behavioral experiments and Event-Related Potential (ERP) measurements. Results reveal that mapping structure significantly influences understanding: direct mapping enhances understanding efficiency, enabling quick and accurate interpretation; metaphorical mapping improves semantic processing fluency and emotional arousal, enriching the emotional experience; while metonymical mapping reduces efficiency without notable benefits in semantic coherence or emotional engagement. These findings offer practical insights for designing understandable icons and a theoretical foundation for strategically improving user experience through icon design.

Digital payment systems present usability challenges for older adults, and metaphorical icons may facilitate their learning and use of these systems. This study investigated how graphic and textual metaphors affect older users' understanding of action and knowledge icons in payment interfaces. Behavioral experiments combined with eye-tracking measurements were utilized. The findings suggest that employing appearance-resembling metaphor and metaphorical labels for action icons can significantly enhance icon readability, reduce cognitive load and uncertainty for older users, and improve their sense of self-efficacy when interacting with digital payment systems. Additionally, using appearance-resembling metaphor with textual labels for knowledge icons may improve recognition efficiency, alleviate cognitive load, and enhance older users’ self-efficacy in using these systems. These findings provide insights for designers to enhance older users’ experience with digital payment systems and contribute to improving the overall well-being of the aging population in an increasingly digital world.

Deposition of $\beta$-amyloid (A$\beta$), which is generally observed by A$\beta$-PET, is an important biomarker to evaluate subjects with early-onset dementia. However, acquisition of A$\beta$-PET usually suffers from high expense and radiation hazards, making A$\beta$-PET not commonly used as MRI. As A$\beta$-PET scans are only used to determine whether A$\beta$ deposition is positive or not, it is highly valuable to capture the underlying relationship between A$\beta$ deposition and other neuroimages (i.e., MRI) and detect amyloid status based on other neuroimages to reduce necessity of acquiring A$\beta$-PET. To this end, we propose an image-and-label conditioning latent diffusion model (IL-CLDM) to synthesize A$\beta$-PET scans from MRI scans by enhancing critical shared information to finally achieve MRI-based A$\beta$ classification. Specifically, two conditioning modules are introduced to enable IL-CLDM to implicitly learn joint image synthesis and diagnosis: 1) an image conditioning module, to extract meaningful features from source MRI scans to provide structural information, and 2) a label conditioning module, to guide the alignment of generated scans to the diagnosed label. Experiments on a clinical dataset of 510 subjects demonstrate that our proposed IL-CLDM achieves image quality superior to five widely used models, and our synthesized A$\beta$-PET scans (by IL-CLDM) can significantly help classification of A$\beta$ as positive or negative.

Distribution shift widely exists in medical images acquired from different medical centres and poses a significant obstacle to deploying the pretrained semantic segmentation model in real-world applications. Test-time adaptation has proven its effectiveness in tackling the cross-domain distribution shift during inference. However, most existing methods achieve adaptation by updating the pretrained models, rendering them susceptible to error accumulation and catastrophic forgetting when encountering a series of distribution shifts (i.e., under the continual test-time adaptation setup). To overcome these challenges caused by updating the models, in this paper, we freeze the pretrained model and propose the Visual Prompt-based Test-Time Adaptation (VPTTA) method to train a specific prompt for each test image to align the statistics in the batch normalization layers. Specifically, we present the low-frequency prompt, which is lightweight with only a few parameters and can be effectively trained in a single iteration. To enhance prompt initialization, we equip VPTTA with a memory bank to benefit the current prompt from previous ones. Additionally, we design a warm-up mechanism, which mixes source and target statistics to construct warm-up statistics, thereby facilitating the training process. Extensive experiments demonstrate the superiority of our VPTTA over other state-of-the-art methods on two medical image segmentation benchmark tasks. The code and weights of pretrained source models are available at https://github.com/Chen-Ziyang/VPTTA.

Deposition of β-amyloid is a crucial biomarker to evaluate subjects with early-onset dementia, often evaluated through Aβ-PET imaging. Aβ-PET is expensive and radiation-heavy; thus, it’s advisable to avoid it unless medically necessary. Therefore there is a compelling need to classify Aβ and detect amyloid status using other neuroimaging modalities, capitalizing on the underlying relationship between different modalities. Here, we propose an image and label conditioning latent diffusion model to synthesize Aβ-PET scans for improving Aβ classification based on MRI and FDG-PET scans. We introduce two conditioning modules: (1) an image conditioning module to extract a meaningful feature map from two source modalities to provide structural and metabolism information for guidance, and (2) a label conditioning module to provide the specific guidance direction on image generation. Experiments on the clinical dataset demonstrate that our proposed method’s synthetic Aβ-PET scans are reliable for classifying Aβ.

Cross-domain joint segmentation of optic disc and optic cup on fundus images is essential, yet challenging, for effective glaucoma screening. Although many unsupervised domain adaptation (UDA) methods have been proposed, these methods can hardly achieve complete domain alignment, leading to suboptimal performance. In this paper, we propose a triple-level alignment (TriLA) model to address this issue by aligning the source and target domains at the input level, feature level, and output level simultaneously. At the input level, a learnable Fourier domain adaptation (LFDA) module is developed to learn the cut-off frequency adaptively for frequency-domain translation. At the feature level, we disentangle the style and content features and align them in the corresponding feature spaces using consistency constraints. At the output level, we design a segmentation consistency constraint to emphasize the segmentation consistency across domains. The proposed model is trained on the RIGA+ dataset and widely evaluated on six different UDA scenarios. Our comprehensive results not only demonstrate that the proposed TriLA substantially outperforms other state-of-the-art UDA methods in joint segmentation of optic disc and optic cup, but also suggest the effectiveness of the triple-level alignment strategy.

Image registration of liver dynamic contrast-enhanced computed tomography (DCE-CT) is crucial for diagnosis and image-guided surgical planning of liver cancer. However, intensity variations due to the flow of contrast agents combined with complex spatial motion induced by respiration brings great challenge to existing intensity-based registration methods. To address these problems, we propose a novel structure-aware registration method by incorporating structural information of related organs with segmentation-guided deep registration network. Existing segmentation-guided registration methods only focus on volumetric registration inside the paired organ segmentations, ignoring the inherent attributes of their anatomical structures. In addition, such paired organ segmentations are not always available in DCE-CT images due to the flow of contrast agents. Different from existing segmentation-guided registration methods, our proposed method extracts structural information in hierarchical geometric perspectives of line and surface. Then, according to the extracted structural information, structure-aware constraints are constructed and imposed on the forward and backward deformation field simultaneously. In this way, all available organ segmentations, including unpaired ones, can be fully utilized to avoid the side effect of contrast agent and preserve the topology of organs during registration. Extensive experiments on an in-house liver DCE-CT dataset and a public LiTS dataset show that our proposed method can achieve higher registration accuracy and preserve anatomical structure more effectively than state-of-the-art methods.

Glaucoma is one of the leading causes of irreversible blindness. Segmentation of optic disc (OD) and optic cup (OC) on fundus images is a crucial step in glaucoma screening. Although many deep learning models have been constructed for this task, it remains challenging to train an OD/OC segmentation model that could be deployed successfully to different healthcare centers. The difficulties mainly comes from the domain shift issue, i.e., the fundus images collected at these centers usually vary greatly in the tone, contrast, and brightness. To address this issue, in this paper, we propose a novel unsupervised domain adaptation (UDA) method called Reconstruction-driven Dynamic Refinement Network (RDR-Net), where we employ a due-path segmentation backbone for simultaneous edge detection and region prediction and design three modules to alleviate the domain gap. The reconstruction alignment (RA) module uses a variational auto-encoder (VAE) to reconstruct the input image and thus boosts the image representation ability of the network in a self-supervised way. It also uses a style-consistency constraint to force the network to retain more domain-invariant information. The low-level feature refinement (LFR) module employs input-specific dynamic convolutions to suppress the domain-variant information in the obtained low-level features. The prediction-map alignment (PMA) module elaborates the entropy-driven adversarial learning to encourage the network to generate source-like boundaries and regions. We evaluated our RDR-Net against state-of-the-art solutions on four public fundus image datasets. Our results indicate that RDR-Net is superior to competing models in both segmentation performance and generalization ability.

Multi-modality medical data provide complementary information, and hence have been widely explored for computer-aided AD diagnosis. However, the research is hindered by the unavoidable missing-data problem, i.e., one data modality was not acquired on some subjects due to various reasons. Although the missing data can be imputed using generative models, the imputation process may introduce unrealistic information to the classification process, leading to poor performance. In this paper, we propose the Disentangle First, Then Distill (DFTD) framework for AD diagnosis using incomplete multi-modality medical images. First, we design a region-aware disentanglement module to disentangle each image into inter-modality relevant representation and intra-modality specific representation with emphasis on disease-related regions. To progressively integrate multi-modality knowledge, we then construct an imputation-induced distillation module, in which a lateral inter-modality transition unit is created to impute representation of the missing modality. The proposed DFTD framework has been evaluated against six existing methods on an ADNI dataset with 1248 subjects. The results show that our method has superior performance in both AD-CN classification and MCI-to-AD prediction tasks, substantially over-performing all competing methods.

Positron Emission Tomography (PET) is an important nuclear medical imaging technique, and has been widely used in clinical applications, e.g., tumor detection and brain disease diagnosis. As PET imaging could put patients at risk of radiation, the acquisition of high-quality PET images with standard-dose tracers should be cautious. However, if dose is reduced in PET acquisition, the imaging quality could become worse and thus may not meet clinical requirement. To safely reduce the tracer dose and also maintain high quality of PET imaging, we propose a novel and effective approach to estimate high-quality Standard-dose PET (SPET) images from Low-dose PET (LPET) images. Specifically, to fully utilize both the rare paired and the abundant unpaired LPET and SPET images, we propose a semi-supervised framework for network training. Meanwhile, based on this framework, we further design a Region-adaptive Normalization (RN) and a structural consistency constraint to track the task-specific challenges. RN performs region-specific normalization in different regions of each PET image to suppress negative impact of large intensity variation across different regions, while the structural consistency constraint maintains structural details during the generation of SPET images from LPET images. Experiments on real human chest-abdomen PET images demonstrate that our proposed approach achieves state-of-the-art performance quantitatively and qualitatively.

Incomplete data problem is commonly existing in classification tasks with multi-source data, particularly the disease diagnosis with multi-modality neuroimages, to track which, some methods have been proposed to utilize all available subjects by imputing missing neuroimages. However, these methods usually treat image synthesis and disease diagnosis as two standalone tasks, thus ignoring the specificity conveyed in different modalities, i.e., different modalities may highlight different disease-relevant regions in the brain. To this end, we propose a disease-image-specific deep learning (DSDL) framework for joint neuroimage synthesis and disease diagnosis using incomplete multi-modality neuroimages. Specifically, with each whole-brain scan as input, we first design a Disease-image-Specific Network (DSNet) with a spatial cosine module to implicitly model the disease-image specificity. We then develop a Feature-consistency Generative Adversarial Network (FGAN) to impute missing neuroimages, where feature maps (generated by DSNet) of a synthetic image and its respective real image are encouraged to be consistent while preserving the disease-image-specific information. Since our FGAN is correlated with DSNet, missing neuroimages can be synthesized in a diagnosis-oriented manner. Experimental results on three datasets suggest that our method can not only generate reasonable neuroimages, but also achieve state-of-the-art performance in both tasks of Alzheimer's disease identification and mild cognitive impairment conversion prediction.

Deep-learning methods (especially convolutional neural networks) using structural magnetic resonance imaging (sMRI) data have been successfully applied to computer-aided diagnosis (CAD) of Alzheimer's disease (AD) and its prodromal stage [i.e., mild cognitive impairment (MCI)]. As it is practically challenging to capture local and subtle disease-associated abnormalities directly from the whole-brain sMRI, most of those deep-learning approaches empirically preselect disease-associated sMRI brain regions for model construction. Considering that such isolated selection of potentially informative brain locations might be suboptimal, very few methods have been proposed to perform disease-associated discriminative region localization and disease diagnosis in a unified deep-learning framework. However, those methods based on task-oriented discriminative localization still suffer from two common limitations, that is: 1) identified brain locations are strictly consistent across all subjects, which ignores the unique anatomical characteristics of each brain and 2) only limited local regions/patches are used for model training, which does not fully utilize the global structural information provided by the whole-brain sMRI. In this article, we propose an attention-guided deep-learning framework to extract multilevel discriminative sMRI features for dementia diagnosis. Specifically, we first design a backbone fully convolutional network to automatically localize the discriminative brain regions in a weakly supervised manner. Using the identified disease-related regions as spatial attention guidance, we further develop a hybrid network to jointly learn and fuse multilevel sMRI features for CAD model construction. Our proposed method was evaluated on three public datasets (i.e., ADNI-1, ADNI-2, and AIBL), showing superior performance compared with several state-of-the-art methods in both tasks of AD diagnosis and MCI conversion prediction.

Multi-modal neuroimages, such as magnetic resonance imaging (MRI) and positron emission tomography (PET), can provide complementary structural and functional information of the brain, thus facilitating automated brain disease identification. Incomplete data problem is unavoidable in multi-modal neuroimage studies due to patient dropouts and/or poor data quality. Conventional methods usually discard data-missing subjects, thus significantly reducing the number of training samples. Even though several deep learning methods have been proposed, they usually rely on pre-defined regions-of-interest in neuroimages, requiring disease-specific expert knowledge. To this end, we propose a spatially-constrained Fisher representation framework for brain disease diagnosis with incomplete multi-modal neuroimages. We first impute missing PET images based on their corresponding MRI scans using a hybrid generative adversarial network. With the complete (after imputation) MRI and PET data, we then develop a spatially-constrained Fisher representation network to extract statistical descriptors of neuroimages for disease diagnosis, assuming that these descriptors follow a Gaussian mixture model with a strong spatial constraint (i.e., images from different subjects have similar anatomical structures). Experimental results on three databases suggest that our method can synthesize reasonable neuroimages and achieve promising results in brain disease identification, compared with several state-of-the-art methods.

Image classification is an essential and challenging task in computer vision. Despite its prevalence, the combination of the deep convolutional neural network (DCNN) and the Fisher vector (FV) encoding method has limited performance since the class-irrelevant background used in the traditional FV encoding may result in less discriminative image features. In this paper, we propose the foreground FV (fgFV) encoding algorithm and its fast approximation for image classification. We try to separate implicitly the class-relevant foreground from the class-irrelevant background during the encoding process via tuning the weights of the partial gradients corresponding to each Gaussian component under the supervision of image labels and, then, use only those local descriptors extracted from the class-relevant foreground to estimate FVs. We have evaluated our fgFV against the widely used FV and improved FV (iFV) under the combined DCNN-FV framework and also compared them to several state-of-the-art image classification approaches on ten benchmark image datasets for the recognition of fine-grained natural species and artificial manufactures, categorization of course objects, and classification of scenes. Our results indicate that the proposed fgFV encoding algorithm can construct more discriminative image presentations from local descriptors than FV and iFV, and the combined DCNN-fgFV algorithm can improve the performance of image classification.

A top-down image segmentation method is proposed in this paper, utilizing level set image representation and the piecewise-constant Mumford-Shah functional. The method achieves top-down hierarchical segmentation by taking advantage of the tree structure provided by level set image representation. The piecewise-constant Mumford-Shah functional is utilized in the proposed method to determine if each node in the tree segments the image. Experimental results show that this method is able to segment complicated real images.

A novel preferential image segmentation method is proposed that performs image segmentation and object recognition using mathematical morphologies. The method preferentially segments objects that have intensities and boundaries similar to those of objects in a database of prior images. A tree of shapes is utilized to represent the content distributions in images, and curve matching is applied to compare the boundaries. The algorithm is invariant to contrast change and similarity transformations of translation, rotation and scale. A performance evaluation of the proposed method using a large image dataset is provided. Experimental results show that the proposed approach is promising for applications such as object segmentation and video tracking with cluttered backgrounds.

Curve evolution implementations of the Mumford-Shah functional are of broad interest in image segmentation. These implementations, however, have initialization problems. A mathematical analysis of the initialization problem for the Chan-Vese implementation is provided in this paper. The initialization problem is a result of the non-convexity of the MumfordShah functional and the top-down hierarchy of the model's use of global region information in the image. Based on the analysis, efficient implementation methods are proposed for the Chan-Vese models. The proposed methods do not have to solve PDEs and thus work fast. The advantages of level set methods, such as automatic handling of topological changes, are preserved. These methods work well for images without strong noise. Initialization problems, however, still exist. A bottom-up image segmentation method is proposed that alleviates the initialization problem, based on region competition and the Mumford Shah functional. This algorithm extends the method in Jean Michel Morel, et al., (1995) and is able to automatically and efficiently segment objects in complicated images. Using a bottom-up hierarchy, the method avoids the initialization problem in the Chan-Vese model and works for images with multiple junctions and color images. It is then extended to textured images using Gabor filters and fractal methods. Experimental results show that the proposed method works well and is robust to the effects of noise

Curve evolution implementations of the Mumford-Shah functional are of broad interest in image segmentation. These implementations, however, have initialization problems. A mathematical analysis of the initialization problem for the bi-modal Chan-Vese model is provided in this paper. The initialization problem is a result of the non-convexity of the Mumford-Shah functional and the top-down hierarchy of the model's use of global region information in the image. An efficient image segmentation method is proposed that alleviates the initialization problem, based on region growing, region competition and the Mumford Shah functional. This algorithm is able to automatically and efficiently segment objects in complicated images. Using a bottom-up hierarchy, the method avoids the initialization problem in the Chan-Vese model and works for images with multiple junctions and color images. It can be extended to textured images. Experimental results show that the proposed method is robust to the effects of noise

Efficient implementation methods are proposed for Chan-Vese models. The proposed methods do not require solutions of PDEs and are therefore fast. The advantages of level set methods, such as automatic handling of topological changes, are preserved. These methods utilize region information to guide the evolution of initial curves. Gaussian smoothing is applied to regularize the evolving curves. These algorithms are able to automatically and efficiently segment objects in complicated images. Experimental results show that the proposed methods work efficiently for images without strong noise. However, they still have initialization problems, as do the Chan-Vese models

In this paper, a new geometric snake model is proposed, based upon techniques of curve evolution and the utilization of gradient information and region information simultaneously. This model successfully solves the boundary leakage problem. With the help of a hierarchical approach, it can handle complicated cases, such as triple junctions. Furthermore, it supports vector-valued images and can be easily extended to handle color and textured images. Experimental results demonstrate the model's power in image segmentation.

In this paper, a new model is proposed for image segmentation that integrates the curve evolution and anisotropic diffusion methods. The curve evolution method, utilizing both gradient and region information, segments an image into multiple regions. During the evolution of the curve, anisotropic diffusion is adaptively applied to the image to remove noise while preserving boundary information. Coupled partial differential equations (PDE’s) are used to implement the method. Experimental results show that the proposed model is successful for complex images with high noise.