Vascular segmentation is a critical task in clinical medical image processing and a prerequisite for accurately diagnosing vascular-related diseases. The development of automated segmentation methods is challenged by internal variability in vessel representations. Recently, topology guidance has shown potential for capturing semantically consistent representations. However, current topology-guided methods lack modeling of global-to-local dependencies. This limitation forces latent representations subject to a trade-off between learning global topology and local geometries within the vascular network. In this paper, we propose a Bayesian-based topology-guided (BayeTopo) learning approach to capture global-to-local dependencies. It introduces a prior that explicitly models local geometry as a probability conditioned on global topology within topology-sensitive regions of the vascular network. We further implement a topology-guided diffusion model to optimize the conditional probability. It gradually infers local geometry from the restored global topology with multi-scale noise, enabling rich global-to-local representations. Then, an inhomogeneous diffusion process is involved, where noise initially accumulates in topology-sensitive regions before achieving uniformity. It ensures an orderly degradation of information from global topology to local geometry, thereby enabling effective global-to-local supervision. Extensive experiments on six datasets, involving three types of vascular networks under four imaging modalities, demonstrate the superior performance and generalization capability of our method compared to previous topology-guided learning and diffusion-based models. A series of case studies further validates the effectiveness of our designs in enhancing semantic consistency within local vascular regions, thereby improving topological accuracy.

As a leading global cause of mortality, coronary ischemia requires accurate diagnostics for effective management. The combining coronary angiography with fractional flow reserve (FFR) offers structural and functional assessment of coronary stenosis to guide revascularization. However, traditional FFR measurements are invasive, requiring pressure wire placement. Image-based FFR estimation methods integrate vascular morphology with biomechanics but face challenges in modelling the complex fluid-structure interaction (FSI) of coronary flow and vessel walls. Therefore, we propose a physics-guided variational domain progressing method (PVDPM) for non-invasive FFR estimation through FSI system. PVDPM employs the principle of virtual work to model FSI system. This approach can improve the modelling of interdependent physical processes, enabling accurate FFR estimation based on coronary angiography-derived vascular morphology. The PVDPM demonstrates 91% accuracy in clinical datasets and offers solution for diagnosing coronary ischemia based on coronary angiography.

Motion estimation of left ventricle myocardium on the cardiac image sequence is crucial for assessing cardiac function. However, the intensity variation of cardiac image sequences brings the challenge of uncertain interference to myocardial motion estimation. Such imaging-related uncertain interference appears in different cardiac imaging modalities. We propose adaptive sequential Bayesian iterative learning to overcome the challenge. Specifically, our method applies the adaptive structural inference to state transition and observation to cope with a complex myocardial motion under uncertain setting. In state transition, adaptive structural inference establishes a hierarchical structure recurrence to obtain the complex latent representation of cardiac image sequences. In state observation, the adaptive structural inference forms a chain structure mapping to correlate the latent representation of the cardiac image sequence with that of the motion. Extensive experiments on US, CMR, and TMR datasets concerning 1270 patients (650 patients for CMR, 500 patients for US and 120 patients for TMR) have shown the effectiveness of our method, as well as the superiority to eight state-of-the-art motion estimation methods.

Federated learning has emerged as a transformative paradigm for distributed data collaboration, facilitating knowledge aggregation across multiple local clients through a global server while rigorously preserving data privacy. However, its performance is significantly hindered by the global head-tail imbalance, where tail classes with scarce data are often dominated by head classes. This challenge, known as federated long-tailed learning, arises from the intrinsic conflict between class knowledge acquisition and privacy preservation. Existing methodologies falter in resolving this conflict, as the abstraction of data knowledge in federated communication complicates the extraction of class-level knowledge, resulting in imbalanced global models and diminished performance. To simultaneously address this imbalance and uphold privacy, we introduce FedGRE, a gradient-refined federated learning approach that constructs global gradients and facilitates refined global gradient descent. FedGRE enhances gradients through two pivotal mechanisms: accumulation diffusion and accumulation refinement. The former amalgamates accumulated gradients with stochastic gradient perturbations to alleviate class imbalance, while the latter utilizes the accumulation as an anchor to calibrate global gradient updates, ensuring consistency and mitigating oscillations. Additionally, we implement a consistency integration technique to incorporate the refined accumulation into the global model, guaranteeing privacy-preserving and class-balanced global optimization. Extensive experiments on six datasets demonstrate that FedGRE significantly outperforms 14 state-of-the-art (SOTA) methods in federated long-tailed classification while maintaining robust privacy protection.

Computerized tomography (CT) is a clinically primary technique to differentiate benign-malignant pulmonary nodules for lung cancer diagnosis. Early classification of pulmonary nodules is essential to slow down the degenerative process and reduce mortality. The interactive paradigm assisted by neural networks is considered to be an effective means for early lung cancer screening in large populations. However, some inherent characteristics of pulmonary nodules in high-resolution CT images, e.g., diverse shapes and sparse distribution over the lung fields, have been inducing inaccurate results. On the other hand, most existing methods with neural networks are dissatisfactory from a lack of transparency. In order to overcome these obstacles, a united framework is proposed, including the classification and feature visualization stages, to learn distinctive features and provide visual results. Specifically, a bilateral scheme is employed to synchronously extract and aggregate global-local features in the classification stage, where the global branch is constructed to perceive deep-level features and the local branch is built to focus on the refined details. Furthermore, an encoder is built to generate some features, and a decoder is constructed to simulate decision behavior, followed by the information bottleneck viewpoint to optimize the objective. Extensive experiments are performed to evaluate our framework on two publicly available datasets, namely, 1) the Lung Image Database Consortium and Image Database Resource Initiative (LIDC-IDRI) and 2) the Lung and Colon Histopathological Image Dataset (LC25000). For instance, our framework achieves 92.98% accuracy and presents additional visualizations on the LIDC. The experiment results show that our framework can obtain outstanding performance and is effective to facilitate explainability. It also demonstrates that this united framework is a serviceable tool and further has the scalability to be introduced into clinical research.

Limited-angle tomographic reconstruction is one of the typical ill-posed inverse problems, leading to edge divergence with degraded image quality. Recently, deep learning has been introduced into image reconstruction and achieved great results. However, existing deep reconstruction methods have not fully explored data consistency, resulting in poor performance. In addition, deep reconstruction methods are still mathematically inexplicable and unstable. In this work, we propose an iterative residual optimization network (IRON) for limited-angle tomographic reconstruction. First, a new optimization objective function is established to overcome false negative and positive artifacts induced by limited-angle measurements. We integrate neural network priors as a regularizer to explore deep features within residual data. Furthermore, the block-coordinate descent is employed to achieve a novel iterative framework. Second, a convolution assisted transformer is carefully elaborated to capture both local and long-range pixel interactions simultaneously. Regarding the visual transformer, the multi-head attention is further redesigned to reduce computational costs and protect reconstructed image features. Third, based on the relative error convergence property of the convolution assisted transformer, a mathematical convergence analysis is also provided for our IRON. Both numerically simulated and clinically collected real cardiac datasets are employed to validate the effectiveness and advantages of the proposed IRON. The results show that IRON outperforms other state-of-the-art methods.

Aortic segmentation from computed tomography (CT) is crucial for facilitating aortic intervention, as it enables clinicians to visualize aortic anatomy for diagnosis and measurement. However, aortic segmentation faces the challenge of variable geometry in space, as the geometric diversity of different diseases and the geometric transformations that occur between raw and measured images. Existing constraint-based methods can potentially solve the challenge, but they are hindered by two key issues: inaccurate definition of properties and inappropriate topology of transformation in space. In this paper, we propose a deformable constraint transport network (DCTN). The DCTN adaptively extracts aortic features to define intra-image constrained properties and guides topological implementation in space to constrain inter-image geometric transformation between raw and curved planar reformation (CPR) images. The DCTN contains a deformable attention extractor, a geometry-aware decoder and an optimal transport guider. The extractor generates variable patches that preserve semantic integrity and long-range dependency in long-sequence images. The decoder enhances the perception of geometric texture and semantic features, particularly for low-intensity aortic coarctation and false lumen, which removes background interference. The guider explores the geometric discrepancies between raw and CPR images, constructs probability distributions of discrepancies, and matches them with inter-image transformation to guide geometric topology in space. Experimental studies on 267 aortic subjects and four public datasets show the superiority of our DCTN over 23 methods. The results demonstrate DCTN's advantages in aortic segmentation for different types of aortic disease, for different aortic segments, and in the measurement of clinical indexes.

Estimation of the fractional flow reserve (FFR) pullback curve from invasive coronary imaging is important for the intraoperative guidance of coronary intervention. Machine/deep learning has been proven effective in FFR pullback curve estimation. However, the existing methods suffer from inadequate incorporation of intrinsic geometry associations and physics knowledge. In this paper, we propose a constraint-aware learning framework to improve the estimation of the FFR pullback curve from invasive coronary imaging. It incorporates both geometrical and physical constraints to approximate the relationships between the geometric structure and FFR values along the coronary artery centerline. Our method also leverages the power of synthetic data in model training to reduce the collection costs of clinical data. Moreover, to bridge the domain gap between synthetic and real data distributions when testing on real-world imaging data, we also employ a diffusion-driven test-time data adaptation method that preserves the knowledge learned in synthetic data. Specifically, this method learns a diffusion model of the synthetic data distribution and then projects real data to the synthetic data distribution at test time. Extensive experimental studies on a synthetic dataset and a real-world dataset of 382 patients covering three imaging modalities have shown the better performance of our method for FFR estimation of stenotic coronary arteries, compared with other machine/deep learning-based FFR estimation models and computational fluid dynamics-based model. The results also provide high agreement and correlation between the FFR predictions of our method and the invasively measured FFR values. The plausibility of FFR predictions along the coronary artery centerline is also validated.

Multi-dimensional analysis in echocardiography has attracted attention due to its potential for clinical indices quantification and computer-aided diagnosis. It can utilize various information to provide the estimation of multiple cardiac indices. However, it still has the challenge of inter-task conflict. This is owing to regional confusion, global abnormalities, and time-accumulated errors. Task mapping methods have the potential to address inter-task conflict. However, they may overlook the inherent differences between tasks, especially for multi-level tasks (e.g., pixel-level, image-level, and sequence-level tasks). This may lead to inappropriate local and spurious task constraints. We propose cross-space consistency (CSC) to overcome the challenge. The CSC embeds multi-level tasks to the same-level to reduce inherent task differences. This allows multi-level task features to be consistent in a unified latent space. The latent space extracts task-common features and constrains the distance in these features. This constrains the task weight region that satisfies multiple task conditions. Extensive experiments compare the CSC with fifteen state-of-the-art echocardiographic analysis methods on five datasets (10,908 patients). The result shows that the CSC can provide left ventricular (LV) segmentation, (DSC = 0.932), keypoint detection (MAE = 3.06mm), and keyframe identification (accuracy = 0.943). These results demonstrate that our method can provide a multi-dimensional analysis of cardiac function and is robust in large-scale datasets.

The rapid development of the Internet of Things (IoT) widely supports the smart healthcare system. IoT-based smart health has significant importance for the diagnosis of cardiovascular disease (CVD) in clinical practice. Combined with advanced artificial intelligence techniques, IoT-based smart health provides valuable and accurate diagnosis information remotely for cardiovascular disease. The functional assessment of CVD is an essential task in clinical practice. It aims to determine the extent of myocardial ischemia through the measurement of the hemodynamic parameters of the coronary artery. However, the clinical adoption of the hemodynamic parameters is limited due to the potential risks and high health costs during measurements. Recent advances in artificial intelligence have enabled the computation of hemodynamic parameters based on the anatomical features of coronary arteries. However, the existing methods still lack explainability in the prediction. To address this issue, we present a physics-guided deep learning network for the functional assessment of CVD in an IoT-based manner. We specifically design an attentive network to determine the effective features by considering the importance of coronary artery anatomy features and artery segments. To obtain the functional assessment with explainability, we incorporate physical knowledge related to the blood flow into the loss function. It can ensure that functional assessment follows the physical laws. Extensive experiments are performed on a synthetic data set and a real-world clinical data set. The results show that our approach can achieve accurate and physically consistent assessment. Moreover, our method promotes deeper adoption of IoT and deep learning in the field of smart health.

Iodinated contrast medium (ICM) dose reduction is beneficial for decreasing potential health risk to renal-insufficiency patients in CT scanning. Due to the low-intensity vessel in ultra-low-dose-ICM CT angiography, it cannot provide clinical diagnosis of vascular diseases. Angiography reconstruction for ultra-low-dose-ICM CT can enhance vascular intensity for directly vascular diseases diagnosis. However, the angiography reconstruction is challenging since patient individual differences and vascular disease diversity. In this paper, we propose a Multiple Adversarial Learning based Angiography Reconstruction (i.e., MALAR) framework to enhance vascular intensity. Specifically, a bilateral learning mechanism is developed for mapping a relationship between source and target domains rather than the image-to-image mapping. Then, a dual correlation constraint is introduced to characterize both distribution uniformity from across-domain features and sample inconsistency within domain simultaneously. Finally, an adaptive fusion module by combining multi-scale information and long-range interactive dependency is explored to alleviate the interference of high-noise metal. Experiments are performed on CT sequences with different ICM doses. Quantitative results based on multiple metrics demonstrate the effectiveness of our MALAR on angiography reconstruction. Qualitative assessments by radiographers confirm the potential of our MALAR for the clinical diagnosis of vascular diseases.

Vessel contour detection (VCD) in intravascular images is important for the quantitative assessment of vessels. However, it is still a challenging task due to a high degree of morphology variability. Images from a single modality lack sufficient information on the vessel morphology due to the natural limitation of the imaging capability. Therefore, the single-modality VCD methods have difficulty extracting sufficient morphological information. Cross-modality methods have the potential to overcome morphology variability by extracting more information from different modalities. However, they still face the difficulty of the domain discrepancy, i.e., feature space discrepancy and label space inconsistency. In this paper, we aim to address the domain discrepancy for VCD. To overcome label space inconsistency, our method divides the label space into private label space and shared label space. It constructs subdomains for the private label space and the shared label space, and minimizes the task risk at the subdomain level. To overcome feature space discrepancy, it extracts domain-invariant features via domain adaptation between the subdomains. Finally, it uses the domain-invariant features as auxiliary information for each subdomain. Extensive experiments on 130 IVUS sequences (135663 images) and 124 OCT sequences (39857 images) show that our method is effective (e.g., the Dice index [Formula: see text] 0.949), and superior to the nineteen state-of-the-art VCD methods.

Main coronary segmentation from the X-ray angiography images is important for the computer-aided diagnosis and treatment of coronary disease. However, it confronts the challenge at three different image granularities (the semantic, surrounding, and local levels). The challenge includes the semantic confusion between the main and collateral vessels, low contrast between the foreground vessel and background surroundings, and local ambiguity near the vessel boundaries. The traditional hand-crafted feature-based methods may be insufficient because they may lack the semantic relationship information and may not distinguish the main and collateral vessels. The existing deep learning-based methods seem to have issues due to the deficiency in the long-distance semantic relationship capture, the foreground and background interference adaptability, and the boundary detail information preservation. To solve the main coronary segmentation challenge, we propose the progressive perception learning (PPL) framework to inspect these three different image granularities. Specifically, the PPL contains the context, interference, and boundary perception modules. The context perception is designed to focus on the main coronary vessel based on the semantic dependence capture among different coronary segments. The interference perception is designed to purify the feature maps based on the foreground vessel enhancement and background artifact suppression. The boundary perception is designed to highlight the boundary details based on boundary feature extraction through the intersection between the foreground and background predictions. Extensive experiments on 1085 subjects show that the PPL is effective (e.g., the overall Dice is greater than 95%), and superior to thirteen state-of-the-art coronary segmentation methods.

Automated and accurate segmentations of left atrium (LA) and atrial scars from late gadolinium-enhanced cardiac magnetic resonance (LGE CMR) images are in high demand for quantifying atrial scars. The previous quantification of atrial scars relies on a two-phase segmentation for LA and atrial scars due to their large volume difference (unbalanced atrial targets). In this paper, we propose an inter-cascade generative adversarial network, namely JAS-GAN, to segment the unbalanced atrial targets from LGE CMR images automatically and accurately in an end-to-end way. Firstly, JAS-GAN investigates an adaptive attention cascade to automatically correlate the segmentation tasks of the unbalanced atrial targets. The adaptive attention cascade mainly models the inclusion relationship of the two unbalanced atrial targets, where the estimated LA acts as the attention map to adaptively focus on the small atrial scars roughly. Then, an adversarial regularization is applied to the segmentation tasks of the unbalanced atrial targets for making a consistent optimization. It mainly forces the estimated joint distribution of LA and atrial scars to match the real ones. We evaluated the performance of our JAS-GAN on a 3D LGE CMR dataset with 192 scans. Compared with the state-of-the-art methods, our proposed approach yielded better segmentation performance (Average Dice Similarity Coefficient (DSC) values of 0.946 and 0.821 for LA and atrial scars, respectively), which indicated the effectiveness of our proposed approach for segmenting unbalanced atrial targets.

Semi-supervised learning provides great significance in left atrium (LA) segmentation model learning with insufficient labelled data. Generalising semi-supervised learning to cross-domain data is of high importance to further improve model robustness. However, the widely existing distribution difference and sample mismatch between different data domains hinder the generalisation of semi-supervised learning. In this study, we alleviate these problems by proposing anAdaptive Hierarchical Dual Consistency(AHDC) for the semi-supervised LA segmentation on cross-domain data. The AHDC mainly consists of a Bidirectional Adversarial Inference module (BAI) and a Hierarchical Dual Consistency learning module (HDC). The BAI overcomes the difference of distributions and the sample mismatch between two different domains. It mainly learns two mapping networks adversarially to obtain two matched domains through mutual adaptation. The HDC investigates a hierarchical dual learning paradigm for cross-domain semi-supervised segmentation based on the obtained matched domains. It mainly builds two dual-modelling networks for mining the complementary information in both intra-domain and inter-domain. For the intra-domain learning, a consistency constraint is applied to the dual-modelling targets to exploit the complementary modelling information. For the inter-domain learning, a consistency constraint is applied to the LAs modelled by two dual-modelling networks to exploit the complementary knowledge among different data domains. We demonstrated the performance of our proposed AHDC on four 3D late gadolinium enhancement cardiac MR (LGE-CMR) datasets from different centres and a 3D CT dataset. Compared to other state-of-the-art methods, our proposed AHDC achieved higher segmentation accuracy, which indicated its capability in the cross-domain semi-supervised LA segmentation.

Tissue segmentation is the mainstay of pathological examination, whereas the manual delineation is unduly burdensome. To assist this time-consuming and subjective manual step, researchers have devised methods to automatically segment structures in pathological images. Recently, automated machine and deep learning based methods dominate tissue segmentation research studies. However, most machine and deep learning based approaches are supervised and developed using a large number of training samples, in which the pixel-wise annotations are expensive and sometimes can be impossible to obtain. This paper introduces a novel unsupervised learning paradigm by integrating an end-to-end deep mixture model with a constrained indicator to acquire accurate semantic tissue segmentation. This constraint aims to centralise the components of deep mixture models during the calculation of the optimisation function. In so doing, the redundant or empty class issues, which are common in current unsupervised learning methods, can be greatly reduced. By validation on both public and in-house datasets, the proposed deep constrained Gaussian network achieves significantly (Wilcoxon signed-rank test) better performance (with the average Dice scores of 0.737 and 0.735, respectively) on tissue segmentation with improved stability and robustness, compared to other existing unsupervised segmentation approaches. Furthermore, the proposed method presents a similar performance (p-value >0.05) compared to the fully supervised U-Net.

Class imbalance is one of the most basic and important problems of web data. The key to overcoming the class imbalance problems is to increase the effective instances of the minority, that is, data augmentation. Generative Adversarial Networks (GANs), which have recently been successfully applied in the field of image generation, can be used for data augmentation because they can learn the data distribution given ample training data instances and generate more data. However, learning the distributions from the imbalanced data can make GANs easily get stuck in a local optimum. In this work, we propose a new training strategy called Annealing Genetic GAN (AGGAN), which incorporates simulated annealing genetic algorithm into the training process of GANs. And this can help GANs avoid the local optimum trapping problem, which easily occurs when the training set is imbalanced. Unlike existing GANs, which use a fixed adversarial learning objective alternately training a generator, we use multiple adversarial learning objectives to train a set of generators and use the Metropolis criterion in simulated annealing to decide whether the generator should update. More specifically, the Metropolis criterion accepts worse solutions with a certain probability, so it can make our AGGAN escape from the local optimum and find a better solution. Theory and mathematical analysis provide strong theoretical support for the proposed training strategy. And experiments on several datasets demonstrate that AGGAN achieves convincing ability to solve the class imbalanced problem and reduces the training problems inherent in existing GANs.

The development of the Internet of Things (IoT) plays an important role in smart health. The combination of IoT and ultrasound has great potential applications in telemedicine. The medical information in the echocardiogram stimulates the image analysis of ultrasound IoT devices. Medical image segmentation is a fundamental step in medical image analysis for providing help to the subsequent tasks. However, accurate left ventricular (LV) segmentation in echocardiography is a challenging task due to the cross-vendor and cross-center, multiview ultrasound image characteristics. We introduce an intermediate supervision deep neural networks method in this article, named as deep atrous pyramid intermediate supervision (DAPIS). The SAD block is used as a fixed feature extractor, pyramid pooling module is used to extract context semantic information, global attention upsampling is used for feature fusion module, and intermediate supervision is used to learn a better semantic representation. The performance of the model is evaluated using two echocardiographic data: 1) public data set (CAMUS) and 2) self-made data set (total 18906 images from 160 patients). Sufficient experiments show that DAPIS is prominent generalization and robustness, surpassing other state-of-the-art methods. The correlation graph and Bland–Altman analysis show good clinical correlation, proving the clinical potential of the model.

Deep convoloutional networks have been widely deployed in modern cyber-physical systems performing different visual classification tasks. As the fog and edge devices have different computing capacity and perform different subtasks, models trained for one device may not be deployable on another. Knowledge distillation technique can effectively compress well trained convolutional neural networks into light-weight models suitable to different devices. However, due to privacy issue and transmission cost, manually annotated data for training the deep learning models are usually gradually collected and archived in different sites. Simply training a model on powerful cloud servers and compressing them for particular edge devices failed to use the distributed data stored at different sites. This offline training approach is also inefficient to deal with new data collected from the edge devices. To overcome these obstacles, in this article, we propose the heterogeneous brain storming (HBS) method for object recognition tasks in real-world Internet of Things (IoT) scenarios. Our method enables flexible bidirectional federated learning of heterogeneous models trained on distributed datasets with a new “brain storming” mechanism and optimizable temperature parameters. In our comparison experiments, this HBS method outperformed multiple state-of-the-art single-model compression methods, as well as the newest multinetwork knowledge distillation methods with both homogeneous and heterogeneous classifiers. The ablation experiment results proved that the trainable temperature parameter into the conventional knowledge distillation loss can effectively ease the learning process of student networks in different methods. To the best of authors' knowledge, this is the first IoT-oriented method that allows asynchronous bidirectional heterogeneous knowledge distillation in deep networks.

The estimation of multitype cardiac indices from cardiac magnetic resonance imaging (MRI) and computed tomography (CT) images attracts great attention because of its clinical potential for comprehensive function assessment. However, the most exiting model can only work in one imaging modality (MRI or CT) without transferable capability. In this article, we propose the multitask learning method with the reverse inferring for estimating multitype cardiac indices in MRI and CT. Different from the existing forward inferring methods, our method builds a reverse mapping network that maps the multitype cardiac indices to cardiac images. The task dependencies are then learned and shared to multitask learning networks using an adversarial training approach. Finally, we transfer the parameters learned from MRI to CT. A series of experiments were conducted in which we first optimized the performance of our framework via ten-fold cross-validation of over 2900 cardiac MRI images. Then, the fine-tuned network was run on an independent data set with 2360 cardiac CT images. The results of all the experiments conducted on the proposed adversarial reverse mapping show excellent performance in estimating multitype cardiac indices.

Three-dimensional late gadolinium enhanced (LGE) cardiac MR (CMR) of left atrial scar in patients with atrial fibrillation (AF) has recently emerged as a promising technique to stratify patients, to guide ablation therapy and to predict treatment success. This requires a segmentation of the high intensity scar tissue and also a segmentation of the left atrium (LA) anatomy, the latter usually being derived from a separate bright-blood acquisition. Performing both segmentations automatically from a single 3D LGE CMR acquisition would eliminate the need for an additional acquisition and avoid subsequent registration issues. In this paper, we propose a joint segmentation method based on multiview two-task (MVTT) recursive attention model working directly on 3D LGE CMR images to segment the LA (and proximal pulmonary veins) and to delineate the scar on the same dataset. Using our MVTT recursive attention model, both the LA anatomy and scar can be segmented accurately (mean Dice score of 93% for the LA anatomy and 87% for the scar segmentations) and efficiently (∼0.27 s to simultaneously segment the LA anatomy and scars directly from the 3D LGE CMR dataset with 60–68 2D slices). Compared to conventional unsupervised learning and other state-of-the-art deep learning based methods, the proposed MVTT model achieved excellent results, leading to an automatic generation of a patient-specific anatomical model combined with scar segmentation for patients in AF.

Trustful Internet of Things (IoT) plays an important role in smart cities. The trust information in surveillance data motivates the analysis of images from numerous IoT devices. Saliency detection is a fundamental step in surveillance data analysis for providing help to the subsequent tasks, but unsuitable to IoT applications owing to the neglect of image similarity and difference from diverse IoT devices. To solve this problem, we enable the co-saliency detection in IoT, which detects the common and salient foreground regions in the group surveillance images. The main contributions include: 1) enable a multistage context perception scheme to efficiently extract the contextual information corresponding to different-size receptive fields in the single image; 2) construct a two-path information propagation to extract the interimage similarity and difference from the high-level image feature representations of the group images; and 3) propose the stage-wise refinement to allocate the label information to different parts of the network for helping the network to learn the enriched semantically common knowledge. The extensive experiments performed on three public data sets can demonstrate the effectiveness of our approach and its superiority to four state-of-the-art co-saliency detection methods.

In modern Internet of Things (IoT), visual analysis and predictions are often performed by deep learning models. Salient object detection (SOD) is a fundamental preprocessing for these applications. Executing SOD on the fog devices is a challenging task due to the diversity of data and fog devices. To adopt convolutional neural networks (CNN) on fog-cloud infrastructures for SOD-based applications, we introduce a semisupervised adversarial learning method in this article. The proposed model, named as SaliencyGAN, is empowered by a novel concatenated generative adversarial network (GAN) framework with partially shared parameters. The backbone CNN can be chosen flexibly based on the specific devices and applications. In the meanwhile, our method uses both the labeled and unlabeled data from different problem domains for training. Using multiple popular benchmark datasets, we compared state-of-the-art baseline methods to our SaliencyGAN obtained with 10-100% labeled training data. SaliencyGAN gained performance comparable to the supervised baselines when the percentage of labeled data reached 30%, and outperformed the weakly supervised and unsupervised baselines. Furthermore, our ablation study shows that SaliencyGAN were more robust to the common “mode missing” (or “mode collapse”) issue compared to the selected popular GAN models. The visualized ablation results have proved that SaliencyGAN learned a better estimation of data distributions. To the best of our knowledge, this is the first IoT-oriented semisupervised SOD method.

Intracoronary imaging is a crucial imaging technology in coronary disease diagnosis as it visualizes the internal tissue morphologies of coronary arteries. Vessel border detection in intracoronary images (VBDI) is desired because it can help the succeeding procedures of computer-aided disease diagnosis. However, existing VDBI methods suffer from the challenge of vessel-environment variability (i.e. high intra- and inter-subject diversity of vessels and their surrounding tissues appeared in images). This challenge leads to the ineffectiveness in the vessel region representation for hand-crafted features, in the receptive field extraction for deeply-represented features, as well as performance suppression derived from clinical data limitation. To solve this challenge, we propose a novel privileged modality distillation (PMD) framework for VBDI. PMD transforms the single-input-single-task (SIST) learning problem in the single-mode VBDI to a multiple-input-multiple-task (MIMT) problem by using the privileged image modality to help the learning model in the target modality. This learns the enriched high-level knowledge with similar semantics and generalizes PMD on diversity-increased low-level image features for improving the model adaptation to diverse vessel environments. Moreover, PMD refines MIMT to SIST by distilling the learned knowledge from multiple to one modality. This eliminates the reliance on privileged modality in the test phase, and thus enables the applicability to each of different intracoronary modalities. A structure-deformable neural network is proposed as an elaborately-designed implementation of PMD. It expands a conventional SIST network structure to the MIMT structure, and then recovers it to the final SIST structure. The PMD is validated on intravascular ultrasound imaging and optical coherence tomography imaging. One modality is the target, and the other one can be considered as the privileged modality owing to their semantic relatedness. The experiments show that our PMD is effective in VBDI (e.g. the Dice index is larger than 0.95), as well as superior to six state-of-the-art VBDI methods.

Quantification of coronary artery stenosis on X-ray angiography (XRA) images is of great importance during the intraoperative treatment of coronary artery disease. It serves to quantify the coronary artery stenosis by estimating the clinical morphological indices, which are essential in clinical decision making. However, stenosis quantification is still a challenging task due to the overlapping, diversity and small-size region of the stenosis in the XRA images. While efforts have been devoted to stenosis quantification through low-level features, these methods have difficulty in learning the real mapping from these features to the stenosis indices. These methods are still cumbersome and unreliable for the intraoperative procedures due to their two-phase quantification, which depends on the results of segmentation or reconstruction of the coronary artery. In this work, we are proposing a hierarchical attentive multi-view learning model (HEAL) to achieve a direct quantification of coronary artery stenosis, without the intermediate segmentation or reconstruction. We have designed a multi-view learning model to learn more complementary information of the stenosis from different views. For this purpose, an intra-view hierarchical attentive block is proposed to learn the discriminative information of stenosis. Additionally, a stenosis representation learning module is developed to extract the multi-scale features from the keyframe perspective for considering the clinical workflow. Finally, the morphological indices are directly estimated based on the multi-view feature embedding. Extensive experiment studies on clinical multi-manufacturer dataset consisting of 228 subjects show the superiority of our HEAL against nine comparing methods, including direct quantification methods and multi-view learning methods. The experimental results demonstrate the better clinical agreement between the ground truth and the prediction, which endows our proposed method with a great potential for the efficient intraoperative treatment of coronary artery disease.

Due to the variation in factors surrounding humans, the physiological impact of stress is reported to be different for each individual. Thus, an efficient stress monitoring system needs to assess both the physiological and psychological impact of stress on individual basis and translate these assessments into an accurate quantitative metric that is of value to the individual. Therefore, this study proposed a logistic regression based model that integrates data from psychological Stress Response Inventory, biochemical (salivary cortisol), and physiological (HRV measures) domains via a principle of triangulation for achieving high reliability and consistency during stress assessment. With the proposed model, a mental stress index (MSI) based on the correlation between salivary cortisol and HRV time-/frequency-domain features were established. A total of 30 college students were recruited to verify the feasibility of proposed method by identifying targeted stressful event. The obtained results reveal that MSI values were sensitive to acute stress, and could predict the association level of normal individual to a stress group with approximately 97% accuracy. Findings from this study could provide potential insight on self-tracking and training of individual's stress with adoption of wearable sensor system in a dynamic setting.

Segmentation of carotid intima-media (IM) borders from ultrasound sequences is challenging because of unknown image noise and varying IM border morphologies and/or dynamics. In this paper, we have developed a state-space framework to sequentially segment the carotid IM borders in each image throughout the cardiac cycle. In this framework, an ${\mathrm{H}}_{\mathrm{\infty }}$ filter is used to solve the state-space equations, and a grayscale-derivative constraint snake is used to provide accurate measurements for the ${\mathrm{H}}_{\mathrm{\infty }}$ filter. We have evaluated the performance of our approach by comparing our segmentation results to the manually traced contours of ultrasound image sequences of three synthetic models and 156 real subjects from four medical centers. The results show that our method has a small segmentation error (lumen intima, LI: 53 $\pm\, 67\;{\mathrm{\mu }}$m; media-adventitia, MA: 57 $\pm\, 63\;{\mathrm{\mu }}$m) for synthetic and real sequences of different image characteristics, and also agrees well with the manual segmentation (LI: bias = 1.44 ${\mathrm{\mu }}$m; MA: bias = $-$3.38 ${\mathrm{\mu }}$m). Our approach can robustly segment the carotid ultrasound sequences with various IM border morphologies, dynamics, and unknown image noise. These results indicate the potential of our framework to segment IM borders for clinical diagnosis.

The motion of the common carotid artery (CCA) wall has been established to be useful in early diagnosis of atherosclerotic disease. However, tracking the CCA wall motion from ultrasound images remains a challenging task. In this paper, a nonlinear state-space approach has been developed to track CCA wall motion from ultrasound sequences. In this approach, a nonlinear state-space equation with a time-variant control signal was constructed from a mathematical model of the dynamics of the CCA wall. Then, the unscented Kalman filter (UKF) was adopted to solve the nonlinear state transfer function in order to evolve the state of the target tissue, which involves estimation of the motion trajectory of the CCA wall from noisy ultrasound images. The performance of this approach has been validated on 30 simulated ultrasound sequences and a real ultrasound dataset of 103 subjects by comparing the motion tracking results obtained in this study to those of three state-of-the-art methods and of the manual tracing method performed by two experienced ultrasound physicians. The experimental results demonstrated that the proposed approach is highly correlated with (intra-class correlation coefficient ≥ 0.9948 for the longitudinal motion and ≥ 0.9966 for the radial motion) and well agrees (the 95% confidence interval width is 0.8871 mm for the longitudinal motion and 0.4159 mm for the radial motion) with the manual tracing method on real data and also exhibits high accuracy on simulated data (0.1161 ~ 0.1260 mm). These results appear to demonstrate the effectiveness of the proposed approach for motion tracking of the CCA wall.

It is very challenge to investigate the motion of the carotid artery wall in ultrasound images, because of the high nonlinear dynamics of this motion. In our study, the nonlinear dynamics of carotid artery wall motion is first approximated by our nonlinear state-space approach driven by a mathematical model of the mechanical deformation of carotid artery wall. Then, the two-dimensional motion of carotid artery wall is computed by solving the nonlinear state-space approach using the unscented Kalman filter. We have then evaluated the performance of our approach by comparing it with the manual tracing method (the correlation coefficient equals 0.9897 for the radial motion and 0.9703 for the longitudinal motion) and three other state-of-the-art methods for 73 subjects. The results indicate the reliable applicability of our approach in tracking the motion of the carotid artery wall and its potential usefulness in routine clinical diagnosis.

The same as in traditional surgery, surgeons in telerobotic surgery need extensive training to achieve experience and highly accurate instrument manipulation. Traditional training methods like practice in operating room have major drawbacks such as high risk and limited opportunity for which virtual reality (VR) and computer technologies can offer solutions. To accelerate the data transmission speed in our master-slave robotic simulator, GPUDirect was applied to ensure the synchronization and display rate of three computers. By using GPUDirect with InfiniBand card we realized up to 247% performance improvement in data transmission speed on NVIDIA Tesla™ products on different computers compared to that without GPUDirect, which shows that GPUDirect enables better communication between remote GPUs over InfiniBand.

Model-based reconstruction algorithms have shown potentials over conventional strain-based methods in static elas-tographic image by using ”accurate” finite element(FE) or bio-mechanical models. Strictly speaking, however, the measurement noise are always exists and thus do not meet basic assumptions of these algorithms. In addition, the difficulty in determining the proper system response model also greatly affects the quality of the reconstructed images. In this paper, we explore the usage of state space principles for the estimation of materials properties in elastographic imaging. The model-data discrepancy is modeled as uncertainties, i.e. Gaussian white noise, and the measurement noise is treated as another independent Gaussian white noise in the stochastic state space space, and an optimal estimation is computed of full displacement field and Young's modulus simultaneously using an extended Kalman filter (EKF). The performance of the proposed framework is evaluated using phantom data and real data with favorable results.

The recent advances in meaningful constraining models have resulted in increasingly useful quantitative information recovered from cardiac images. Nevertheless, as most frameworks utilize either functional or structural images, the analyses cannot benefit from the complementary information provided by the other image sources. To better characterize subject-specific cardiac physiology and pathology, data fusion of multiple image sources is essential. Traditional image fusion strategies are performed by fusing information of commensurate images through various mathematical operators. Nevertheless, when image data are dissimilar in physical nature and spatiotemporal quantity, such approaches may not provide meaningful connections between different data. In fact, as different image sources provide partial measurements of the same cardiac system dynamics, it is more natural and suitable to utilize cardiac physiological models for the fusions. Therefore, we propose to use the cardiac physiome model as the central link to fuse functional and structural images for more subject-specific cardiac deformation recovery through state-space filtering. Experiments were performed on synthetic and real data for the characteristics and potential clinical applicability of our framework, and the results show an increase of the overall subject specificity of the recovered deformations.

To systematically couple images and physiological models according to their respective merits, state-space filtering frameworks have been proposed for cardiac deformation recovery with promising results. Nevertheless, as thousands of forward simulations are required in every filtering step, the computational complexity is too high to be practical. To reduce the computational complexity without a significant loss of accuracy, we have adopted the mode superposition approach which transforms the cardiac system dynamics to a mathematically equivalent space of much lower dimensions. With the corresponding filtering procedures and components proposed, nonlinear cardiac deformation recovery can be performed in the transformed space with largely reduced computational complexity. Experiments were performed on synthetic data to evaluate the computational complexity and accuracy, and on human data for the clinical relevance.

Volumetric details of cardiac electrophysiology, such as transmembrane potential dynamics and tissue excitability of the myocardium, are of fundamental importance for understanding normal and pathological cardiac mechanisms, and for aiding the diagnosis and treatment of cardiac arrhythmia. Noninvasive observations, however, are made on body surface as an integration-projection of the volumetric phenomena inside patient's heart. We present a physiological-model-constrained statistical framework where prior knowledge of general myocardial electrical activity is used to guide the reconstruction of patient-specific volumetric cardiac electrophysiological details from body surface potential data. Sequential data assimilation with proper computational reduction is developed to estimate transmembrane potential and myocardial excitability inside the heart, which are then utilized to depict arrhythmogenic substrates. Effectiveness and validity of the framework is demonstrated through its application to evaluate the location and extent of myocardial infract using real patient data.

To noninvasively reconstruct transmembrane potential (TMP) dynamics throughout the 3D myocardium using body surface potential recordings, it is necessary to combine prior physiological models and patient's data with regard to their respective uncertainties. To fulfill model-data melding for this large-scale and high-dimensional system, data assimilation with proper computational reduction is needed for computational feasibility and efficiency. In this paper, we develop a reduced-rank square root TMP estimation algorithm, using dominant components of estimation uncertainties to guide a more efficient model-data coupling in the square root structure. The SVD-based reduced-rank error covariance is used to represent and track the dominant estimation errors, and unified into an integrated square root filtering framework. Phantom experiments demonstrate the ability of this framework to bring substantial computational reduction at slight expense of degraded estimation accuracy. It therefore improves the efficiency and applicability of the volumetric myocardial TMP imaging in practice.

To recover physiologically meaningful cardiac deformation from medical images, realistic physiological models are essential to constrain the recovery process, and a statistical filtering framework is required to couple the models and images according to their respective uncertainties. As realistic cardiac models are usually nonlinear, existing cardiac deformation recovery frameworks either ignore the statistical filtering part, or linearize the model and apply linear filtering techniques such as the extended Kalman filtering. This reduces the physiological plausibility and statistical optimality of the recovery results. In this paper, we propose a nonlinear cardiac deformation recovery framework with unscented Kalman filtering which preserves the intact system nonlinearity. Experiments were done on both synthetic data and magnetic resonance images to show the benefits and clinical relevance of our framework.

Body surface potential (BSP) has been used as the single data source in inverse electrocardiography (IECG). The lack of volumetric spatial resolutions in BSP, however, hinders the noninvasive mapping of volumetric cardiac transmembrane potentials (TMPs). Tomographic image sequence, which contains temporally sparse but spatially dense cardiac kinematic measures, becomes ideal dynamic complements to BSPs through cardiac electromechanical (EM) coupling. In this paper, we present a model-constrained Bayesian framework to integrate tomographic image sequence and BSP maps (BSPM) for volumetric cardiac TMP mapping. A priori physiological knowledge is incorporated via stochastic modeling of the cardiac electrophysiological system with unknown systematic errors. With this system as a platform for data integration, adaptive data assimilation is used to estimate patient specific TMPs from BSPMs under the guidance of tomographic images. In this way, complementary images are integrated in accord with their respective merits and limitations. Phantom and real data experiments exhibit notable improvements and practicability of the presented framework.

There is an increasing interest in the interactions of factors more directly related to the neutral activity in hemodynamic response (HR) with respect to different experimental conditions. In this work, we present a state-space approach, based on the Balloon model for blood oxygenation level dependent (BOLD) responses, which allows the estimation of the hidden state variables and parameters of the hemodynamic response at the same time. It offers an alternative strategy for understanding the interactions of indirectly observed factors and exploring the changes of biophysical model parameters in variant experimental conditions.