Epicardial adipose tissue (EAT) is increasingly recognized as an important contributor to cardiovascular disease (CVD), but its spatial distribution across the heart remains insufficiently characterized due to limitations in existing segmentation and analysis methods. In this study, we developed a deep learning-based framework that integrates automated cardiac MRI segmentation with spatial statistical modeling to enable quantitative, chamber-resolved characterization of EAT distribution. An asymmetric multi-modal CNN-Transformer network was developed to segment the four cardiac chambers and EAT from cardiac MRI. Based on the resulting whole-heart segmentations, EAT was automatically partitioned into four chamber-specific subregions, voxel-wise thickness maps were reconstructed, and spatial statistics were applied to identify localized clustering patterns of EAT. Chamber-resolved and region-specific quantitative features were extracted to characterize the spatial distribution of EAT across the heart. The proposed approach was evaluated on a cohort including individuals with type 2 diabetes (T2D) and matched controls, revealing distinct T2D-associated remodeling, including increased chamber-specific burden, localized thickening, and spatial hotspot clustering in metabolically vulnerable regions. This work introduces a scalable and automated method for regional EAT phenotyping and provides spatially resolved imaging biomarkers that may support CVD research and risk stratification.

Low-Rank Adaptation (LoRA) has proven effective in reducing computational costs while maintaining performance comparable to fully fine-tuned foundation models across various tasks. However, its fixed low-rank structure restricts its adaptability in scenarios with substantial domain gaps, where higher ranks are often required to capture domain-specific complexities. Current adaptive LoRA methods attempt to overcome this limitation by dynamically expanding or selectively allocating ranks, but these approaches frequently depend on computationally intensive techniques such as iterative pruning, rank searches, or additional regularization. To address these challenges, we introduce Stable Rank-Guided Low-Rank Adaptation (SR-LoRA), a novel framework that utilizes the stable rank of pre-trained weight matrices as a natural prior for layer-wise rank allocation. By leveraging the stable rank, which reflects the intrinsic dimensionality of the weights, SR-LoRA enables a principled and efficient redistribution of ranks across layers, enhancing adaptability without incurring additional search costs. Empirical evaluations on few-shot tasks with significant domain gaps show that SR-LoRA consistently outperforms recent adaptive LoRA variants, achieving a superior trade-off between performance and efficiency. Our code is available at https://github.com/EndoluminalSurgicalVision-IMR/SR-LoRA.

Robot assisted endoluminal intervention is an emerging technique for both benign and malignant luminal lesions. With vision-based navigation, when combined with pre-operative imaging data as priors, it is possible to recover position and pose of the endoscope without the need of additional sensors. In practice, however, aligning pre-operative and intra-operative domains is complicated by significant texture differences. Although methods such as style transfer can be used to address this issue, they require large datasets from both source and target domains with prolonged training times. This paper proposes an efficient domain transfer method based on stylized Gaussian splatting, only requiring a few of real images (10 images) with very fast training time. Specifically, the transfer process includes two phases. In the first phase, the 3D models reconstructed from CT scans are represented as differential Gaussian point clouds. In the second phase, only color appearance related parameters are optimized to transfer the style and preserve the visual content. A novel structure consistency loss is applied to latent features and depth levels to enhance the stability of the transferred images. Detailed validation was performed to demonstrate the performance advantages of the proposed method compared to that of the current state-of-the-art, highlighting the potential for intra-operative surgical navigation.

Robot assisted endoluminal intervention is an emerging tool for treating luminal lesions. Vision-based endoluminal navigation, particularly through video-CT registration, is a tangible way of obtaining absolute camera position information. By using pre-operative CT data, accurate endoscope localization can be achieved, without the need of additional tracking hardware intraoperatively. However, aligning preoperative CT with intraoperative domain remains a challenge. Although approaches such as style transfer have been explored, patient-specific textures and intra-operative artifacts can significantly complicate the task. To overcome these challenges, we propose R2Nav, a robust, real-time test time adaptation method for endoluminal navigation. R2Nav constructs a confidence buffer during the testing phase, refining the model only for frames with high uncertainty. We introduce a registration-augmented model refinement strategy, which enhances both accuracy and efficiency of the system by selecting relevant training samples from the virtual gallery. Additionally, we propose a novel warm-up strategy for the registration encoder during the initial testing phase, enabling the extraction of more robust features when the model is suboptimal. Extensive validation demonstrates that R2Nav outperforms the current state-of-the-art methods, offering significant advantages for real-time, intra-operative endoluminal navigation. Code is at: https://github.com/EndoluminalSurgicalVision-IMR/R2Nav.

Automatic and precise multi-class vertebrae segmentation from CT images is crucial for various clinical applications. However, due to similar appearances between adjacent vertebrae and the existence of various pathologies, existing single-stage and multi-stage methods suffer from imprecise vertebrae segmentation. Essentially, these methods fail to explicitly impose both contour precision and intra-vertebrae voxel consistency constraints synchronously, resulting in the intra-vertebrae segmentation inconsistency, which refers to multiple label predictions inside a singular vertebra. In this work, we intend to label complete binary masks with sequential indices to address that challenge. Specifically, a contour generation network is proposed based on Structural Low-Rank Descriptors for shape consistency, termed SLoRD. For a structural representation of vertebral contours, we adopt the spherical coordinate system and devise the spherical centroid to calculate contour descriptors. Due to vertebrae's similar appearances, basic contour descriptors can be acquired offline to restore original contours. Therefore, SLoRD leverages these contour priors and explicit shape constraints to facilitate regressed contour points close to vertebral surfaces. Quantitative and qualitative evaluations on VerSe 2019 and 2020 demonstrate the superior performance of our framework over other single-stage and multi-stage state-of-the-art (SOTA) methods. Further, SLoRD is a plug-and-play framework to refine the segmentation inconsistency existing in coarse predictions from other approaches.

Medical image segmentation is a fundamental task for medical image analysis and surgical planning. In recent years, UNet-based networks have prevailed in the field of medical image segmentation. However, convolutional neural networks (CNNs) suffer from limited receptive fields, which fail to model the long-range dependency of organs or tumors. Besides, these models are heavily dependent on the training of the final segmentation head. And existing methods can not well address aforementioned limitations simultaneously. Hence, in our work, we proposed a novel shape prior module (SPM), which can explicitly introduce shape priors to promote the segmentation performance of UNet-based models. The explicit shape priors consist of global and local shape priors. The former with coarse shape representations provides networks with capabilities to model global contexts. The latter with finer shape information serves as additional guidance to relieve the heavy dependence on the learnable prototype in the segmentation head. To evaluate the effectiveness of SPM, we conduct experiments on three challenging public datasets. And our proposed model achieves state-of-the-art performance. Furthermore, SPM can serve as a plug-and-play structure into classic CNNs and Transformer-based backbones, facilitating the segmentation task on different datasets. Source codes are available at https://github.com/AlexYouXin/Explicit-Shape-Priors.

Test-time adaptation (TTA) has emerged as a promising paradigm to handle the domain shifts at test time for medical images from different institutions without using extra training data. However, existing TTA solutions for segmentation tasks suffer from 1) dependency on modifying the source training stage and access to source priors or 2) lack of emphasis on shape-related semantic knowledge that is crucial for segmentation tasks. Recent research on visual prompt learning achieves source-relaxed adaptation by extended parameter space but still neglects the full utilization of semantic features, thus motivating our work on knowledge-enriched deep prompt learning. Beyond the general concern of image style shifts, we reveal that shape variability is another crucial factor causing the performance drop. To address this issue, we propose a TTA framework called PASS (Prompting to Adapt Styles and Semantic shapes), which jointly learns two types of prompts: the input-space prompt to reformulate the style of the test image to fit into the pretrained model and the semantic-aware prompts to bridge high-level shape discrepancy across domains. Instead of naively imposing a fixed prompt, we introduce an input decorator to generate the self-regulating visual prompt conditioned on the input data. To retrieve the knowledge representations and customize target-specific shape prompts for each test sample, we propose a cross-attention prompt modulator, which performs interaction between target representations and an enriched shape prompt bank. Extensive experiments demonstrate the superior performance of PASS over state-of-the-art methods on multiple medical image segmentation datasets. The code is available at https://github.com/EndoluminalSurgicalVision-IMR/PASS.

• This paper investigates the capacity of AI models for airway modelling on national datasets with paired clinical metadata. • We evaluated AI models against unharmonised, noisy, and out-of-distribution data, as well as the prognostication for FLD. • We found a new biomarker for mortality prediction, outperforming existing clinical measurements (FVC% and fibrosis scores). • In-depth analysis of AI models on airway modelling and prognosis, highlighting challenges and future research directions. Airway-related quantitative imaging biomarkers are crucial for examination, diagnosis, and prognosis in pulmonary diseases. However, the manual delineation of airway structures remains prohibitively time-consuming. While significant efforts have been made towards enhancing automatic airway modelling, current public-available datasets predominantly concentrate on lung diseases with moderate morphological variations. The intricate honeycombing patterns present in the lung tissues of fibrotic lung disease patients exacerbate the challenges, often leading to various prediction errors. To address this issue, the 'Airway-Informed Quantitative CT Imaging Biomarker for Fibrotic Lung Disease 2023′ (AIIB23) competition was organized in conjunction with the official 2023 International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI). The airway structures were meticulously annotated by three experienced radiologists. Competitors were encouraged to develop automatic airway segmentation models with high robustness and generalization abilities, followed by exploring the most correlated QIB of mortality prediction. A training set of 120 high-resolution computerised tomography (HRCT) scans were publicly released with expert annotations and mortality status. The online validation set incorporated 52 HRCT scans from patients with fibrotic lung disease and the offline test set included 140 cases from fibrosis and COVID-19 patients. The results have shown that the capacity of extracting airway trees from patients with fibrotic lung disease could be enhanced by introducing voxel-wise weighted general union loss and continuity loss. In addition to the competitive image biomarkers for mortality prediction, a strong airway-derived biomarker (Hazard ratio>1.5, p < 0.0001) was revealed for survival prognostication compared with existing clinical measurements, clinician assessment and AI-based biomarkers.

Olfactory-enhanced virtual reality (OVR) creates a complex and rich emotional experience, thus promoting a new generation of human-computer interaction experiences in real-world scenarios. However, with the rise of virtual reality (VR) as a mood induction procedure (MIP), few studies have incorporated olfactory stimuli into emotion induction in three-dimensional (3D) environments. Considering the differences in electroencephalography (EEG) dynamics between sensory stimuli, all previous two-dimensional (2D) and 3D emotional studies have been less effective in reality because they only use visual and audio senses. To overcome these limitations, we developed a novel EEG signal dataset based on OVR. We systematically analyzed the influence of olfactory stimuli on emotion induction in a VR environment from a neurophysiological perspective. Specifically, synchronous EEG signals were collected from 65 participants as they watched positive and negative videos in traditional VR and OVR. Their power spectral densities (PSDs) were then calculated to compare the differences in brain activation between their VR and OVR modes during the induction of positive and negative emotions, while their brain states were classified after feature selection. The results showed that olfactory stimuli enhanced EEG responses for positive emotions, but the opposite was true for negative emotions. Additionally, the recognition rate of brain emotional states was more than 90% under both positive and negative emotions, while the high-frequencyβandγbands could effectively distinguish VR and OVR modes. This study introduced the olfaction into the field of human-computer interaction, which could promote research on emotion induction and recognition in real-world environments.

Detailed pulmonary airway segmentation is a clinically important task for endobronchial intervention and treatment of peripheral pulmonary lesions. Convolutional Neural Networks (CNNs) are promising for automated analysis of medical imaging, which however performs poorly on airway segmentation. Specifically, breakage of small bronchi distals cannot be effectively eliminated in the prediction results of CNNs, which is detrimental to use as a reference for bronchoscopic-assisted surgery. In this paper, we proposed a connectivity-aware segmentation framework to improve the performance of airway segmentation. A Connectivity-Aware Surrogate (CAS) module is first proposed to balance the training progress within-class distribution. Furthermore, a Local-Sensitive Distance (LSD) module is designed to identify the breakage and minimize the variation of the distance map between the prediction and ground-truth. The proposed method is validated with the publically available reference airway segmentation datasets. The detected rate of branch and length on public EXACT'09 and BAS datasets are 82.1%/79.6% and 96.5%/91.5% respectively, demonstrating the effectiveness of the method in terms of improving the connectedness of the segmentation performance. The source code is available at: https://github.com/Puzzled-Hui/Connectivity-Aware-Airway-Segmentation.

Intrathoracic airway segmentation in computed tomography is a prerequisite for various respiratory disease analyses such as chronic obstructive pulmonary disease, asthma and lung cancer. Due to the low imaging contrast and noises execrated at peripheral branches, the topological-complexity and the intra-class imbalance of airway tree, it remains challenging for deep learning-based methods to segment the complete airway tree (on extracting deeper branches). Unlike other organs with simpler shapes or topology, the airway's complex tree structure imposes an unbearable burden to generate the "ground truth" label (up to 7 or 3 hours of manual or semi-automatic annotation per case). Most of the existing airway datasets are incompletely labeled/annotated, thus limiting the completeness of computer-segmented airway. In this paper, we propose a new anatomy-aware multi-class airway segmentation method enhanced by topology-guided iterative self-learning. Based on the natural airway anatomy, we formulate a simple yet highly effective anatomy-aware multi-class segmentation task to intuitively handle the severe intra-class imbalance of the airway. To solve the incomplete labeling issue, we propose a tailored iterative self-learning scheme to segment toward the complete airway tree. For generating pseudo-labels to achieve higher sensitivity (while retaining similar specificity), we introduce a novel breakage attention map and design a topology-guided pseudo-label refinement method by iteratively connecting breaking branches commonly existed from initial pseudo-labels. Extensive experiments have been conducted on four datasets including two public challenges. The proposed method achieves the top performance in both EXACT'09 challenge using average score and ATM'22 challenge on weighted average score. In a public BAS dataset and a private lung cancer dataset, our method significantly improves previous leading approaches by extracting at least (absolute) 6.1% more detected tree length and 5.2% more tree branches, while maintaining comparable precision.

Recently, webly supervised learning (WSL) has been studied to leverage numerous and accessible data from the Internet. Most existing methods focus on learning noise-robust models from web images while neglecting the performance drop caused by the differences between web domain and real-world domain. However, only by tackling the performance gap above can we fully exploit the practical value of web datasets. To this end, we propose a Few-shot guided Prototypical (FoPro) representation learning method, which only needs a few labeled examples from reality and can significantly improve the performance in the real-world domain. Specifically, we initialize each class center with few-shot real-world data as the ``realistic" prototype. Then, the intra-class distance between web instances and ``realistic" prototypes is narrowed by contrastive learning. Finally, we measure image-prototype distance with a learnable metric. Prototypes are polished by adjacent high-quality web images and involved in removing distant out-of-distribution samples. In experiments, FoPro is trained on web datasets with a few real-world examples guided and evaluated on real-world datasets. Our method achieves the state-of-the-art performance on three fine-grained datasets and two large-scale datasets. Compared with existing WSL methods under the same few-shot settings, FoPro still excels in real-world generalization. Code is available at https://github.com/yuleiqin/fopro.

In this paper, we focus on the few-shot domain adaptation problem. With limited training data in target domain, a new approach is emerging to acquire the transferable knowledge from the source domain. Previous methods aligned the embedding space between domains by reducing the pair-wise distance. However, these methods are reporting the misalignment and poor generalization. To solve this problem, we propose a variational feature disentanglement framework. The embedding features are explicitly disentangled into domaininvariant and domain-specific components. The distributions of domain-invariant variance are estimated and aligned by the variational inference. For further disentanglement, the domain-invariant and domain-specific components are separated by the orthogonal constraints of subspaces. The experiments on Digits dataset and VisDA-C dataset demonstrate that the proposed method can outperform the state-of-the-art methods.

Endobronchial intervention is increasingly used as a minimally invasive means for the treatment of pulmonary diseases. In order to reduce the difficulty of manipulation in complex airway networks, robust lumen detection is essential for intraoperative guidance. However, these methods are sensitive to visual artifacts which are inevitable during the surgery. In this work, a cross domain feature interaction (CDFI) network is proposed to extract the structural features of lumens, as well as to provide artifact cues to characterize the visual features. To effectively extract the structural and artifact features, the Quadruple Feature Constraints (QFC) module is designed to constrain the intrinsic connections of samples with various imaging-quality. Furthermore, we design a Guided Feature Fusion (GFF) module to supervise the model for adaptive feature fusion based on different types of artifacts. Results show that the features extracted by the proposed method can preserve the structural information of lumen in the presence of large visual variations, bringing much-improved lumen detection accuracy.

Webly supervised learning has attracted increasing attention for its effectiveness in exploring publicly accessible data at scale without manual annotation. However, most existing methods of learning with web datasets are faced with challenges from label noise, and they have limited assumptions on clean samples under various noise. For instance, web images retrieved with queries of ”tiger cat“ (a cat species) and ”drumstick“ (a musical instrument) are almost dominated by images of tigers and chickens, which exacerbates the challenge of fine-grained visual concept learning. In this case, exploiting both web images and their associated texts is a requisite solution to combat real-world noise. In this paper, we propose Cross-modality Aligned Prototypes (CAPro), a unified prototypical contrastive learning framework to learn visual representations with correct semantics. For one thing, we leverage textual prototypes, which stem from the distinct concept definition of classes, to select clean images by text matching and thus disambiguate the formation of visual prototypes. For another, to handle missing and mismatched noisy texts, we resort to the visual feature space to complete and enhance individual texts and thereafter improve text matching. Such semantically aligned visual prototypes are further polished up with high-quality samples, and engaged in both cluster regularization and noise removal. Besides, we propose collective bootstrapping to encourage smoother and wiser label reference from appearance-similar instances in a manner of dictionary look-up. Extensive experiments on WebVision1k and NUS-WIDE (Web) demonstrate that CAPro well handles realistic noise under both single-label and multi-label scenarios. CAPro achieves new state-of-the-art performance and exhibits robustness to open-set recognition. Codes are available at https://github.com/yuleiqin/capro.

Emotion is a human attitude experience and corresponding behavioral response to objective things. Effective emotion recognition is important for the intelligence and humanization of brain-computer interface (BCI). Although deep learning has been widely used in emotion recognition in recent years, emotion recognition based on electroencephalography (EEG) is still a challenging task in practical applications. Herein, we proposed a novel hybrid model that employs generative adversarial networks to generate potential representations of EEG signals while combining graph convolutional neural networks and long short-term memory networks to recognize emotions from EEG signals. Experimental results on DEAP and SEED datasets show that the proposed model achieved the promising emotion classification performance compared with the state-of-the-art methods.

Endomicroscopy is an emerging imaging modality for real-time optical biopsy. One limitation of existing endomicroscopy based on coherent fibre bundles is that the image resolution is intrinsically limited by the number of fibres that can be practically integrated within the small imaging probe. To improve the image resolution, Super-Resolution (SR) techniques combined with image priors can enhance the clinical utility of endomicroscopy whereas existing SR algorithms suffer from the lack of explicit guidance from ground truth high-resolution (HR) images. In this article, we propose an unsupervised SR pipeline to allow stable offline and kernel-generic learning. Our method takes advantage of both internal statistics and external cross-modality priors. To improve the joint learning process, we present a Sharpness-aware Contrastive Generative Adversarial Network (SCGAN) with two dedicated modules, a sharpness-aware generator and a contrastive-learning discriminator. In the generator, an auxiliary task of sharpness discrimination is formulated to facilitate internal learning by considering the rankings of training instances in various sharpness levels. In the discriminator, we design a contrastive-learning module to mitigate the ill-posed nature of SR tasks via constraints from both positive and negative images. Experiments on multiple datasets demonstrate that SCGAN reduces the performance gap between previous unsupervised approaches and the upper bounds defined in supervised settings by more than 50%, delivering a new state-of-the-art performance score for endomicroscopy super-resolution. Further application on a realistic Voronoi-based pCLE downsampling kernel proves that SCGAN attains PSNR of 35.851 dB, improving 5.23 dB compared with the traditional Delaunay interpolation.

Chromosome recognition is a critical way to diagnose various hematological malignancies and genetic diseases, which is however a repetitive and time-consuming process in karyotyping. To explore the relative relation between chromosomes, in this work, we start from a global perspective and learn the contextual interactions and class distribution features between chromosomes within a karyotype. We propose an end-to-end differentiable combinatorial optimization method, KaryoNet, which captures long-range interactions between chromosomes with the proposed Masked Feature Interaction Module (MFIM) and conducts label assignment in a flexible and differentiable way with Deep Assignment Module (DAM). Specially, a Feature Matching Sub-Network is built to predict the mask array for attention computation in MFIM. Lastly, Type and Polarity Prediction Head can predict chromosome type and polarity simultaneously. Extensive experiments on R-band and G-band two clinical datasets demonstrate the merits of the proposed method. For normal karyotypes, the proposed KaryoNet achieves the accuracy of 98.41% on R-band chromosome and 99.58% on G-band chromosome. Owing to the extracted internal relation and class distribution features, KaryoNet can also achieve state-of-the-art performances on karyotypes of patients with different types of numerical abnormalities. The proposed method has been applied to assist clinical karyotype diagnosis. Our code is available at: https://github.com/xiabc612/KaryoNet.

Detailed anatomical labeling of bronchial trees extracted from CT images can be used as fine-grained maps for intra-operative navigation. To cater to the sparse distribution of airway voxels and large class imbalance in 3D image space, a graph-neural-network-based method is proposed to map branches to nodes in a graph space and assign anatomical labels down to subsegmental level. To address the inherent problem of overlapping distribution of positional and morphological features, especially for subsegmental categories, the proposed method focuses on the relative position between sibling subsegments which is fixed in most cases. The hierarchical nomenclature is represented by multi-level labeling and each category is associated with one or two subtrees in the graph. Hyperedges are used to extract the representation of subtrees while a hypergraph neural network is developed to encode their intrinsic relationship through hyperedge interaction. A filter module is further designed to guide feature aggregation between nodes and hyperedges. With the proposed method, the final accuracies for segmental and subsegmental node classification can achieve 93.6% and 82.0% respectively. The corresponding code is publicly available at https://github.com/haozheng-sjtu/airway-labeling.

Endobronchial intervention is increasingly used as a minimally invasive means for the treatment of pulmonary diseases. In order to acquire the position of bronchoscopy, vision-based localization approaches are clinically preferable but are sensitive to visual variations. The static nature of pre-operative planning makes mapping of intraoperative anatomical features challenging for learning-based methods using visual features alone. In this work, we propose a robust navigation framework based on Vision Kinematic Interaction (VKI) for monocular bronchoscopic videos. To address visual-imbalance between the virtual and real views of bronchoscopy images, a Visual Similarity Network (VSN) is proposed to extract domain-invariant features to represent the lumen structure from endoscopic views, as well as domain-specific features to characterize the surface texture and visual artefacts. To improve the robustness of online estimation of camera pose, we also introduce a Kinematic Refinement Network (KRN) that allows progressive refinement of camera pose estimation based on network prediction and robot control signals. The accuracy of camera localization is validated on phantom and porcine lung datasets from a robotically controlled endobronchial intervention system, with both quantitative and qualitative results demonstrating the performance of the techniques. Results show that the features extracted by the proposed method can preserve the structural information of small airways in the presence of large visual variations along with the much-improved camera localization accuracy. The absolute trajectory errors (ATE) on phantom data and porcine data are 8.01 mm and 8.62 mm respectively.

Representation learning is the critical task for medical image analysis in computer-aided diagnosis. However, it is challenging to learn discriminative features due to the limited size of the dataset and the lack of labels. In this paper, we propose a stochastic routing normalization and neighborhood embedding framework with application to breast tissue classification by learning discriminative features of probe-based confocal laser endomicroscopy. In order to align the low-level and mid-level of pCLE and histology domain, we firstly build the domain-specific normalization module with stochastic activation strategy considering both depth-wise and feature-wise criterion. For high-level features, the latent centers are learned from the histology domain as the template for feature matching. The proposed method is evaluated on a clinical database with 700 pCLE mosaics. The accuracy of image classification with limited training samples demonstrates that the proposed method can outperform previous works on domain alignment.

Segmentation of skin lesions is important for disease diagnoses and treatment planning. Over the years, semi-supervised methods using pseudo labels have boosted the segmentation performance with limited labeled data and abundant unlabeled data. However, the unreliable targets in pseudo labels might lead to meaningless guidance for unlabeled data. In this paper, to solve this issue, we propose a novel confidence aware semi-supervised learning method based on a mean teacher scheme. Concretely, we design a confidence module to predict the model confidence guided by the True Class Probability. Then in the mean teacher framework, the student model gradually learns trustworthy targets from teacher model. To further improve the segmentation quality, we fine-tune the student model with reliable content in pseudo labels. We conduct extensive experiments on 2018 ISIC skin lesion segmentation dataset and our method outperforms other state-of-the-art semi-supervised approaches.

Few-Shot Domain Adaptation aims to transfer knowledge learned from a known domain to a closely related novel domain with only a few training data available for each class. The limited number of target training data makes it challenging to bridge the domain gap and can easily lead to overfitting. In this paper, we proposed HardMix as a regularization technique which interpolates the data in feature space and assigns augmented features with ‘hard’ labels to eliminate the domain discrepancy. In order to generate a better decision boundary and a more compact intra-class distribution, an adaptive triplet loss is proposed to constrain the ‘hard’ samples near the decision boundary. We demonstrated its effectiveness by comparing our results with the state-of-the-art methods on several benchmark datasets.

Training convolutional neural networks (CNNs) for segmentation of pulmonary airway, artery, and vein is challenging due to sparse supervisory signals caused by the severe class imbalance between tubular targets and background. We present a CNNs-based method for accurate airway and artery-vein segmentation in non-contrast computed tomography. It enjoys superior sensitivity to tenuous peripheral bronchioles, arterioles, and venules. The method first uses a feature recalibration module to make the best use of features learned from the neural networks. Spatial information of features is properly integrated to retain relative priority of activated regions, which benefits the subsequent channel-wise recalibration. Then, attention distillation module is introduced to reinforce representation learning of tubular objects. Fine-grained details in high-resolution attention maps are passing down from one layer to its previous layer recursively to enrich context. Anatomy prior of lung context map and distance transform map is designed and incorporated for better artery-vein differentiation capacity. Extensive experiments demonstrated considerable performance gains brought by these components. Compared with state-of-the-art methods, our method extracted much more branches while maintaining competitive overall segmentation performance. Codes and models are available at http://www.pami.sjtu.edu.cn/News/56.

Automated airway segmentation is a prerequisite for pre-operative diagnosis and intra-operative navigation for pulmonary intervention. Due to the small size and scattered spatial distribution of peripheral bronchi, this is hampered by a severe class imbalance between foreground and background regions, which makes it challenging for CNN-based methods to parse distal small airways. In this paper, we demonstrate that this problem is arisen by gradient erosion and dilation of the neighborhood voxels. During back-propagation, if the ratio of the foreground gradient to background gradient is small while the class imbalance is local, the foreground gradients can be eroded by their neighborhoods. This process cumulatively increases the noise information included in the gradient flow from top layers to the bottom ones, limiting the learning of small structures in CNNs. To alleviate this problem, we use group supervision and the corresponding WingsNet to provide complementary gradient flows to enhance the training of shallow layers. To further address the intra-class imbalance between large and small airways, we design a General Union loss function that obviates the impact of airway size by distance-based weights and adaptively tunes the gradient ratio based on the learning process. Extensive experiments on public datasets demonstrate that the proposed method can predict the airway structures with higher accuracy and better morphological completeness than the baselines.

Representation learning is a critical task for medical image analysis in computer-aided diagnosis. However, it is challenging to learn discriminative features due to the limited size of the data set and the lack of labels. In this article, we propose a deep graph-based multimodal feature embedding (DGMFE) framework for medical image retrieval with application to breast tissue classification by learning discriminative features of probe-based confocal laser endomicroscopy (pCLE). We first build a multimodality graph model based on the visual similarity between pCLE data and reference histology images. The latent similar pCLE-histology pairs are extracted by walking with the cyclic path on the graph while the dissimilar pairs are extracted based on the geodesic distance. Given the similar and dissimilar pairs, the latent feature space is discovered by reconstructing the similarity between pCLE and histology images via deep Siamese neural networks. The proposed method is evaluated on a clinical database with 700 pCLE mosaics. The accuracy of image retrieval demonstrates that DGMFE can outperform previous works on feature learning. Especially, the top-1 accuracy in an eight-class retrieval task is 0.739, thus demonstrating a 10% improvement compared to the state-of-the-art method.

Most of existing virtual network embedding (VNE) algorithms only consider how to construct virtual networks more efficiently on a physical infrastructure, without considering the possibility that the constructed virtual networks may be further virtualized to multiple smaller ones. We define the former scenario as single-layer VNE and the later as multi-layer VNE. As the increasing popularity of deploying large datacenter networks and wide area networks with Software Defined Network (SDN) architectures, it becomes a new requirement and possibility to provide multi-layer encapsulated network services for large tenants who have hierarchical organizational structures or need fine-grained service isolation. However, existing VNE algorithm are not specifically designed for the above requirement and not flexible enough to deal with mapping virtual network requirements (VNRs) to a physical network and smaller VNRs to a mapped virtual network. In this paper, we aim to propose a unified and flexible multi-layer VNE algorithm combining with reinforcement learning to solve the embedding of multi-layer VNRs, which can better distinguish the differences between VNRs and physical networks. Simulation results show that our algorithm achieves good performance both in single-layer and multi-layer VNE scenarios.

Content-based medical image retrieval is an important computer-aided diagnosis technique providing the clinicians with interpretative references based on visual similarity. In this paper, we focus on the tasks of histopathological image retrieval for breast cancer diagnosis. The densely-connected multi-magnification (DCMMH) framework is proposed to generate the discriminative binary codes by exploiting the histopathological images with multiple magnification factors. The low-magnification images are boosted by the accumulated similarity based on local patches that also regularize the feature learning of high-magnification images. In order to fully utilize the information across different magnification levels, a densely-connected architecture is finally deployed for high-low magnification pairs of datasets. Experiments on BreakHis dataset demonstrate that, DCMMH outperforms the previous hashing methods on histopathological image retrieval.

Probe-based confocal laser endomicroscopy (pCLE) is an emerging tool for epithelial cancer diagnosis, which enables in-vivo microscopic imaging during endoscopic procedures and facilitates the development of automatic recognition algorithms to identify the status of tissues. In this paper, we propose a transfer recurrent feature learning framework for classification tasks on pCLE videos. At the first stage, the discriminative feature of single pCLE frame is learned via generative adversarial networks based on both pCLE and histology modalities. At the second stage, we use recurrent neural networks to handle the varying length and irregular shape of pCLE mosaics taking the frame-based features as input. The experiments on real pCLE data sets demonstrate that our approach outperforms, with statistical significance, state-of-the-art approaches. A binary classification accuracy of 84.1% has been achieved.

The technique of robust suture detection is vital in many applications including trainee suturing skill evaluation, suture augmentation in robotic-assisted surgery and suture recognition for automatic suturing. Due to the complicated environment of surgery, the detection of a suture is challenged by high deformation and frequent occlusion. In this paper, we propose a deep multi-stage framework for suture detection. The fully convolutional neural networks are firstly used to predict a gradient map which not only serves as a segmentation mask, but also provides useful structure information for the following thread centerline reconstruction. An overlapping map is also predicted to improve the quality of the gradient map in self-intersection area. Based on the gradient map, multiple segments of the thread are extracted and linked to form the whole thread using a curvilinear structure detector. Experiments on two types of threads demonstrate that the proposed method is able to detect the thread with human level performance when the thread is no occlusion or under finite self-intersection.

Task autonomy is an important consideration for the development of future surgical robots. For robot-assisted anastomosis, suture thread detection is a prerequisite for subsequent robot manipulation. Previous works on automatic thread detection are focused on the learning of the models with specific surgical settings that are poorly generalisable to generic settings. In this paper, we propose a joint feature learning framework that caters for the foreground and background adaptation for surgical suture thread detection. The proposed method is developed in the context of semi-supervised and unsupervised domain adaptation, leveraging the labelled training data from the source domain to learn the detection model for unlabelled or partially labelled target domain, which can also be from different types of threads or organs. Based on adversarial learning, we further preserve the semantic identity and introduce curriculum adaptation to generate synthetic data. Experiments on four domain adaptation tasks for suture thread detection demonstrate the strength of the proposed method being able to generate good quality synthetic data and transfer between specific domains with limited or even no labelled data of the target domain.

Automatic diagnosis for fetal echocardiography plays an important part in diagnostic aid in the discrimination of congenital heart disease (CHD). Instead of traditional methods analyzing 2D cardiac echo video that need to find the standard view for discrimination, in this paper, we proposed a new system for automatic discrimination of CHD applying 4D original echocardiogram, which avoids the challenging work of searching standard views. We extracted the features of 3D static structure and 3D motion via 3D SIFT and 3D Histogram of Optical Flow (HOF) from the original 4D (3D+T) echocardiogram data, respectively. Bag of Words (BoW) method was employed to construct the quantized feature. Both static and motion features were fused to form the final image representation. One-Class SVM classifier was utilized to discriminate CHD due to the lack of CHD data and the significant difference in all the CHD cases. Experiments on the real data demonstrate the improved discrimination accuracy due to the fused feature.

Hashing for large-scale multimedia is a popular research topic, attracting much attention in computer vision and visual information retrieval. Previous works mostly focus on hashing the images and texts while the approaches designed for videos are limited. In this paper, we propose a \textit{Supervised Recurrent Hashing} (SRH) that explores the discriminative representation obtained by deep neural networks to design hashing approaches. The long-short term memory (LSTM) network is deployed to model the structure of video samples. The max-pooling mechanism is introduced to embedding the frames into fixed-length representations that are fed into supervised hashing loss. Experiments on UCF-101 dataset demonstrate that the proposed method can significantly outperforms several state-of-the-art methods.

Learning a hashing function for cross-media search is very desirable due to its low storage cost and fast query speed. However, the data crawled from Internet cannot always guarantee good correspondence among different modalities which affects the learning for hashing function. In this paper, we focus on cross-modal hashing with partially corresponded data. The data without full correspondence are made in use to enhance the hashing performance. The experiments on Wiki and NUS-WIDE datasets demonstrates that the proposed method outperforms some state-of-the-art hashing approaches with fewer correspondence information.

Person re-identification (RE-ID) aims at associating the same pedestrian over non-overlapping surveillance scenes. A large number of approaches have emerged in recent years, and they mainly focus on designing middle or high level features to highlight the most discriminative aspects of pedestrians. Due to the nonrigid structure of pedestrians, it is difficult to re-identify pedestrians by low-level features. We investigate the results of conventional person RE-ID approaches, and find that the inadequate utilization of low-level features lead to the poor performance. In this work, we propose a novel framework to utilize the low-level visual features in a more effective way. Given a result obtained from the conventional person RE-ID method, the framework returns a more reasonable result. The framework is extended from the manifold ranking method, and several adjustments are made taking the requirements of person RE-ID into consideration. Our framework is validated through experiments on two person RE-ID datasets (VIPeR and ETHZ), and results from four different conventional approaches show significant improvement.

Superpixels become more and more popular as image preprocessing step in computer vision applications. In this paper, we propose an improved simple linear iterative clustering (SLIC) superpixel approach based on nonstationarity measure (NS-M), which is called nSLIC. An adjustive distance measure is developed in the five-dimensional [labxy] space. The nSLIC superpixel replaces the predefined fixed value of compactness parameter by the nonstationarity measure map of each image, which exploits the image information and is therefore adaptive to the color feature of the image. It also avoids the difficulty of pre-setting compactness parameter and reduces the parameters needed setting to only one indeed. The nSLIC superpixel improves not only segmentation quality bust also computational efficiency by the way of achieving faster convergence. Experiments done on BSD500 dataset show that nSLIC adheres better to image edges meanwhile producing regular and compact superpixels as much as possible, compared to various popular versions of SLIC.

Visual saliency detection has gained its popularity in computer vision in recent years. Depth information is proven as a fundamental element of human vision while it is underutilized in existing saliency detection approaches. In this paper, an effective visual object saliency detection model via RGB and depth cues mutual guided manifold ranking is proposed. The depth features are extracted to guide the saliency ranking of RGB image while the RGB saliency is used as the guide of depth map ranking as well. We obtain the final result by fusing the RGB and depth saliency maps. The experimental result on a benchmark dataset which contains 1000 RGB-D images demonstrates the effectiveness and superior performance compared with several state-of-art methods.

Automatic image annotation is an attractive service for users and administrators of online photo sharing websites. In this paper, we propose an image annotation approach that exploits latent semantic community of labels and multikernel learning (LCMKL). First, a concept graph is constructed for labels indicating the relationship between the concepts. Based on the concept graph, semantic communities are explored using an automatic community detection method. For an image to be annotated, a multikernel support vector machine is used to determine the image's latent community from its visual features. Then, a candidate label ranking based approach is determined by intracommunity and intercommunity ranking. Experiments on the NUS-WIDE database and IAPR TC-12 data set demonstrate that LCMKL outperforms some state-of-the-art approaches.

Automatic image annotation is an important function for online photo sharing service. The concurrence of labels is pretty common in multi-label annotation. In this paper, we propose a novel approach called latent-community and multi-kernel learning (LCMKL). The established graph of labels is regarded as a semantic network. Community detection method is introduced that treats the label set as communities. Multi-kernel learning SVM is adopted for specifying communities and settling difficulty of extracting semantically meaningful entities with some simple features. Experiments on NUS-WIDE database demonstrate that LCMKL outperforms other state-of-the-art approaches.

Bad Data Injection (BDI) in Smart Grid is considered to be the most dangerous cyber attack, as it might lead to energy theft on the end users, false dispatch on the distribution process, and device breakdown on the power generation. State Estimation and Bad Data Detection, which are applied to reduce the observation errors and detect false data in the traditional power grid, could not detect the bad data in smart grid. In this paper, three BDI attack cases in IEEE 14-bus system are designed to bypass the traditional bad data detection. The potential risks on economy and security are analyzed exploiting the MATPOWER. A new method based on Distributed State Estimation (DSE) is proposed to detect BDI, named as DSE-based bad data detection. The power system is divided into several subsystems, and a Chi-squares test is applied to detect the bad data respectively in each subsystem. Simulation results demonstrate that the DSE-based bad data detection can detect all bad data in three attack cases. Moreover, it can locate the bad data in specific subsystem which is helpful for the further identification.

Bad data injection is one of most dangerous attacks in smart grid, as it might lead to energy theft on the end users and device breakdown on the power generation. The attackers can construct the bad data evading the bad data detection mechanisms in power system. In this paper, a novel method, named as Adaptive Partitioning State Estimation (APSE), is proposed to detect bad data injection attack. The basic ideas are: 1) the large system is divided into several subsystems to improve the sensitivity of bad data detection; 2) the detection results are applied to guide the subsystem updating and re-partitioning to locate the bad data. Two attack cases are constructed to inject bad data into an IEEE 39-bus system, evading the traditional bad data detection mechanism. The experiments demonstrate that all bad data can be detected and located within a small area using APSE.