Unsupervised anomaly detection and segmentation methods train a model to learn the training distribution as 'normal'. In the testing phase, they identify patterns that deviate from this normal distribution as 'anomalies'. To learn the 'normal' distribution, prevailing methods corrupt the images and train a model to reconstruct them. During testing, the model attempts to reconstruct corrupted inputs based on the learned 'normal' distribution. Deviations from this distribution lead to high reconstruction errors, which indicate potential anomalies. However, corrupting an input image inevitably causes information loss even in normal regions, leading to suboptimal reconstruction and an increased risk of false positives. To alleviate this, we propose IterMask3D, an iterative spatial mask-refining strategy designed for 3D brain MRI. We iteratively spatially mask areas of the image as corruption and reconstruct them, then shrink the mask based on reconstruction error. This process iteratively unmasks 'normal' areas to the model, whose information further guides reconstruction of 'normal' patterns under the mask to be reconstructed accurately, reducing false positives. In addition, to achieve better reconstruction performance, we also propose using high-frequency image content as additional structural information to guide the reconstruction of the masked area. Extensive experiments on the detection of both synthetic and real-world imaging artifacts, as well as segmentation of various pathological lesions across multiple MRI sequences, consistently demonstrate the effectiveness of our proposed method. Code is available at https://github.com/ZiyunLiang/IterMask3D.

Generating high-fidelity dental radiographs is essential for training diagnostic models. Despite the development of numerous methods for other medical data, generative approaches in dental radiology remain unexplored. Due to the intricate tooth structures and specialized terminology, these methods often yield ambiguous tooth regions and incorrect dental concepts when applied to dentistry. In this paper, we take the first attempt to investigate diffusion-based teeth X-ray image generation and propose ToothMaker, a novel framework specifically designed for the dental domain. Firstly, to synthesize X-ray images that possess accurate tooth structures and realistic radiological styles simultaneously, we design control-disentangled fine-tuning (CDFT) strategy. Specifically, we present two separate controllers to handle style and layout control respectively, and introduce a gradient-based decoupling method that optimizes each using their corresponding disentangled gradients. Secondly, to enhance model's understanding of dental terminology, we propose prior-disentangled guidance module (PDGM), enabling precise synthesis of dental concepts. It utilizes large language model to decompose dental terminology into a series of meta-knowledge elements and performs interactions and refinements through hypergraph neural network. These elements are then fed into the network to guide the generation of dental concepts. Extensive experiments demonstrate the high fidelity and diversity of the images synthesized by our approach. By incorporating the generated data, we achieve substantial performance improvements on downstream segmentation and visual question answering tasks, indicating that our method can greatly reduce the reliance on manually annotated data. Code will be public available at https://github.com/CUHK-AIM-Group/ToothMaker.

Annotation ambiguity due to inherent data uncertainties such as blurred boundaries in medical scans and different observer expertise and preferences has become a major ob-stacle for training deep-learning based medical image segmentation models. To address it, the common practice is to gather multiple annotations from different experts, leading to the setting of multi-rater medical image segmentation. Existing works aim to either merge different annotations into the “groundtruth” that is often unattainable in numerous medical contexts, or generate diverse results, or produce personalized results corresponding to individ-ual expert raters. Here, we bring up a more ambitious goal for multi-rater medical image segmentation, i.e., obtaining both diversified and personalized results. Specifi-cally, we propose a two-stage framework named D-Persona (first Diversification and then Personalization). In Stage I, we exploit multiple given annotations to train a Proba-bilistic U-Net model, with a bound-constrained loss to improve the prediction diversity. In this way, a common latent space is constructed in Stage I, where different latent codes denote diversified expert opinions. Then, in Stage II, we design multiple attention-based projection heads to adaptively query the corresponding expert prompts from the shared latent space, and then perform the personalized medical image segmentation. We evaluated the proposed model on our in-house Nasopharyngeal Carcinoma dataset and the public lung nodule dataset (i.e., LIDC-IDRI). Ex-tensive experiments demonstrated our D-Persona can provide diversified and personalized results at the same time, achieving new SOTA performance for multi-rater medical image segmentation. Our code will be released at https://github.com/ycwu1997/D-Persona.

Abstract Designing a single domain generalization (DG) framework that generalizes from one source domain to arbitrary unseen domains is practical yet challenging in medical image segmentation, mainly due to the domain shift and limited source domain information. To tackle these issues, we reason that domain-adaptive classifier learning and domain-agnostic feature extraction are key components in single DG, and further propose an adaptive infinite prototypes (InfProto) scheme to facilitate the learning of the two components. InfProto harnesses high-order statistics and infinitely samples class-conditional instance-specific prototypes to form the classifier for discriminability enhancement. We then introduce probabilistic modeling and provide a theoretic upper bound to implicitly perform the infinite prototype sampling in the optimization of InfProto. Incorporating InfProto, we design a hierarchical domain-adaptive classifier to elasticize the model for varying domains. This classifier infinitely samples prototypes from the instance and mini-batch data distributions, forming the instance-level and mini-batch-level domain-adaptive classifiers, thereby generalizing to unseen domains. To extract domain-agnostic features, we assume each instance in the source domain is a micro source domain and then devise three complementary strategies, i.e., instance-level infinite prototype exchange, instance-batch infinite prototype interaction, and consistency regularization, to constrain outputs of the hierarchical domain-adaptive classifier. These three complementary strategies minimize distribution shifts among micro source domains, enabling the model to get rid of domain-specific characterizations and, in turn, concentrating on semantically discriminative features. Extensive comparison experiments demonstrate the superiority of our approach compared with state-of-the-art counterparts, and comprehensive ablation studies verify the effect of each proposed component. Notably, our method exhibits average improvements of 15.568% and 17.429% in dice on polyp and surgical instrument segmentation benchmarks.

The computer-aided diagnosis (CAD) for rare diseases using medical imaging poses a significant challenge due to the requirement of large volumes of labeled training data, which is particularly difficult to collect for rare diseases. Although Few-shot learning (FSL) methods have been developed for this task, these methods focus solely on rare disease diagnosis, failing to preserve the performance in common disease diagnosis. To address this issue, we propose the Disentangle then Calibrate with Gradient Guidance (DCGG) framework under the setting of generalized few-shot learning, i.e., using one model to diagnose both common and rare diseases. The DCGG framework consists of a network backbone, a gradient-guided network disentanglement (GND) module, and a gradient-induced feature calibration (GFC) module. The GND module disentangles the network into a disease-shared component and a disease-specific component based on gradient guidance, and devises independent optimization strategies for both components, respectively, when learning from rare diseases. The GFC module transfers only the disease-shared channels of common-disease features to rare diseases, and incorporates the optimal transport theory to identify the best transport scheme based on the semantic relationship among different diseases. Based on the best transport scheme, the GFC module calibrates the distribution of rare-disease features at the disease-shared channels, deriving more informative rare-disease features for better diagnosis. The proposed DCGG framework has been evaluated on three public medical image classification datasets. Our results suggest that the DCGG framework achieves state-of-the-art performance in diagnosing both common and rare diseases.

Domain Adaptive Object Detection (DAOD) transfers an object detector to a novel domain free of labels. However, in the real world, besides encountering novel scenes, novel domains always contain novel-class objects de facto, which are ignored in existing research. Thus, we formulate and study a more practical setting, Adaptive Open-set Object Detection (AOOD), considering both novel scenes and classes. Directly combing off-the-shelled cross-domain and open-set approaches is sub-optimal since their low-order dependence, e.g., the confidence score, is insufficient for the AOOD with two dimensions of novel information. To address this, we propose a novel Structured Motif Matching (SOMA) framework for AOOD, which models the high-order relation with motifs, i.e., statistically significant subgraphs, and formulates AOOD solution as motif matching to learn with high-order patterns. In a nutshell, SOMA consists of Structure-aware Novel-class Learning (SNL) and Structure-aware Transfer Learning (STL). As for SNL, we establish an instance-oriented graph to capture the class-independent object feature hidden in different base classes. Then, a high-order metric is proposed to match the most significant motif as high-order patterns, serving for motif-guided novel-class learning. In STL, we set up a semantic-oriented graph to model the class-dependent relation across domains, and match unlabelled objects with high-order motifs to align the crossdomain distribution with structural awareness. Extensive experiments demonstrate that the proposed SOMA achieves state-of-the-art performance. Codes are available at https://github.com/CityU-AIM-Group/SOMA.

Microsurgery involves the dexterous manipulation of delicate tissue or fragile structures, such as small blood vessels and nerves, under a microscope. To address the limitations of imprecise manipulation of human hands, robotic systems have been developed to assist surgeons in performing complex microsurgical tasks with greater precision and safety. However, the steep learning curve for robot-assisted microsurgery (RAMS) and the shortage of well-trained surgeons pose significant challenges to the widespread adoption of RAMS. Therefore, the development of a versatile training system for RAMS is necessary, which can bring tangible benefits to both surgeons and patients. In this paper, we present a Tactile Internet-Based Micromanipulation System (TIMS) based on a ROS-Django web-based architecture for microsurgical training. This system can provide tactile feedback to operators via a wearable tactile display (WTD), while real-time data is transmitted through the internet via a ROS-Django framework. In addition, TIMS integrates haptic guidance to ‘guide’ the trainees to follow a desired trajectory provided by expert surgeons. Learning from demonstration based on Gaussian Process Regression (GPR) was used to generate the desired trajectory. We conducted user studies to verify the effectiveness of our proposed TIMS, comparing users' performance with and without tactile feedback and/or haptic guidance. For more details of this project, please view our website: https://sites.google.com/view/viewtims/home.

This paper studies a practical domain adaptive (DA) semantic segmentation problem where only pseudo-labeled target data is accessible through a black-box model. Due to the domain gap and label shift between two domains, pseudo-labeled target data contains mixed closed-set and open-set label noises. In this paper, we propose a simplex noise transition matrix (SimT) to model the mixed noise distributions in DA semantic segmentation, and leverage SimT to handle open-set label noise and enable novel target recognition. When handling open-set noises, we formulate the problem as estimation of SimT. By exploiting computational geometry analysis and properties of segmentation, we design four complementary regularizers, i.e., volume regularization, anchor guidance, convex guarantee, and semantic constraint, to approximate the true SimT. Specifically, volume regularization minimizes the volume of simplex formed by rows of the non-square SimT, ensuring outputs of model to fit into the ground truth label distribution. To compensate for the lack of open-set knowledge, anchor guidance, convex guarantee, and semantic constraint are devised to enable the modeling of open-set noise distribution. The estimated SimT is utilized to correct noise issues in pseudo labels and promote the generalization ability of segmentation model on target domain data. In the task of novel target recognition, we first propose closed-to-open label correction (C2OLC) to explicitly derive the supervision signal for open-set classes by exploiting the estimated SimT, and then advance a semantic relation (SR) loss that harnesses the inter-class relation to facilitate the open-set class sample recognition in target domain. Extensive experimental results demonstrate that the proposed SimT can be flexibly plugged into existing DA methods to boost both closed-set and open-set class performance.

This paper studies a practical domain adaptive (DA) semantic segmentation problem where only pseudo-labeled target data is accessible through a black-box model. Due to the domain gap and label shift between two domains, pseudo-labeled target data contains mixed closed-set and open-set label noises. In this paper, we propose a simplex noise transition matrix (SimT) to model the mixed noise distributions in DA semantic segmentation and formulate the problem as estimation of SimT. By exploiting computational geometry analysis and properties of segmentation, we design three complementary regularizers, i.e. volume regularization, anchor guidance, convex guarantee, to approximate the true SimT. Specifically, volume regularization minimizes the volume of simplex formed by rows of the non-square SimT, which ensures outputs of segmentation model to fit into the ground truth label distribution. To compensate for the lack of open-set knowledge, anchor guidance and convex guarantee are devised to facilitate the modeling of open-set noise distribution and enhance the discriminative feature learning among closed-set and open-set classes. The estimated SimT is further utilized to correct noise issues in pseudo labels and promote the generalization ability of segmentation model on target domain data. Extensive experimental results demonstrate that the proposed SimT can be flexibly plugged into existing DA methods to boost the performance. The source code is available at https://github.com/CityU-AIM-Group/SimT.

Automatic medical image segmentation plays a crucial role in many medical applications, such as disease diagnosis and treatment planning. Existing deep learning based models usually regarded the segmentation task as pixel-wise classification and neglected the semantic correlations of pixels across different images, leading to vague feature distribution. Moreover, pixel-wise annotated data is rare in medical domain, and the scarce annotated data usually exhibits the biased distribution against the desired one, hindering the performance improvement under the supervised learning setting. In this paper, we propose a novel Labeled-to-unlabeled Distribution Translation (L2uDT) framework with Semantic-oriented Contrastive Learning (SoCL), mainly for addressing the aforementioned issues in medical image segmentation. In SoCL, a semantic grouping module is designed to cluster pixels into a set of semantically coherent groups, and a semantic-oriented contrastive loss is advanced to constrain group-wise prototypes, so as to explicitly learn a feature space with intra-class compactness and inter-class separability. We then establish a L2uDT strategy to approximate the desired data distribution for unbiased optimization, where we translate the labeled data distribution with the guidance of extensive unlabeled data. In particular, a bias estimator is devised to measure the distribution bias, then a gradual-paced shift is derived to progressively translate the labeled data distribution to unlabeled one. Both labeled and translated data are leveraged to optimize the segmentation model simultaneously. We illustrate the effectiveness of the proposed method on two benchmark datasets, EndoScene and PROSTATEx, and our method achieves state-of-the-art performance, which clearly demonstrates its effectiveness for medical image segmentation. The source code is available at https://github.com/CityU-AIM-Group/L2uDT.

Unsupervised domain adaptation (UDA), aiming to adapt the model to an unseen domain without annotations, has drawn sustained attention in surgical instrument segmentation. Existing UDA methods neglect the domain-common knowledge of two datasets, thus failing to grasp the inter-category relationship in the target domain and leading to poor performance. To address these issues, we propose a graph-based unsupervised domain adaptation framework, named Interactive Graph Network (IGNet), to effectively adapt a model to an unlabeled new domain in surgical instrument segmentation tasks. In detail, the Domain-common Prototype Constructor (DPC) is first advanced to adaptively aggregate the feature map into domain-common prototypes using the probability mixture model, and construct a prototypical graph to interact the information among prototypes from the global perspective. In this way, DPC can grasp the co-occurrent and long-range relationship for both domains. To further narrow down the domain gap, we design a Domain-common Knowledge Incorporator (DKI) to guide the evolution of feature maps towards domain-common direction via a common-knowledge guidance graph and category-attentive graph reasoning. At last, the Cross-category Mismatch Estimator (CME) is developed to evaluate the category-level alignment from a graph perspective and assign each pixel with different adversarial weights, so as to refine the feature distribution alignment. The extensive experiments on three types of tasks demonstrate the feasibility and superiority of IGNet compared with other state-of-the-art methods. Furthermore, ablation studies verify the effectiveness of each component of IGNet. The source code is available at https://github.com/CityU-AIM-Group/Prototypical-Graph-DA.

Multi-modal Magnetic Resonance Imaging (MRI) can provide complementary information for automatic brain tumor segmentation, which is crucial for diagnosis and prognosis. While missing modality data is common in clinical practice and it can result in the collapse of most previous methods relying on complete modality data. Current state-of-the-art approaches cope with the situations of missing modalities by fusing multi-modal images and features to learn shared representations of tumor regions, which often ignore explicitly capturing the correlations among modalities and tumor regions. Inspired by the fact that modality information plays distinct roles to segment different tumor regions, we aim to explicitly exploit the correlations among various modality-specific information and tumor-specific knowledge for segmentation. To this end, we propose a Dual Disentanglement Network (D2-Net) for brain tumor segmentation with missing modalities, which consists of amodality disentanglement stage(MD-Stage) and atumor-region disentanglement stage(TD-Stage). In the MD-Stage, a spatial-frequency joint modality contrastive learning scheme is designed to directly decouple the modality-specific information from MRI data. To decompose tumor-specific representations and extract discriminative holistic features, we propose an affinity-guided dense tumor-region knowledge distillation mechanism in the TD-Stage through aligning the features of a disentangled binary teacher network with a holistic student network. By explicitly discovering relations among modalities and tumor regions, our model can learn sufficient information for segmentation even if some modalities are missing. Extensive experiments on the public BraTS-2018 database demonstrate the superiority of our framework over state-of-the-art methods in missing modalities situations. Codes are available athttps://github.com/CityU-AIM-Group/D2Net.

Unsupervised domain adaptation (UDA) aims to transfer the knowledge from the labeled source domain to the unlabeled target domain. Existing self-training based UDA approaches assign pseudo labels for target data and treat them as ground truth labels to fully leverage unlabeled target data for model adaptation. However, the generated pseudo labels from the model optimized on the source domain inevitably contain noise due to the domain gap. To tackle this issue, we advance a MetaCorrection framework, where a Domain-aware Meta-learning strategy is devised to benefit Loss Correction (DMLC) for UDA semantic segmentation. In particular, we model the noise distribution of pseudo labels in target domain by introducing a noise transition matrix (NTM) and construct meta data set with domain-invariant source data to guide the estimation of NTM. Through the risk minimization on the meta data set, the optimized NTM thus can correct the noisy issues in pseudo labels and enhance the generalization ability of the model on the target data. Considering the capacity gap between shallow and deep features, we further employ the proposed DMLC strategy to provide matched and compatible supervision signals for different level features, thereby ensuring deep adaptation. Extensive experimental results highlight the effectiveness of our methodaagainst existing state-of-the-art methods on three benchmarks.

Semantic segmentation serves as a cornerstone for safety autonomous driving and has been achieved remarkable progress at the price of dense annotations. Unsupervised domain adaptation was widely utilized to addresses this labor-intensive problem, which transfers the knowledge learned from labeled synthetic datset to real-world without any annotations. However, most existing adaptation works predict the segmentation results and domain identification results separately only with the last-layer feature, and ignore the intrinsic relationship among these two tasks. To address this issue, we present a CO-Interactive Network (COINet) for unsupervised adaptive segmentation. In particular, we propose a scale-aware distilled decoder to integrate multi-scale features dynamically through the designed inter-distilled module (IDM) and obtain fine-grained feature representations. A dual-task classifier is advanced with this decoder, to jointly predict the segmentation results and pixel-wise domain prediction results, which extracts shared complementary information for accurate segmentation. We further devise a co-interactive loss to explicitly model the intrinsic relationship among the segmentation and domain prediction, enabling the feature distribution alignment in pixel-level and an optimal segmentation decision boundary. We demonstrate the effectiveness of the proposed COINet on benchmark adaptation settings with extensive experimental and ablation results, and our model shows favorable performance against existing algorithms.

Terahertz modulation is always realized by the dynamic meta-surface with quasi-optical transmission mode, which limited the modulation speed and application in the integrated system. Here we propose a new way by combing the microstructure, active GaAs Schottky diodes, and microstrip to construct an active meta-chip that could realize low insertion loss, high modulation depth, and on-chip THz modulation. The diodes are controlled by the bias voltage to change the switching characteristics. Then, the resonant frequency is controlled to realize the amplitude modulation of THz (terahertz) waves. The simulation results indicate that the modulator can achieve an insertion loss of 2dB and a maximum modulation depth of 97.1 %.

Accurate segmentation of the polyps from colonoscopy images provides useful information for the diagnosis and treatment of colorectal cancer. Despite deep learning methods advance automatic polyp segmentation, their performance often degrades when applied to new data acquired from different scanners or sequences (target domain). As manual annotation is tedious and labor-intensive for new target domain, leveraging knowledge learned from the labeled source domain to promote the performance in the unlabeled target domain is highly demanded. In this work, we propose a mutual-prototype adaptation network to eliminate domain shifts in multi-centers and multi-devices colonoscopy images. We first devise a mutual-prototype alignment (MPA) module with the prototype relation function to refine features through self-domain and cross-domain information in a coarse-to-fine process. Then two auxiliary modules: progressive self-training (PST) and disentangled reconstruction (DR) are proposed to improve the segmentation performance. The PST module selects reliable pseudo labels through a novel uncertainty guided self-training loss to obtain accurate prototypes in the target domain. The DR module reconstructs original images jointly utilizing prediction results and private prototypes to maintain semantic consistency and provide complement supervision information. We extensively evaluate the proposed model in polyp segmentation performance on three conventional colonoscopy datasets: CVC-DB, Kvasir-SEG, and ETIS-Larib. The comprehensive experimental results demonstrate that the proposed model outperforms state-of-the-art methods.

The degradation in image resolution harms the performance of medical image diagnosis. By inferring high-frequency details from low-resolution (LR) images, super-resolution (SR) techniques can introduce additional knowledge and assist high-level tasks. In this paper, we propose a SR enhanced diagnosis framework, consisting of an efficient SR network and a diagnosis network. Specifically, a Multi-scale Refined Context Network (MRC-Net) with Refined Context Fusion (RCF) is devised to leverage global and local features for SR tasks. Instead of learning from scratch, we first develop a recursive MRC-Net with temporal context, and then propose a recursion distillation scheme to enhance the performance of MRC-Net from the knowledge of the recursive one and reduce the computational cost. The diagnosis network jointly utilizes the reliable original images and more informative SR images by two branches, with the proposed Sample Affinity Interaction (SAI) blocks at different stages to effectively extract and integrate discriminative features towards diagnosis. Moreover, two novel constraints, sample affinity consistency and sample affinity regularization, are devised to refine the features and achieve the mutual promotion of these two branches. Extensive experiments of synthetic and real LR cases are conducted on wireless capsule endoscopy and histopathology images, verifying that our proposed method is significantly effective for medical image diagnosis.

The automatic segmentation of polyp in endoscopy images is crucial for early diagnosis and cure of colorectal cancer. Existing deep learning-based methods for polyp segmentation, however, are inadequate due to the limited annotated dataset and the class imbalance problems. Moreover, these methods obtained the final polyp segmentation results by simply thresholding the likelihood maps at an eclectic and equivalent value (often set to 0.5). In this paper, we propose a novel ThresholdNet with a confidence-guided manifold mixup (CGMMix) data augmentation method, mainly for addressing the aforementioned issues in polyp segmentation. The CGMMix conducts manifold mixup at the image and feature levels, and adaptively lures the decision boundary away from the under-represented polyp class with the confidence guidance to alleviate the limited training dataset and the class imbalance problems. Two consistency regularizations, mixup feature map consistency (MFMC) loss and mixup confidence map consistency (MCMC) loss, are devised to exploit the consistent constraints in the training of the augmented mixup data. We then propose a two-branch approach, termed ThresholdNet, to collaborate the segmentation and threshold learning in an alternative training strategy. The threshold map supervision generator (TMSG) is embedded to provide supervision for the threshold map, thereby inducing better optimization of the threshold branch. As a consequence, ThresholdNet is able to calibrate the segmentation result with the learned threshold map. We illustrate the effectiveness of the proposed method on two polyp segmentation datasets, and our methods achieved the state-of-the-art result with 87.307% and 87.879% dice score on the EndoScene dataset and the WCE polyp dataset. The source code is available at https://github.com/Guo-Xiaoqing/ThresholdNet.

Linguistic steganography based on text carrier auto-generation technology is a current topic with great promise and challenges. Limited by the text automatic generation technology or the corresponding text coding methods, the quality of the steganographic text generated by previous methods is inferior, which makes its imperceptibility unsatisfactory. In this paper, we propose a linguistic steganography based on recurrent neural networks, which can automatically generate high-quality text covers on the basis of a secret bitstream that needs to be hidden. We trained our model with a large number of artificially generated samples and obtained a good estimate of the statistical language model. In the text generation process, we propose fixed-length coding and variable-length coding to encode words based on their conditional probability distribution. We designed several experiments to test the proposed model from the perspectives of information hiding efficiency, information imperceptibility, and information hidden capacity. The experimental results show that the proposed model outperforms all the previous related methods and achieves the state-of-the-art performance.