The rapid advancement of diffusion models has increased the need for customized image generation. However, current customization methods face several limitations: 1) typically accept either image or text conditions alone; 2) customization in complex visual scenarios often leads to subject leakage or confusion; 3) image-conditioned outputs tend to suffer from inconsistent backgrounds; and 4) high computational costs. To address these issues, this paper introduces Multi-party Collaborative Attention Control (MCA-Ctrl), a tuning-free method that enables high-quality image customization using both text and complex visual conditions. Specifically, MCA-Ctrl leverages two key operations within the self-attention layer to coordinate multiple parallel diffusion processes and guide the target image generation. This approach allows MCA-Ctrl to capture the content and appearance of specific subjects while maintaining semantic consistency with the conditional input. Additionally, to mitigate subject leakage and confusion issues common in complex visual scenarios, we introduce a Subject Localization Module that extracts precise subject and editable image layers based on user instructions. Extensive quantitative and human evaluation experiments show that MCA-Ctrl outperforms existing methods in zero-shot image customization, effectively resolving the mentioned issues.

Multi-modal medical images provide complementary soft-tissue characteristics that aid in the screening and diagnosis of diseases. However, limited scanning time, image corruption and various imaging protocols often result in incomplete multi-modal images, thus limiting the usage of multi-modal data for clinical purposes. To address this issue, in this paper, we propose a novel unified multi-modal image synthesis method for missing modality imputation. Our method overall takes a generative adversarial architecture, which aims to synthesize missing modalities from any combination of available ones with a single model. To this end, we specifically design a Commonality- and Discrepancy-Sensitive Encoder for the generator to exploit both modality-invariant and specific information contained in input modalities. The incorporation of both types of information facilitates the generation of images with consistent anatomy and realistic details of the desired distribution. Besides, we propose a Dynamic Feature Unification Module to integrate information from a varying number of available modalities, which enables the network to be robust to random missing modalities. The module performs both hard integration and soft integration, ensuring the effectiveness of feature combination while avoiding information loss. Verified on two public multi-modal magnetic resonance datasets, the proposed method is effective in handling various synthesis tasks and shows superior performance compared to previous methods.

The relationship between various objects in real life is very important and universal. However, existing object detection models, especially Two-stage models, mostly rely solely on instance learning of individual objects, which use limited global information to extract regions of interest and neglect the correlation between object instances. Therefore, this article proposes an Object Correlation Matrix (OCM) for measuring the correlation between objects, obtained by statistical analysis of instance information in the MS COCO dataset. Based on OCM, we design a Secondary Scoring Module (SSM) that combines the confidence and correlation of surrounding objects to correct the predicted outputs of the original network. Then apply the SSM to the post-processing stage, which is entirely independent of the original network and does not require adding any parameters or retraining the original network. Hence, this module is easy to embed in object detection networks and has achieved performance improvement.

Camouflaged object detection (COD) aims to accurately recognize targets in intricate environments that blend into the background. Although numerous camouflage object identification techniques have demonstrated effectiveness, they often possess a substantial number of parameters. Therefore, we propose a new lightweight edge-guided multilevel feature fusion camouflaged object detection network, codenamed as LEMFNet. Initially, we adopt a lightweight CNN network model for feature extraction to reduce model complexity. Subsequently, we introduced a neighborhood feature association module (NFAM) to integrate feature information from different stages to obtain complementary feature representations and enhance the overall model performance. Furthermore, to obtain a more complete object structure, we introduce a boundary aggregation module (BAM) to delve into the edge semantics associated with the target and integrate the edge features into the proposed edge-guided aggregation module (EGAM). Experiments on three challenging benchmarks demonstrate our approach, with fewer parameters, achieves comparable or superior performance, effectively balancing resource utilization and accuracy.

Radiologists possess diverse training and clinical experiences, leading to variations in the segmentation annotations of lung nodules and resulting in segmentation uncertainty. Conventional methods typically select a single annotation as the learning target or attempt to learn a latent space comprising multiple annotations. However, these approaches fail to leverage the valuable information inherent in the consensus and disagreements among the multiple annotations. In this paper, we propose an Uncertainty-Aware Attention Mechanism (UAAM) that utilizes consensus and disagreements among multiple annotations to facilitate better segmentation. To this end, we introduce the Multi-Confidence Mask (MCM), which combines a Low-Confidence (LC) Mask and a High-Confidence (HC) Mask. The LC mask indicates regions with low segmentation confidence, where radiologists may have different segmentation choices. Following UAAM, we further design an Uncertainty-Guide Multi-Confidence Segmentation Network (UGMCS-Net), which contains three modules: a Feature Extracting Module that captures a general feature of a lung nodule, an Uncertainty-Aware Module that produces three features for the annotations' union, intersection, and annotation set, and an Intersection-Union Constraining Module that uses distances between the three features to balance the predictions of final segmentation and MCM. To comprehensively demonstrate the performance of our method, we propose a Complex-Nodule Validation on LIDC-IDRI, which tests UGMCS-Net's segmentation performance on lung nodules that are difficult to segment using common methods. Experimental results demonstrate that our method can significantly improve the segmentation performance on nodules that are difficult to segment using conventional methods.

In this paper, we propose a novel feature decoupling method to tackle two critical problems in the lung nodule segmentation task: (i) ambiguity of nodule boundary leads to the imprecise segmentation boundary and (ii) the high false positive rate of segmentation result. Our motivation is that an accurate segmentation network needs explicitly modeling the nodule boundary and texture information, and suppressing the noise information. To do so, a novel Deep Feature Decoupling Module (DFDM) is proposed to decouple the nodule boundary, noise, and texture information from the original feature maps. The decoupled boundary and texture information is used to benefit the segmentation, and the noise information is removed from the input features to reduce the false positive rate. The proposed DFDM consists of three parallel branches, including Boundary Sensitive Branch (BSB), Noise Removal Branch (NRB), and Texture Preserving Branch (TPB) to decouple the mentioned three information, respectively. In particular, we design our BSB with a novel architecture to effectively capture the boundary information of lung nodules. We apply the proposed DFDM to the U-Net architecture and achieve convincing segmentation results on the LIDC–IDRI dataset. Code and models are available at https://github.com/chinichenw/DFDM.

Many medical images domains suffer from inherent ambiguities. A feasible approach to resolve the ambiguity of lung nodule in the segmentation task is to learn a distribution over segmentations based on a given 2D lung nodule image. Whereas lung nodule with 3D structure contains dense 3D spatial information, which is obviously helpful for resolving the ambiguity of lung nodule, but so far no one has studied it. To this end we propose a probabilistic generative segmentation model consisting of a V-Net and a conditional variational autoencoder. The proposed model obtains the 3D spatial information of lung nodule with V-Net to learn a density model over segmentations. It is capable of efficiently producing multiple plausible semantic lung nodule segmentation hypotheses to assist radiologists in making further diagnosis to resolve the present ambiguity. We evaluate our method on publicly available LIDC-IDRI dataset and achieves a new state-of-the-art result with 0.231±0.005 in $D_{GED}^2$. This result demonstrates the effectiveness and importance of leveraging the 3D spatial information of lung nodule for such problems. Code is available at: https://github.com/jiangjiangxiaolong/PV-Net.

Many deep learning methods have been proposed for the diagnosis of COVID-19 since the global pandemic. However, few studies have focused on the disease course classification of COVID-19, which is crucial for radiologists to determine treatment plans. This paper proposes a Multi-Modal Fusion Cascade (MMFC) framework for this task, which can make the most of multi-modal information, including CT image and bio-information (laboratory examination, clinical characterization, etc.). The proposed framework consists of two parts: Bio-Visual Feature Learning Module (BFL) and Joint Decision Module (JD). Firstly, BFL learns the discriminative visual features from the mediastinal window with the assistance of bio-information. According to the official Treatment Protocol of China, the bio-information is chosen and helps the BFL better extract the images’ bio-visual features and then obtained a disease course classification result based on CT images. Secondly, JD uses bio-information again and fuses the confidence of BFL’s result to make the joint decision. Experimental results show that our framework significantly improves accuracy and sensitivity compared to the baseline.

The important cues for a realistic lung nodule synthesis include the diversity in shape and background, controllability of semantic feature levels, and overall CT image quality. To incorporate these cues as the multiple learning targets, we introduce the Multi-Target Co-Guided Adversarial Mechanism, which utilizes the foreground and background mask to guide nodule shape and lung tissues, takes advantage of the CT lung and mediastinal window as the guidance of spiculation and texture control, respectively. Further, we propose a Multi-Target Co-Guided Synthesizing Network with a joint loss function to realize the co-guidance of image generation and semantic feature learning. The proposed network contains a Mask-Guided Generative Adversarial Sub-Network (MGGAN) and a Window-Guided Semantic Learning Sub-Network (WGSLN). The MGGAN generates the initial synthesis using the mask combined with the foreground and background masks, guiding the generation of nodule shape and background tissues. Meanwhile, the WGSLN controls the semantic features and refines the synthesis quality by transforming the initial synthesis into the CT lung and mediastinal window, and performing the spiculation and texture learning simultaneously. We validated our method using the quantitative analysis of authenticity under the Fréchet Inception Score, and the results show its state-of-the-art performance. We also evaluated our method as a data augmentation method to predict malignancy level on the LIDC-IDRI database, and the results show that the accuracy of VGG-16 is improved by 5.6%. The experimental results confirm the effectiveness of the proposed method.

Accurate segmentation for lung nodules in lung computed tomography (CT) scans plays a key role in the early diagnosis of lung cancer. Many existing methods, especially U-Net, have made significant progress in lung nodule segmentation. However, due to the complex shapes of lung nodules and the similarity of visual characteristics between nodules and lung tissues, an accurate segmentation with low false positives of lung nodules is still a challenging problem. Considering the fact that both boundary and texture information of lung nodules are important for obtaining an accurate segmentation result, we propose a novel Mask and Texture-driven Generative Adversarial Network (MTGAN) with a joint multi-scale L1 loss for lung nodule segmentation, which takes full advantages of U-Net and adversarial training. The proposed MTGAN leverages adversarial learning strategy guided by the boundary and texture information of lung nodules to generate more accurate segmentation results with lesser false positives. We validate our model with the LIDC-IDRI dataset, and experimental results show that our method achieves excellent segmentation results for a variety of lung nodules, especially for juxtapleural nodules and low-dense nodules. Without any bells and whistles, the proposed MTGAN achieves significant segmentation performance with the Dice similarity coefficient (DSC) of 85.24% on the LIDC-IDRI dataset.

Computer-Aided Diagnosis (CAD) systems for lung nodule diagnosis based on deep learning have attracted much attention in recent years. However, most existing methods ignore the relationships between the segmentation and classification tasks, which leads to unstable performances. To address this problem, we propose a novel multi-task framework, which can provide lung nodule segmentation mask, malignancy prediction, and medical features for interpretable diagnosis at the same time. Our framework mainly contains two sub-network: (1) Multi-Channel Segmentation Sub-network (MSN) for lung nodule segmentation, and (2) Joint Classification Sub-network (JCN) for interpretable lung nodule diagnosis. In the proposed framework, we use U-Net down-sampling processes for extracting low-level deep learning features, which are shared by two sub-networks. The JCN forces the down-sampling processes to learn better low-level deep features, which lead to a better construct of segmentation masks. Meanwhile, two additional channels constructed by OTSU and super-pixel (SLIC) methods, are utilized as the guideline of the feature extraction. The proposed framework takes advantages of deep learning methods and classical methods, which can significantly improve the performances of all tasks. We evaluate the proposed framework on public dataset LIDC-IDRI. Our framework achieves a promising Dice score of 86.43% in segmentation, 87.07% in malignancy level prediction, and convincing results in interpretable medical feature predictions.

Residuals and inception are two commonly used module that makes the network deeper and wider to achieve better performance. And the combination of these two modules which is usually referred to as inception-resnet can get a better result. In this paper, we propose a new type of combination to give full play to the role of residuals and inception, making network learning more abundant features. The new proposed module is called Residual Inception (RI) which enjoys the same width as the inception module in GoogLeNet. In RI, each parallel cascade structure is replaced by a densely block or a modified residual block for gaining a better performance and a lower computational cost. Finally, we evaluate our proposed network on three highly competitive datasets and the results demonstrate its superiority in comparison with the state-of-the-art.