Accurately segmenting polyp regions in colonoscopy images is crucial for the diagnosis and intervention of colorectal cancer. However, the task of polyp segmentation remains challenging due to the diverse size and shape variations among polyps, their extreme similarity to the background, and frequent rotation of the lens, which further increases the diversity in polyp presentation. To address these challenges effectively, we propose a comprehensive polyp segmentation network (CPSNet). Specifically, we introduce a Comprehensive Spatial Feature Extraction Module (CFEM) that progressively and densely integrates features while forming receptive windows with various shapes. This enables enhanced perception of polyps at manifold sizes and shapes. Additionally, we propose a Fine-grained Region Strengthen Module (FGSM) to supplement uncertain areas around polyps by mitigating background noise interference. In terms of training strategy, we further introduce a Rotation-augmented Constrained Loss (RC Loss), which reinforces consistency constraints on polyp images under multiple rotation angles. Qualitative and quantitative experiments conducted on five public datasets demonstrate both the plug-and-play capability of CFEM as well as the effectiveness and excellence achieved by CPSNet.

In semi-supervised medical image segmentation, two main challenges arise. First, the quality of pseudo-labels generated by segmentation networks in data-limited scenarios is often poor, reducing segmentation accuracy. Second, many methods fail to effectively utilize the temporal context in video data. Moreover, the intricate relationships among multiple targets make the conventional teacher-student network consistency loss unsuitable for multi-target tasks, leading to inaccurate feature capture and degraded performance. To address these issues, we propose a registration-based pseudo-label generation strategy that produces pseudo-labels more closely aligned with actual labels. Additionally, we introduce multiple guidance mechanisms into the teacher-student network to ensure correct optimization. Experimental results show that our approach significantly outperforms leading semi-supervised segmentation methods. Specifically, at a 50% label rate, our method improves the Dice coefficient by 6.9% over existing methods.

To improve the accuracy and robustness of left ventricular segmentation, This paper proposes a diffusion probabilistic model-based left ventricular segmentation network (DPM-LVSN). DPM-LVSN integrates a U-shaped encoder-decoder for feature extraction, self-attention for semantic information, and a diffusion model for noise recovery. And the efficiency is improved by using feature fusion based on Fourier transform and time step fusion based on confidence. The experimental results on EchoNet-Dynamic dataset show the effectiveness of the proposed method

For pursuing accurate skeleton-based action recognition, many existing graph-based approaches deploy the higher-order polynomials of the skeletal adjacency matrix to model the node correlations of distant neighbours. To further capture robust graphical patterns, a novel multi-scale graph convolution operator is proposed, which enables to aggregate multi-scale dependencies and capture long-range joint relationships on human skeleton graph. Additionally, a novel corrective contrastive learning strategy is proposed, which aims to distinguish the representative clues and calibrate the confused action clips in the feature space. Comprehensive experiments validate the effectiveness and superiority of our proposed method over state-of-the-art approaches on three large-scale datasets: NTU RGB+D 60, NTU RGB+D 120, and Kinetics Skeleton 400.

Developing efficient collaborative strategies for UAV swarms is crucial for achieving accurate and rapid execution of the Multi-Target Detecting (MTD) tasks which involve two primary stages in practical scenarios: dispersed search by multi-UAVs in unknown dynamic environments to locate targets, and subsequent aggregation to gather all targets information within the scene, which called detecting and aggregation processes. In recent years, several collaborative strategy methods have been developed for application in UAV swarm mission scenarios. These methods are typically designed for single-stage tasks, and therefore their performance is likely to be suboptimal when applied to multi-stage tasks like the MTD tasks, which have distinctly different characteristics and objectives across stages. In this paper, we propose a novel integrated deep reinforcement learning decision framework that can offer effective collaborative strategies for tasks characterized by distinct stages, denoted as the STDGNet. The STDGNet comprises a Transformer-based Deep Graph Network (TDGN) module alongside two Specialized optimization strategies: the location-dispersion strategy and the cluster-action-consistency strategy. The TDGN module is designed to extract features from observations and interaction dynamics among UAVs, aimed at generating collaborative strategies. The integration of two specialized strategies enables the STDGNet framework to adapt well to multi-stage tasks: during the detection stage, the location-dispersion strategy maintains UAV dispersal to expedite the discovery of more targets; during the aggregation stage, the cluster-action-consistency strategy ensures that UAVs within the same cluster move in the same direction, facilitating the formation of interconnected communication networks. To assess the efficiency and resilience of the proposed framework, we construct an MTD environment where extensive experimentation shows that the STDGNet framework surpasses baseline methodologies.

The existing approaches for skeleton-based action recognition based on graph convolutional networks (GCNs) primarily emphasize the construction of human skeletal structure by leveraging inherent connections. However, the static skeletal topology used across all action categories fails to capture discriminative relationships between joint pairs, while current graph structures struggle to model dynamic motion information, limiting their ability to represent both temporal and motion-specific dependencies. To address this limitation, we propose the decoupled static-dynamic co-occurrence graph convolution (DSDC-GConv), which specifically aims to learn and adapt the graph topology by refining the inter-frame and intra-frame joint dependencies through decomposed manner. Additionally, a multi-level context-aware module is proposed to comprehensively model the latent saliencies of multiple domains in skeletal sequences. This module refines the spatial nodes, temporal dynamics, channel-wise characteristics, and motional dependencies within the graph convolution block. Furthermore, a hierarchical densely connected temporal convolution is proposed to enhance the representation of local features through partial dense connections and enrich the temporal information during the convolution process. Findings from our evaluations on five large-scale benchmark datasets (i.e., NTU RGB+D 60, NTU RGB+D 120, Kinetics Skeleton 400, Northwestern-UCLA, PKU-MMD) demonstrate the effectiveness and superiority of our proposed method over competing approaches, with an recognition accuracy of 93.0% and 97.1% on NTU RGB+D 60, 89.9% and 90.6% on NTU RGB+D 120, 38.6% and 63.4% on Kinetics Skeleton 400, 97.4% on Northwestern-UCLA, 97.6% and 63.6% on PKU-MMD.

Feature matching is widely applied in the image processing field. However, both traditional feature matching methods and previous deep learning-based methods struggle to accurately match the features with severe deformations and large displacements, particularly in gastroscopy scenario. To fill this gap, an effective feature matching framework named TransMatch is proposed, which addresses the largely displacements issue by matching features with global information leveraged via Transformer structure. To address the severe deformation of features, an effective bridging strategy with a novel bidirectional quadratic interpolation network is employed. This bridging strategy decomposes and simplifies the matching of features undergoing severe deformations. A deblurring module for gastroscopy scenario is specifically designed to address the potential blurriness. Experiments have illustrated that proposed method achieves state-of-the-art performance of feature matching and frame interpolation in gastroscopy scenario. Moreover, a large-scale gastroscopy dataset is also constructed for multiple tasks.

In semi-supervised medical image segmentation, the use of CutMix in the Mean Teacher architecture is considered an effective strong data augmentation strategy. However, we believe that randomly selecting patches from the source image might mislead the model into learning unexpected feature representations. Therefore, we propose Gradient Saliency-aware CutMix for semi-supervised medical image segmentation (GSC-Seg). Utilizing the gradient from pre-trained models to detect salient regions and then copies and pastes the large gradient areas from labeled data into corresponding areas of unlabeled data based on the gradient, and vice versa, guiding the model to learn more appropriate feature representations. Furthermore, we propose a gradient augmentation strategy, which generates disruptions in the gradient through the network itself and enhances the gradient representation abilities of the network. Experiment results show that our approach achieves the state-of-the-art performance on three medical image segmentation datasets. Code is available at https://github.com/UESTC-Med424-JYX/GSC-Seg.

A modality gap exists between infrared and visible images, presenting challenges for image fusion. Despite the modality heterogeneity, both types of images inherently capture the same scene, suggesting the presence of common information. Effectively extracting shared features while distinguishing modality-specific ones is pivotal for bridging this gap and achieving superior fusion outcomes. To address this, we propose the S hared-S pecific F usion Net work (S2F-Net). The S2F-Net introduces a three-branch feature extractor, which retains two branches for extracting features from each modality, innovatively creating an additional branch dedicated to facilitating the separation of shared features from modality-specific ones. This facilitates guiding the fusion of cross-modal information to generate efficient fusion features, ensuring the effective integration of complementary information from different modalities. To achieve feature fusion and image reconstruction, we propose two fusion modules: the Cross-modality Attention-Guided Fusion Module (CAGFM) and the Multi-Level Fusion Module (MLFM). The former utilizes shared and specific features by employing cross-modality channel attention, enabling effective integration of information across modalities. The latter facilitates feature interaction across different levels. Additionally, to effectively disentangle shared and specific features, we introduce the shared-specific learning module. Extensive experiments conducted on open-source datasets validate the superior performance of our proposed method.

Medical images often contain sensitive information, and one typical security measure is to encrypt medical images prior to storage and analysis. A number of solutions, such as those utilizing deep learning, have been proposed for medical image encryption and decryption. However, our research shows that deep learning-based encryption models can potentially be vulnerable to backdoor attacks. In this paper, a backdoor attack paradigm for encryption and decryption network is proposed and corresponding attacks are respectively designed for encryption and decryption scenarios. For attacking the encryption model, a backdoor discriminator is adopted, which is randomly trained with the normal discriminator to confuse the encryption process. In the decryption scenario, a number of subnetwork parameters are replaced and the subnetwork can be activated when detecting the trigger embedded into the input (encrypted image) to degrade the decryption performance. Considering the model performance degradation due to parameter replacement, the model pruning is also adopted to further strengthen the attacking performance. Furthermore, the image steganography is adopted to generate invisible triggers for each image; subsequently, improving the stealthiness of backdoor attacks. Our research on designing backdoor attacks for encryption and decryption network can serve as an attacking mode for such networks, and provides another research direction for improving the security of such models. This research is also one of the earliest works to realize the backdoor attack on the deep learning based medical encryption and decryption network to evaluate the security performance of these networks. Extensive experimental results show that the proposed method can effectively threaten the security performance both for the encryption and decryption network.

As a popular research topic in computer vision, video frame interpolation is widely used in video processing tasks. However, this task is often limited by slow processing speed or high memory consumption in practical applications. To address these drawbacks, a frame interpolation network focusing on motion regions named MFNet is proposed, which consists of a sampler for adaptive and efficient separation of motion regions from the background, a fine-grained module for direct approximation of intermediate streams, and a lightweight module for bi-directional optical stream fusion. Extensive experiments show that our MFNet achieves optimal accuracy on some frame interpolation tasks and is much faster than other state-of-the-art methods. In addition, transplantation of the core components of MFNet to other frame interpolation networks can significantly improve the performance.

Most existing semantic segmentation methods primarily employ supervised learning with discriminative models. Although these methods are straightforward, they overlook the modeling of underlying data distributions. In this paper, we propose a novel medical image segmentation framework called Diff-SFCT based on Diffusion Model. We formulate semantic segmentation as a generative problem for segmentation masks, replacing the conventional pixel-wise discriminative learning with a latent prior learning process to produce more accurate segmentation results. Diff-SFCT employs a backbone network combining Convolutional Neural Network (CNN) and Transformer, and utilizes the local perception of CNN and the global information modeling capability of Transformer. In Diff-SFCT, we design a Semantic Encoder that effectively extracts fine-grained semantic features from real images. Meanwhile, we propose a novel Spatial-Frequency Cross Transformer (SFCT) framework, which can effectively model and interact the global features of the diffuse noise mask and the real semantic features, reducing the domain gap between the two and enhancing the model’s representational capacity. Additionally, to preserve spatial and frequency information in the diffusion model, we design a Spatial-Frequency Attention Module (SFAM) as part of the Convolutional Block. This module improves the model’s spatial and frequency perception abilities while incurring negligible computational overhead. Experimental results evince that our DiffSFCT substantially outperforms other segmentation methods, exhibiting remarkable performance across various medical image segmentation datasets.

Histopathology diagnosis is an important standard for breast tumors identifying. However, histopathology image analysis is complex, tedious and error-prone, due to the super-resolution image. In recent years, deep learning technology has been successfully applied to histopathology image analysis and made great progress. The well-known deep neural networks usually have tens of million parameters, which consume much memory to deploy the state-of-the-art model. In addition, deep neural networks rely on high-performance hardware resources, which impede the deployment of state-of-the-art model on portable equipment. In this work, a novel framework which consists of a weight accumulation method and a lightweight fast neural network (FastNet) was proposed for tumor fast identification (TFI) in mobile computer-assisted devices. The weight accumulation method was designed to obtain the tissue mask regions of interest and remove the useless background area in histopathology images, which greatly reduces the redundant computation cost. Furthermore, we proposed the lightweight FastNet to improve the computational efficiency on mobile devices. A novel attention loss function was designed and applied in FastNet. The attention loss function pays more attention on the positive samples and the indistinguishable samples, which greatly improves performance. The proposed FastNet was compared with three state-of-the-art methods commonly used for image classification and object detection. Experimental results indicated that FastNet achieves highest recall of 96.94%, highest F1 score of 97.33% and highest accuracy of 97.34%, besides least trainable parameters of 0.22M and smallest floating point operations of 210M FLOPs.

In the area of medical image segmentation, the spatial information can be further used to enhance the image segmentation performance. And the 3D convolution is mainly used to better utilize the spatial information. However, how to better utilize the spatial information in the 2D convolution is still a challenging task. In this paper, we propose an image segmentation network based on reinforcement learning (RLSegNet), which can translate the image segmentation process into a serial of decision-making problem. The proposed RLSegNet is a U-shaped network, which is composed of three components: the feature extraction network, the Mask Prediction Network (MPNet), and the up-sampling network with the cascade attention module. The deep semantic feature in the image is first extracted by adopting the feature extraction network. Then, the Mask Prediction Network (MPNet) is proposed to generate the prediction mask for the current frame based on the prior knowledge (segmentation result). And the proposed cascade attention module is mainly used to generate the weighted feature mask so that the up-sampling network pays more attention to the interesting region. Specifically, the state, action and reward used in the reinforcement learning are redesigned in the proposed RLSegNet to translate the segmentation process as the decision-making process, which performs as the reinforcement learning to realize the brain tumor segmentation. Extensive experiments are conducted on the BRATS 2015 dataset to evaluate the proposed RLSegNet. The experimental results demonstrate that the proposed method can achieve a better segmentation performance, in comparison with other state-of-the-art methods.

Mitigating adversarial deep learning attacks remains challenging, partly because of the ease and low cost in carrying out such attacks. Therefore, in this paper, we focus on the understanding of universal adversarial example attack on image classification models. Specifically, we seek to understand the difference(s) between adversarial examples in two adversarial datasets (DAmageNet and PGD dataset) and clean examples in ImageNet learned by the classification model, and whether we can use such findings to resist adversarial example attacks. We also seek to determine if we can retrain a discriminator to discriminate whether the input image is an adversarial example, using adversarial training. We then design a number of experiments (e.g., class activation map (CAM) analysis, feature map analysis, feature maps/filters changing, adversarial training, and binary classification model) to help us determine whether the universal adversarial dataset can be successfully used to attack the classification model. This, in turn, contributes to a better understanding of adversarial defenses over pretrained classification model from an interpretation perspective. To the best of our knowledge, this work is one of the earliest works to systematically investigate the interpretation of universal adversarial example attack on image classification models, both visually and quantitatively.

Domain adaptation methods train a model to find similar feature representations between a source and target domain. Recent methods leverage self-supervised learning to discover the analogous representations of the two domains. However, prior self-supervised methods have three significant drawbacks: (1) leveraging pretext tasks that are susceptible to learning low-level representations, (2) aligning the two domains using adversarial loss without considering if the extracted features are low-level representations, (3) the models are not flexible to accommodate various proportions of target labels, i.e., they assume target labels are always available. This paper presents a Generic Domain Adaptation Network (GDAN) to address these issues. First, we introduce a criterion based on instance discrimination to select appropriate pretext tasks to learn high-level domain invariant representations. Then, we propose a semantic neighbor cluster to align the two domain features. The semantic neighbor cluster implements a clustering technique in a feature embedding space to form clusters according to high-level semantic similarities. Finally, we present a weighted target loss function to balance the model weights according to the target labels. This loss function makes GDAN flexible for semi-supervised scenarios, i.e., partly labeled target data. We evaluate the proposed methods on four domain adaptation benchmark datasets. The experiment findings show that the proposed methods align the two domains well and achieve competitive results.

Medical practitioners generally rely on multimodal brain images, for example based on the information from the axial, coronal, and sagittal views, to inform brain tumor diagnosis. Hence, to further utilize the 3D information embedded in such datasets, this paper proposes a multi-view dynamic fusion framework (hereafter, referred to as MVFusFra) to improve the performance of brain tumor segmentation. The proposed framework consists of three key building blocks. First, a multi-view deep neural network architecture, which represents multi learning networks for segmenting the brain tumor from different views and each deep neural network corresponds to multi-modal brain images from one single view. Second, the dynamic decision fusion method, which is mainly used to fuse segmentation results from multi-views into an integrated method. Then, two different fusion methods (i.e., voting and weighted averaging) are used to evaluate the fusing process. Third, the multi-view fusion loss (comprising segmentation loss, transition loss, and decision loss) is proposed to facilitate the training process of multi-view learning networks, so as to ensure consistency in appearance and space, for both fusing segmentation results and the training of the learning network. We evaluate the performance of MVFusFra on the BRATS 2015 and BRATS 2018 datasets. Findings from the evaluations suggest that fusion results from multi-views achieve better performance than segmentation results from the single view, and also implying effectiveness of the proposed multi-view fusion loss. A comparative summary also shows that MVFusFra achieves better segmentation performance, in terms of efficiency, in comparison to other competing approaches.

Resisting the adversarial sample attack on encrypted traffic is a challenging task in the Intelligent Transportation System. This paper focuses on the classification, adversarial samples attack and defense method for the encrypted traffic. To be more specific, the one-dimensional encrypted traffic data is firstly translated into the two-dimensional images for further utilization. Then different classification networks based on the deep learning algorithm are adopted to classify the encrypted traffic data. Moreover, various adversarial sample generation methods are employed to generate the adversarial sample to implement the attacking process on the classification network. Furthermore, the passive and active defense method are proposed to resist the adversarial sample attack: 1) the passive defense is used to denoise the perturbation in the adversarial sample and to restore to the original image; and 2) the active defense is used to resist the adversarial sample attack by leveraging the adversarial training method, which can improve the robustness of the classification network. We conduct the extensive experiments on the ISCXVPN2016 dataset to evaluate the effectiveness of classification, adversarial sample attacking and defending.

The need for medical image encryption is increasingly pronounced, for example, to safeguard the privacy of the patients' medical imaging data. In this article, a novel deep learning-based key generation network (DeepKeyGen) is proposed as a stream cipher generator to generate the private key, which can then be used for encrypting and decrypting of medical images. In DeepKeyGen, the generative adversarial network (GAN) is adopted as the learning network to generate the private key. Furthermore, the transformation domain (that represents the "style" of the private key to be generated) is designed to guide the learning network to realize the private key generation process. The goal of DeepKeyGen is to learn the mapping relationship of how to transfer the initial image to the private key. We evaluate DeepKeyGen using three data sets, namely, the Montgomery County chest X-ray data set, the Ultrasonic Brachial Plexus data set, and the BraTS18 data set. The evaluation findings and security analysis show that the proposed key generation network can achieve a high-level security in generating the private key.

Internet of Medical Things (IoMT) can connect many medical imaging equipment to the medical information network to facilitate the process of diagnosing and treating doctors. As medical image contains sensitive information, it is of importance yet very challenging to safeguard the privacy or security of the patient. In this work, a deep-learning-based image encryption and decryption network (DeepEDN) is proposed to fulfill the process of encrypting and decrypting the medical image. Specifically, in DeepEDN, the cycle-generative adversarial network (Cycle-GAN) is employed as the main learning network to transfer the medical image from its original domain into the target domain. The target domain is regarded as “hidden factors” to guide the learning model for realizing the encryption. The encrypted image is restored to the original (plaintext) image through a reconstruction network to achieve image decryption. In order to facilitate the data mining directly from the privacy-protected environment, a region of interest (ROI)-mining network is proposed to extract the interesting object from the encrypted image. The proposed DeepEDN is evaluated on the chest X-ray data set. Extensive experimental results and security analysis show that the proposed method can achieve a high level of security with a good performance in efficiency.

The identification of discriminative features of the Alzheimer's disease contributes to the diagnostic accuracy. Recently, the combination of different types of features has been actively used in the area of the AD classification. In this paper, we proposed a novel classification framework to jointly select features, which are extracted from the VBM analysis and texture analysis to distinguish between the AD and the NC. Furthermore, in order to capture robust discriminative features, we improve the feature subset selection by combining the SVM-RFE and covariance to take into account the relationship among features. In order to evaluate the proposed method, we have performed evaluations on the MRI acquiring from the ADNI database. Our experimental results showed the feature combination has better performance than the either morphometric features and texture features. Also, we demonstrated our method is better than the one without feature selection, PCA or others.