ABSTRACT Multi‐modal diagnosis of ophthalmic disease is becoming increasingly important because combining multi‐modal data allows for more accurate diagnosis. Color fundus photograph (CFP) and optical coherence tomography (OCT) are commonly used as two non‐invasive modalities for ophthalmic examination. However, the diagnosis of each modality is not entirely accurate. Compounding the challenge is the difficulty in acquiring multi‐modal data, with existing datasets frequently lacking paired multi‐modal data. To solve these problems, we propose multi‐modal distribution fusion diagnostic algorithm and cross‐modal generation algorithm. The multi‐modal distribution fusion diagnostic algorithm first calculates the mean and variance separately for each modality, and then generates multi‐modal diagnostic results in a distribution fusion manner. In order to generate the absent modality (mainly OCT data), three sub‐networks are designed in the cross‐modal generation algorithm: cross‐modal alignment network, conditional deformable autoencoder and latent consistency diffusion model (LCDM). Finally, we propose multi‐task collaboration strategy where diagnosis and generation tasks are mutually reinforcing to achieve optimal performance. Experimental results demonstrate that our proposed method yield superior results compared to state‐of‐the‐arts.

Detecting early ischemic lesions (EIL) in computed tomography (CT) images is crucial for reducing diagnostic time and minimizing neuron loss due to oxygen deprivation. This paper introduces DCTP-Net, a dual-branch network for segmenting acute ischemic stroke lesions in CT images, consisting of a segmentation branch and a prompt-aware branch. The segmentation branch uses an encoder-decoder network as the backbone to identify lesions, where the encoder fuses CT image features with prompt features from the prompt-aware branch. To enhance semantic feature extraction and reduce the impact of cerebral structural details, we introduce a cross-collaboration dynamic connection (CCDC) module to link the encoder and decoder. The prompt-aware branch includes a learnable prompt (LP) block to incorporate cerebral prior knowledge, and the prompt-aware encoder (PAE) combines the LP block with multi-level features from the segmentation branch for more precise representation. Additionally, we propose a CLIP-enhance textual prompt (CETP) module that utilizes the CLIP text encoder to generate specialized convolutional parameters for the segmentation head. These parameters are tailored to the unique characteristics of each input image, improving segmentation performance. Qualitative and quantitative studies reveal that DCTP-Net outperforms the current state-of-the-art, IS-Net, with Dice score increases of 3.9% on AISD and 3.8% on ISLES2018, demonstrating its superiority in EIL segmentation.

Abstract The text-to-image (T2I) model based on a single-stage generative adversarial network (GAN) has significantly succeeded in recent years. However, the generation model based on GAN has two disadvantages: the generator does not introduce any image feature manifold structure, which makes it challenging to align the image and text features. Another is the image’s diversity; the text’s abstraction will prevent the model from learning the actual image distribution. This paper proposes a reversed image interaction generative adversarial network (RII-GAN), which consists of four components: text encoder, reversed image interaction network (RIIN), adaptive affine-based generator, and dual-channel feature alignment discriminator (DFAD). RIIN indirectly introduces the actual image distribution into the generation network, thus overcoming the problem that the network lacks the learning of the actual image feature manifold structure and generating the distribution of text-matching images. Each adaptive affine block (AAB) in the proposed affine-based generator can adaptively enhance text information, establishing an updated relation between original independent fusion blocks and the image feature. Moreover, this study designs a DFAD to capture important feature information of images and text in two channels. Such a dual-channel backbone improves semantic consistency by utilizing a particular synchronized bi-modal information extraction structure. We have performed experiments on publicly available datasets to prove the effectiveness of our model.

Few-shot semantic segmentation is a challenging task that aims to segment novel classes in the query images given only a few annotated support samples. Most existing prototype-based approaches extract global or local prototypes by global average pooling (GAP) or clustering to represent all object information. Subsequently, the prototype information is employed as guidance for query image segmentation. However, these frameworks fail to fully mine the object details and ignore information from query images. Consequently, we propose a Dual-Guided Frequency Prototype Network (DGFPNet) to solve these issues. Specifically, to mine the global and local object information, a Frequency Prototype Generation Module (FPGM) is first proposed to extract more comprehensive frequency prototypes by multi-frequency pooling (MFP) in the DCT domain. Then, with the guidance of support and query information, a Dual-Guided Selection Module (DGSM) is presented to produce the query attention mask and select more effective prototypes. Based on the query attention mask and support information, the generalized object information is integrated into the feature with the proposed Feature Generalization Module (FGM). Finally, we propose a Multi-Dimension Feature Enrichment Decoder Module (MDFEDM) to capture multi-dimension object information and tackle hard pixels for refining the final segmentation results. Extensive experiments on PASCAL-5iand COCO-20ishow that our model achieves new state-of-the-art performances. Our code will be released athttps://github.com/ChunLinWen/DGFPNet.

In recent years, learning-based single image dehazing networks have been comprehensively developed. However, performance improvement is limited due to domain shift between trained synthetic hazy images and untrained real-world hazy images. To alleviate this issue, this paper proposes a real-world dehazing targeted training scheme which nearly realizes paired real-world data training. As a result, a Twofold Multi-scale Generative Adversarial Network (TMS-GAN) consisting of a Haze-generation GAN (HgGAN) and a Haze-removal GAN (HrGAN) is designed. HgGAN attributes real haze properties to synthetic images and HrGAN removes haze from both synthetic and generated fake realistic data under supervision. Thus, the proposed method can better adapt to real-world image dehazing using this cooperative training scheme. Meanwhile, several structural advances of TMS-GAN also improve dehazing performance. Specifically, a haze residual map based on atmospheric scattering model is deduced in HgGAN for fake realistic data generation. The dual-branch generator in HrGAN draws attention to detail restoration by one branch along with another color-branch. A plug-and-play Multi-attention Progressive Fusion Module (MAPFM) is proposed and inserted in both HgGAN and HrGAN. MAPFM incorporates multi-attention mechanism to guide multi-scale feature fusion in a progressive manner, in which Adjacency-attention Block (AAB) can capture contributing features of each level and Self-attention Block (SAB) can establish non-local dependency of feature fusion. Experiments on mainstream benchmarks show that the proposed framework is superior especially on real-world hazy images among single image dehazing methods.

In this study, an unsupervised clustering algorithm is proposed to label superpixel density images. Firstly, the authors propose a novel superpixel segmentation algorithm driven by a modified fuzzy C‐means objective function, Kullback–Leibler (KL) divergence, and an entropy term, which generate superpixels with good boundary adherence and intensity homogeneity. In this model, the logarithm of Gaussian distribution as a new distance metric is used to improve the accuracy of boundary pixel classification, the KL divergence is applied to regularise the fuzzy objective function. Based on this model, the generated superpixel intensity images with a highly distinctive background colour from the colour of the target are obtained. Grouping cues generated by superpixels can affect the performance of image clustering greatly. Next, according to the small amount of clustering data generated by the superpixel intensity images, they construct a non‐symmetric mixture model based on a mixture of Gaussian distribution and Cauchy distribution for implementing image clustering. Thus, clustering of colour images is transformed into clustering of these newly generated data. The advantage of this model is its well adaption to different shapes of observed data. Experimental results on publicly available data sets are provided to demonstrate the effectiveness of the proposed algorithm.

Full convolution network (FCN) is widely used in medical image segmentation and its performance is better than other conventional techniques. This paper proposes a new fusion algorithm that combined the convolutional neural network U-net with a new modified level set method to segment overlapping cervical smear cells. U-net could provide more excellent segmentation results of nuclei and cytoplasm cluster. Then, a modified level set energy function with distance map and a new shape prior term is applied to extract the contour of cervical cells. Owing to this new level set energy function, the segmentation of every individual cell performed well, especially in overlapping area of cells. The evaluation of results also proves the improvement of our fusion algorithm.

Deep convolutional neural networks (DCNN) are applied widely in image recognition and segmentation. In this paper, a novel algorithm (U-SMM) which incorporates the convolutional neural network U-net and modified Student's- t mixture model (MSMM) is provided. The proposed framework considers the spatial relationships in segmenting medical images with MSMM and then uses U-net to correct the mistake labels made by unsupervised method. Because a few error-segmented regions may be caused by MSMM, the U-net is then applied to learn the features of these regions. In our method, the purpose of U-net is to assist the MSMM in improving the accuracy of segmentation and acquiring rich details in image segmentation tasks. Finally, the proposed framework is evaluated on real MR images with several related supervised and unsupervised methods, and the experimental results confirm the effectiveness of our approach.

In this paper, a nonsymmetric Student's-t distribution model is proposed for magnetic resonance (MR) image segmentation based on Markov random field (MRF) and weighted mean template. The presented nonsymmetric Student's-t distribution with longer tails and one more parameter compared to Gaussian distribution is implemented because in real applications, the distribution of data set does not totally follow symmetric distribution. Thus, our method fits much closer to different shapes of observed data. With the help of MRF and weighted mean template, the spatial information is also taken into consideration in MR image segmentation. Then, the expectation-maximization (EM) algorithm is introduced to solve the problem of parameter learning. The accuracy and effectiveness of the proposed method is quantitatively assessed in both simulated and clinical MR images.

The rotation invariance of the classical disc-based moments, such as Zernike moments (ZMs), pseudo-ZMs (PZMs), and orthogonal Fourier-Mellin moments (OFMMs), makes them attractive as descriptors for the purpose of recognition tasks. However, less work has been performed for the generalization of these moment functions. In this paper, four general forms are developed to obtain a class of disc-based generalized radial polynomials that are orthogonal over the unit circle. These radial polynomials are scaled to ensure numerical stability, and some useful properties are discussed for potential applications they could be used in. Then, these scaled radial polynomials are used as kernel functions to construct a series of unit discbased generalized orthogonal moments (DGMs). The variation of parameters in DGMs can form various types of orthogonal moments: 1) generalized ZMs; 2) generalized PZMs; and 3) generalized OFMMs. The classical ZMs, PZMs, and OFMMs correspond to a special case of these three generalized moments for which the free parameter α = 0. Each member of this family will share some excellent properties for image representation and recognition tasks, such as orthogonality and rotation invariance. In addition, we have also developed two algorithms, the so-called m-recursive and n-recursive methods for the computation of these proposed radial polynomials to improve the numerical stability. Experimental results show that the proposed methods are superior to the classical disc-based moments in terms of image representation capability and classification accuracy.

It is well known that the implementation of Tchebichef transforms (TT) does not involve any numerical approximation, since the basis set is orthogonal in the discrete domain of the image coordinate space. Therefore, it can be effectively used in the analysis of images processing area. However, the direct computation of discrete orthogonal TT is very expensive. In this study, we introduce a new framework for TT based on the discrete Tchebichef polynomials and develop a new block-based directional Tchebichef transforms (BDTT) to compensate for the defects of the direct computation TT in the application of image description. Moreover, this new algorithm is integrated into the compressed sensing (CS). By choosing the best directional mode of TT, the proposed methods simultaneous realizes sampling, compression of image, and better image description. Several numerical experiments demonstrate that the proposed algorithm has better performance than the conventional methods.

This paper presents three new separable 2-D discrete orthogonal moments. The kernel functions of the proposed Meixner Krawtchouk moments (MKM), Tchebichef-Charlier moments (TCM), and Meixner-Hahn moments (MHM) are mutually orthogonal and separable. Unlike the traditional 2-D discrete orthogonal moments, in the proposed separable 2-D discrete orthogonal moments, the kernel functions can be expressed as two separable terms by producing two different classical orthogonal polynomials of a variable. Specifically, the tense product of Meixner and Krawtchouk polynomials can be used to generate kernel functions for 2-D discrete orthogonal MKM. The global extraction capabilities of proposed moments are described by analyzing the reconstructed image's accuracy. The experimental results show that these proposed moments have better image description capabilities.

This paper presents two new sets of nonseparable discrete orthogonal Charlier and Meixner moments describing the images with noise and that are noise-free. The basis functions used by the proposed nonseparable moments are bivariate Charlier or Meixner polynomials introduced by Tratnik et al. This study discusses the computational aspects of discrete orthogonal Charlier and Meixner polynomials, including the recurrence relations with respect to variable x and order n. The purpose is to avoid large variation in the dynamic range of polynomial values for higher order moments. The implementation of nonseparable Charlier and Meixner moments does not involve any numerical approximation, since the basis function of the proposed moments is orthogonal in the image coordinate space. The performances of Charlier and Meixner moments in describing images were investigated in terms of the image reconstruction error, and the results of the experiments on the noise sensitivity are given.

Geometric distortions are generally simple and effective attacks for many watermarking methods. They can make detection and extraction of the embedded watermark difficult or even impossible by destroying the synchronization between the watermark reader and the embedded watermark. In this paper, we propose a new watermarking approach which allows watermark detection and extraction under affine transformation attacks. The novelty of our approach stands on a set of affine invariants we derived from Legendre moments. Watermark embedding and detection are directly performed on this set of invariants. We also show how these moments can be exploited for estimating the geometric distortion parameters in order to permit watermark extraction. Experimental results show that the proposed watermarking scheme is robust to a wide range of attacks: geometric distortion, filtering, compression, and additive noise.

In this paper, we proposed a set of translation and rotation invariants extracted from Tchebichef moments. A set of Tchebichef moment invariants is derived from the relationship between Tchebichef moments of the original image and those of the transformed image. These invariants are then used for symmetric image recognition. Contrarily to the methods based on the complex moments in symmetric image analysis, our method does not need the pre-selection of moment values. Experimental results show that the proposed method achieves better performance compared to the existing methods.

In real imaging system, the observed image is usually corrupted by blurring, spatial degradations. The classical recognition methods in degraded image analysis are to obtain blur invariants based on geometric moments or complex moments. In this paper, we introduce blur invariants based on Zernike moments which are orthogonal over a unit circle. Both the expression of Zernike moments of blurred image and the set of blur invariants based on Zernike moments are presented and proved mathematically. Compared with the pattern classification results of complex moments, the experimental results of Zernike moment demonstrate that the proposed method performs well in object and pattern recognition.

Due to the inherent ill-posedness of PET reconstruction, the reconstructed images usually have noise and edge artifacts, and regularization techniques are needed to produce reasonable results. We propose a new minimum cross-entropy (MXE) image reconstruction method for PET based on the total variation (TV) norm constraint. The use of TV is due to the fact that it can effectively reduce the noise in 2D images while preserving edges. In addition, a multiple level set method was incorporated into image reconstruction to identify the shape of the radioactive objects. It is important for some special applications where the shape of tumors should be identified. The initial emission rates used by the multiphase level set method were estimated using a discrete reconstruction method. Experimental results show that the proposed method is more effective.

This paper presents an efficient method for automatic segmentation of blood vessels in retinal images. A set of directional basis filters based on dyadic wavelet transform is designed to enhance blood vessels. It attempts to utilize the linear combination of the wavelet transforms to fix on the blood vessels directional information in retinal images. The directional maps are processed by thresholding scheme in order to segment blood vessels from the background. The proposed thresholding approach evaluates 2-D entropies based on the gray level-gradient co-occurrence matrix. The 2-D thresholding vector that maximizes the edge class entropies is selected. The thresholding method utilizes the gray level and gradient information in the enhanced image. The new method promises the simpleness and flexibility in many image enhancement and segmentation applications.

A novel automated method for the segmentation of blood vessels in retinal images based upon enhancement and maximum entropy thresholding is proposed. Blood vessels usually have poor local contrast. Before thresholding fundus images, several matched filters are employed to enhance the contrast of blood vessels. The matched-filter-response (MFR) image is processed by a thresholding scheme in order to extract blood vessels from the background. Then, the proposed thresholding approach evaluates two-dimensional entropies based on the gray level-gradient cooccurrence matrix. The 2D threshold vector that maximizes the edge class entropies is selected. This thresholding method utilizes the information of gray level and gradient in the MFR image. It is found that the proposed algorithm works well in normal or abnormal retinal images.