In this paper, we propose an unsupervised learning approach for the task of infrared and visible image fusion. This approach is called a distinctive-feature guided and hierarchical channel enhanced network-based infrared and visible image Fusion (DFG-HCEN). Instead of using complex fusion rules, DFG-HCEN uses multi-level fusion to achieve fusion results, effectively avoiding information loss during feature extraction. To improve the fusion effect, we designed a distinctive-feature guided module that strengthens the relationship between modules. Moreover, the proposed hierarchical channel enhanced and distinctive-Feature guided module aims to facilitate the fusion framework in efficiently integrating the multilevel complementary features of the source pictures. In addition, we incorporate a hybrid loss method for unsupervised training of the provided DFG-HCEN. The fidelity loss is used to constrain the pixel similarity between the fused result and source images. The application of luminance regularization loss has been shown to be an efficient method for addressing the problem of luminance degradation in fused images. We conducted extensive experiments, including visual examination and quantitative analysis , comparing DFG-HCEN with thirteen other state-of-the-art fusion techniques. The results demonstrate the superiority of DFG-HCEN. Moreover, the extended object detection experiments validate the ability of DFG-HCEN to fully support downstream tasks.

Pansharpening aims at merging the spectral information from a low-resolution multispectral (LRMS) image with the spatial details from a panchromatic (PAN) image to produce a high-resolution multispectral (HRMS) image. Regrettably, existing techniques tend to concentrate on utilizing spectral and spatial information at a single resolution to reconstruct HRMS images, which leads to a deficiency in fully exploiting the semantic information from different resolution levels. In consideration of the aforementioned issues, we proposed a deep multiresolution representation framework for pansharpening, termed DMR-Pan. With the idea of maintaining high-resolution (HR) and low-resolution (LR) representations, we proposed an effective strategy for the extraction of multiresolution semantics, where a PAN branch and an LRMS branch operate in parallel to not only retain HR spatial details and spectral information but also extract multilevel semantics from different resolutions. Through cross-modality and cross-resolution guidance mechanisms, the extracted multiresolution semantics are aggregated with minimal information loss. Finally, a novel query fusion mechanism is introduced to capture the latent interdependency between dual modalities with cross-modality channel-group attention (CCGA), thereby maximizing complementary semantics and significantly improving the fusion ability of the framework. Rigorous experimentation conducted on multiple datasets illustrates that our DMR-Pan surpasses comparable techniques both in qualitative and quantitative assessments.

Abstract Infrared and visible images come from different sensors, and they have their advantages and disadvantages. In order to make the fused images contain as much salience information as possible, a practical fusion method, termed EDAfuse, is proposed in this paper. In EDAfuse, the authors introduce an encoder–decoder with the atrous spatial pyramid network for infrared and visible image fusion. The authors use the encoding network which includes three convolutional neural network (CNN) layers to extract deep features from input images. Then the proposed atrous spatial pyramid model is utilized to get five different scale features. The same scale features from the two original images are fused by our fusion strategy with the attention model and information quantity model. Finally, the decoding network is utilized to reconstruct the fused image. In the training process, the authors introduce a loss function with saliency loss to improve the ability of the model for extracting salient features from original images. In the experiment process, the authors use the average values of seven metrics for 21 fused images to evaluate the proposed method and the other seven existing methods. The results show that our method has four best values and three second‐best values. The subjective assessment also demonstrates that the proposed method outperforms the state‐of‐the‐art fusion methods.

Abstract In this paper, a novel conditional focus probability learning model, termed MCNN, is proposed for multi‐focus image fusion (MFIF). Given a pair of source images, their conditional focus probabilities can be generated by using the well‐trained MCNN, which is further converted into the binary focus masks to directly produce an all‐focus image with no postprocessing. To this end, a fully convolutional encoder is designed with two mutually coupled Siamese branches in MCNN, which include a coupling block that bridge between the two branches to provide conditional information to each other, at different layers, such that the encoder can more strongly extract conditional focus features and further encourage the decoder pixel‐wisely to give more robust conditional focus probabilities. Moreover, a hybrid loss is designed with a structural sparse fidelity loss and a structural similarity loss to force the network to learn more accurate conditional focus probabilities. Particularly, a convolutional norm with good structural group sparse is proposed, to construct the structural sparse fidelity loss. Simulation results substantiate the superiority of our MCNN over other state‐of‐the‐art, in terms of both visual perception and quantitative evaluation.

Abstract In the non‐homogeneous texture synthesis task, the overall visual characteristics should be consistent when extending the local patterns of the exemplar. The existing methods mainly focus on the local visual features of patterns but ignore the relative position features that are important for non‐homogeneous texture synthesis. Although these methods have achieved success on homogeneous textures, they cannot perform well on non‐homogeneous textures. Thus, it is desirable to model the dependence between pixels to improve the synthesis performance. To ensure synthesis results from both the local detail structure and the overall structure, this paper proposes a non‐homogeneous texture extended synthesis model (GAGCN) combining the generate adversarial network (GAN) and the graph convolutional network (GCN). The GAN learns the internal distribution of image patches, which makes the synthetic image have rich local details. The GCN learns the latent dependence between pixels according to the statistical characteristics of the image. Based on this, a novel graph similarity loss is proposed. This loss describes the latent spatial differences between the sample image and the generated image, which helps the model to better capture global features. Experiments show that our method outperforms existing methods on non‐homogeneous textures.

Medical image fusion technology is an essential component of computer-aided diagnosis, which aims to extract useful cross-modality cues from raw signals to generate high-quality fused images. Many advanced methods focus on designing fusion rules, but there is still room for improvement in cross-modal information extraction. To this end, we propose a novel encoder-decoder architecture with three technical novelties. First, we divide the medical images into two attributes, namely pixel intensity distribution attributes and texture attributes, and thus design two self-reconstruction tasks to mine as many specific features as possible. Second, we propose a hybrid network combining a CNN and a transformer module to model both long-range and short-range dependencies. Moreover, we construct a self-adaptive weight fusion rule that automatically measures salient features. Extensive experiments on a public medical image dataset and other multimodal datasets show that the proposed method achieves satisfactory performance.

With the development of image style transfer technologies, portrait style transfer has attracted growing attention in this research community. In this article, we present an asymmetric double-stream generative adversarial network (ADS-GAN) to solve the problems that caused by cartoonization and other style transfer techniques when they are applied to portrait photos, such as facial deformation, contours missing, and stiff lines. By observing the characteristics between source and target images, we propose an edge contour retention (ECR) regularized loss to constrain the local and global contours of generated portrait images to avoid the portrait deformation. In addition, a content-style feature fusion module is introduced for further learning of the target image style, which uses a style attention mechanism to integrate features and embeds style features into content features of portrait photos according to the attention weights. Finally, a guided filter is introduced in content encoder to smooth the textures and specific details of source image, thereby eliminating its negative impact on style transfer. We conducted overall unified optimization training on all components and got an ADS-GAN for unpaired artistic portrait style transfer. Qualitative comparisons and quantitative analyses demonstrate that the proposed method generates superior results than benchmark work in preserving the overall structure and contours of portrait; ablation and parameter study demonstrate the effectiveness of each component in our framework.

Abstract The work on grayscale image colourisation has been significantly improved. Currently, learning‐based methods have achieved some great colourisation effects, but existing colour edge bleeding, especially when colourful cartoon characters. In this paper, we focus on the colourisation of cartoon characters from a series in an adversarial environment with a line art network, whose name is LineGAN . LineGAN learns the corresponding colour mapping from datasets, improving the accuracy of image colourisation. Our methods limit the colour boundary overflow by adding a line art frame in the generator. Extensive experiment results on cartoon image colourisation tasks demonstrate that the proposed method can achieve effective results.

Abstract This paper introduce an efficient object detection network named Trident‐You Only Look Once (YOLO), which is designed for mobile devices with limited computing power. The new architecture is improved based on YOLO v4‐tiny. The authors redesign the network structure and propose a trident feature pyramid network (Trident‐FPN), which can improve the precision and recall of lightweight object detection. Specifically, Trident‐FPN increases the computational complexity by only a small amount of floating point operations per second (FLOPs) and obtains a multi‐scale feature map of the model, which significantly lightweight object detection performance. To enlarge the receptive field of the network with the fewest FLOPs, this paper redesign the receptive field block (RFB) and spatial pyramid pooling (SPP) layer and propose tinier cross‐stage partial RFBs and smaller cross‐stage partial SPPs. This paper present extensive experiments, and Trident‐YOLO shows strong performance compared to that of other popular models on the PASCAL VOC and MS COCO. On the MS COCO and PASCAL VOC 2007 test sets, the mean average precision (mAP) of Trident‐YOLO improved by 4.5% and 5.0%, respectively. Trident‐YOLO also reduce the network size by more than 54.4% compared to YOLO v4‐tiny. With a 23.7% FLOP reduction, the FPS is improved by 1.9 on an Nvidia Jetson Xavier NX.

Abstract Most existing deep learning‐based multi‐modal medical image fusion (MMIF) methods utilize single‐branch feature extraction strategies to achieve good fusion performance. However, for MMIF tasks, it is thought that this structure cuts off the internal connections between source images, resulting in information redundancy and degradation of fusion performance. To this end, this paper proposes a novel unsupervised network, termed CEFusion. Different from existing architecture, a cross‐encoder is designed by exploiting the complementary properties between the original image to refine source features through feature interaction and reuse. Furthermore, to force the network to learn complementary information between source images and generate the fused image with high contrast and rich textures, a hybrid loss is proposed consisting of weighted fidelity and gradient losses. Specifically, the weighted fidelity loss can not only force the fusion results to approximate the source images but also effectively preserve the luminance information of the source image through weight estimation, while the gradient loss preserves the texture information of the source image. Experimental results demonstrate the superiority of the method over the state‐of‐the‐art in terms of subjective visual effect and quantitative metrics in various datasets.

A single infrared image or visible image for the same scene is usually insufficient to simultaneously reveal the infrared objects and the scene details. Thus, image fusion techniques play an important role in producing a single image from the images captured by infrared and visible sensors. In this paper, we propose a novel total variation (TV)-based fusion for infrared and visible images. In our model, a weighted fidelity term is employed to fuse both the infrared objects in the infrared image and the salient scenes in the visible image. To this end, a weight estimation method is developed based on the global luminance contrast-based saliency. Also, to overcome the over-fitting, two constraints are further introduced to merge more details from the visible image and prevent the luminance degradation for the fused result, respectively. Moreover, joint norms are exploited to produce a better result.${{\boldsymbol{l}}_{2,1,{\boldsymbol{rc}}}}$provides the structural group sparseness for the fidelity term, whereas${{\boldsymbol{l}}_{1/2}}$presents the better gradient sparse for the detail preserving term and${{\boldsymbol{l}}_2}$is utilized for the luminance degradation preventing term. Experimental results indicate that the proposed method can give state-of-the-art performances both in visual perception and quantitative scores than other methods.

With the booming of deep learning, massive attention has been paid to developing neural models for multilabel text categorization (MLTC). Most of the works concentrate on disclosing word-label relationship, while less attention is taken in exploiting global clues, particularly with the relationship of document-label. To address this limitation, we propose an effective collaborative representation learning (CRL) model in this article. CRL consists of a factorization component for generating shallow representations of documents and a neural component for deep text-encoding and classification. We have developed strategies for jointly training those two components, including an alternating-least-squares-based approach for factorizing the pointwise mutual information (PMI) matrix of label-document and multitask learning (MTL) strategy for the neural component. According to the experimental results on six data sets, CRL can explicitly take advantage of the relationship of document-label and achieve competitive classification performance in comparison with some state-of-the-art deep methods.

Abstract Due to different lighting environments and equipment limitations, low‐light images have high noise, low contrast and unobvious colours. The main purpose of low‐light image enhancement is to preserve the details and suppress noise as much as possible while improving the contrast of the image. Here, different networks are first combined to construct a multi‐branch module for features extraction, and use the module and Retinex theory to extract the reflection map of the image. Then an attention mechanism is introduced into the multi‐branch construction to balance the feature weight of each branch, and get the final result by the reconstruction module. The Retinex theory is used to calculate the L 1 loss and the gradient loss for the intermediate feature map of the entire model to train our framework. The entire process is completed in an end‐to‐end‐way, which avoids the hand‐crafted reconstruction rules and reduces the workload. What's more, a large number of experiments demonstrate that the proposed framework performs better results than state‐of‐the‐art algorithms in both quantitative and qualitative evaluations of image enhancement.

Multimodal medical image fusion (MMIF) is to merge multiple images for better imaging quality with preserving different specific features, which could be more informative for efficient clinical diagnosis. In this paper, a novel fusion framework is proposed for multimodal medical images based on multi-source information exchange encoding (MIEE) by using Pulse Coupled Neural Network (PCNN). We construct an MIEE model by using two types of PCNN, such that the information of an image can be exchanged and encoded to another image. Then the fusion contributions for each pixel are estimated qualitatively according to a logical comparison of exchanged information. Further, the exchanged information is nonlinearly transformed using an exponential function with a functional parameter. Finally, quantitative fusion contributions are produced through a reverse-proportional operator to the exchanged information. Also, particle swarm optimization-based derivative-free optimization and a total vibration-based derivative optimization are used to optimize the PCNN and functional transform parameters, respectively. Experiments demonstrate that our method gives the best results than other state-of-the-art fusion approaches.

A virtual try-on network has gradually become a popular topic in recent years. It aims to transfer images of in-shop clothes onto the image of a target person. Owing to the diversity of clothing attributes, developing an image-based virtual try-on network is a complicated task for computers to perform and requires significant effort. Existing methods are unsatisfactory as they cannot preserve the characteristics of the clothes or the target person's identity well, thereby affecting the perception of the generated images; therefore, further research is required. To address this problem, we propose a novel try-on method that combines attribute transformation and local rendering. First, we employ pixel-level semantic segmentation to identify the try-on area and provide implementation conditions for local rendering. Second, we construct a learnable attribute transformation module to complete the try-on task for different attributes. Third, we use a learnable clothing warping module to fit the pose and figure of the target person well and establish a novel loss function, called modified style loss (M-SL), to handle clothes with rich details. Finally, we adopt a local rendering strategy, using which only renders the clothing area to ensure that the details of the non-target area are not lost. Extensive experiments are performed to test our method. The results demonstrate that our method outperforms other state-of-the-art methods.

This study proposes a fusion method for high dynamic range images based on gradient information transformation. In the proposed work, the authors first measure the three exposure weights of the source images, namely, local contrast, luminance and spatial structure. Then, the exposure weights are merged through a multi‐scale Laplacian pyramid scheme. For the weight maps measurement, the dense scale‐invariant feature transform method is used to calculate the local contrast around each pixel location, rather than a single pixel. The image luminance levels are computed in the gradient domain to get more visual information and the authors leverage the dictionary learning to effectively extract the luminance of images. Additionally, to better preserve the spatial structure of the source images, the just‐noticeable‐distortion technique is employed. By comparing the experimental results both subjectively and objectively, it is evident that the proposed method represents an improvement over some exciting methods.

Nonphotorealistic rendering (NPR) techniques are used to transform real-world images into high-quality aesthetic styles automatically. NPR mainly focuses on transfer hand-painted styles to other content images, and simulates pencil drawing, watercolor painting, sketch painting, Chinese monochromes, calligraphy and, so on. However, digital simulation of Chinese embroidery style has not attracted researcher’s much attention. This study proposes an embroidery style transfer method from a 2D image on the basis of a convolutional neural network (CNN) and evaluates the relevant rendering features. The primary novelty of the rendering technique is that the strokes and needle textures are produced by the CNN and the results can display embroidery styles. The proposed method can not only embody delicate strokes and needle textures but also realize stereoscopic effects to achieve real embroidery features. First, using conditional random fields (CRF), the algorithm segments the target content and the embroidery style images through a semantic segmentation network. Then, the binary mask image is generated to guide the embroidery style transfer for different regions. Next, CNN is used to extract the strokes and texture features from the real embroidery images, and transfer these features to the content images. Finally, the simulating image is generated to show the features of the real embroidery styles. To demonstrate the performance of the proposed method, the simulations are compared with real embroidery artwork and other methods. In addition, the quality evaluation method is used to evaluate the quality of the results. In all the cases, the proposed method is found to achieve needle visual quality of the embroidery styles, thereby laying a foundation for the research and preservation of embroidery works.

We study the problem of multi-focus image fusion, where the key challenge is detecting the focused regions accurately among multiple partially focused source images. Inspired by the conditional generative adversarial network (cGAN) to image-to-image task, we propose a novel FuseGAN to fulfill the images-to-image for multi-focus image fusion. To satisfy the requirement of dual input-to-one output, the encoder of the generator in FuseGAN is designed as a Siamese network. The least square GAN objective is employed to enhance the training stability of FuseGAN, resulting in an accurate confidence map for focus region detection. Also, we exploit the convolutional conditional random fields technique on the confidence map to reach a refined final decision map for better focus region detection. Moreover, due to the lack of a large-scale standard dataset, we synthesize a large enough multi-focus image dataset based on a public natural image dataset PASCAL VOC 2012, where we utilize a normalized disk point spread function to simulate the defocus and separate the background and foreground in the synthesis for each image. We conduct extensive experiments on two public datasets to verify the effectiveness of the proposed method. Results demonstrate that the proposed method presents accurate decision maps for focus regions in multi-focus images, such that the fused images are superior to 11 recent state-of-the-art algorithms, not only in visual perception, but also in quantitative analysis in terms of five metrics.

Manifold learning and classifiers based on sparse representation are widely used in pattern recognition. Most of the conventional manifold learning methods are subjected to the choice of parameters. In this paper, we present a Regularized Locality Projection based on Sparsity Discriminant Analysis (RLPSD) method for Feature Extraction (FE) to understand the high-dimensional data such as face images. In RLPSD, firstly, we show the sparse representation of training samples by collaborative representation-based classification (CRC). Secondly, the idea of part optimization based on sparse representation is used to ensure the within-class compactness which combines with the labels of measurements and the weights of sparse presentation can be as small as possible. Finally, whole optimization can be directly obtained without the iteration of local optimization. Meanwhile, the separability information of between-class can be well discriminated by scatter matrix which is similar to Fisher linear discriminant analysis (LDA). The great recognition performance of the proposed method is verified by comparing with the popular algorithms on Yale, ORL, AR and Extended YaleB face databases and Oxford 102 flowers dataset.

As the optical lenses for cameras always have limited depth of field, the captured images with the same scene are not all in focus. Multifocus image fusion is an efficient technology that can synthesize an all-in-focus image using several partially focused images. Previous methods have accomplished the fusion task in spatial or transform domains. However, fusion rules are always a problem in most methods. In this letter, from the aspect of focus region detection, we propose a novel multifocus image fusion method based on a fully convolutional network (FCN) learned from synthesized multifocus images. The primary novelty of this method is that the pixel-wise focus regions are detected through a learning FCN, and the entire image, not just the image patches, are exploited to train the FCN. First, we synthesize 4500 pairs of multifocus images by repeatedly using a gaussian filter for each image from PASCAL VOC 2012, to train the FCN. After that, a pair of source images is fed into the trained FCN, and two score maps indicating the focus property are generated. Next, an inversed score map is averaged with another score map to produce an aggregative score map, which take full advantage of focus probabilities in two score maps. We implement the fully connected conditional random field (CRF) on the aggregative score map to accomplish and refine a binary decision map for the fusion task. Finally, we exploit the weighted strategy based on the refined decision map to produce the fused image. To demonstrate the performance of the proposed method, we compare its fused results with several start-of-the-art methods not only on a gray data set but also on a color data set. Experimental results show that the proposed method can achieve superior fusion performance in both human visual quality and objective assessment.