Abstract In the environmental security monitoring of smart cities, the infrared and visible image fusion method deployed on intelligent systems based on cloud and fog computing plays a vital role in providing enhanced images for target detection systems. However, the fusion quality can be significantly influenced by the illumination of the monitoring scenario in visible images. Therefore, conventional methods typically suffer a severe performance drop under the condition of insufficient illumination. To tackle this issue, we propose an illumination enhancement and target‐aware fusion method‐based on artificial intelligence, which breaks the boundaries between the task of illumination enhancement and image fusion and provide a fusion result with better visual perception in nighttime scene. Specifically, we use a light‐weight contrast enhancement module restore the brightness of the visible image. Moreover, a Swin Transformer‐based backbone network is utilized to facilitate information exchange between the source images and enhance the capabilities of target awareness. Finally, the fused images are reconstructed by the contrast‐texture retention module and reconstructor. The extensive experiments indicate that the proposed approach achieves improved performance both in human perception and quantitative analysis compared with the state‐of‐the‐art methods.

Abstract In the global monitoring of smart cities, the demands of global object detection systems based on cloud and fog computing in intelligent systems can be satisfied by photographs with globally recognized properties. Nevertheless, conventional techniques are constrained by the imaging depth of field and can produce artefacts or indistinct borders, which can be disastrous for accurately detecting the object. In light of this, this paper proposes an artificial intelligence‐based gradient learning network that gathers and enhances domain information at different sizes in order to produce globally focused fusion results. Gradient features, which provide a lot of boundary information, can eliminate the problem of border artefacts and blur in multi‐focus fusion. The multiple‐receptive module (MRM) facilitates effective information sharing and enables the capture of object properties at different scales. In addition, with the assistance of the global enhancement module (GEM), the network can effectively combine the scale features and gradient data from various receptive fields and reinforce the features to provide precise decision maps. Numerous experiments have demonstrated that our approach outperforms the seven most sophisticated algorithms currently in use.

Abstract The infrared and visible fusion technology holds a pivotal position in smart city for cloud and fog computing, particularly in security system. By fusing infrared and visible image information, this technology enhances target identification, tracking and monitoring precision, bolstering overall system security. However, existing deep learning‐based methods rely heavily on convolutional operations, which excel at extracting local features but have limited receptive fields, hampering global information capture. To overcome this difficulty, we introduce GRDATFusion, a novel end‐to‐end network comprising three key modules: transformer, gradient residual dense and attention residual. The gradient residual dense module extracts local complementary features, leveraging a dense‐shaped network to retain potentially lost information. The attention residual module focuses on crucial input image details, while the transformer module captures global information and models long‐range dependencies. Experiments on public datasets show that GRDATFusion outperforms state‐of‐the‐art algorithms in qualitative and quantitative assessments. Ablation studies validate our approach's advantages, and efficiency comparisons demonstrate its computational efficiency. Therefore, our method makes the security systems in smart city with shorter delay and satisfies the real‐time requirement.

ABSTRACT The super‐resolution (SR) task aims to reconstruct high‐resolution content from low‐resolution images, and its core challenge is to solve the problem of ill‐posedness while balancing the fidelity and perceptual quality of the generated images. Although flow‐based SR models have made significant progress in modelling high‐resolution image distributions, their generated images still have shortcomings in terms of detail representation. To address the issues above, this paper proposes an optimised method that incorporates a conditionally learnt prior (latent module). Specifically, a Dual‐Dimensional Feature Co‐Enhancement (DFCE) module is developed to perform joint optimisation on the channel and spatial dimensions of the features. The experimental results on the public datasets show that this framework effectively improves the generation quality of images with almost no increase in the amount of computation. Furthermore, this framework can be seamlessly integrated into fixed‐scale and arbitrary‐scale streaming models without the need to modify their pre‐training weights or architecture, which provides efficient and flexible solutions for practical applications.

Transformer-based methods excel in remote sensing image super-resolution (RSISR) due to their ability to capture abundant self-similarity textures and structural information, which are critical for restoring high-resolution details. However, the core component of these methods, the self-attention mechanism, requires substantial computational resources, resulting in a restriction of deployment on resource-constrained remote sensing devices. The redundant feature extraction and insufficient refinement of local features also cause the significant RSISR performance drop in multi-degradation scenarios where RSI often suffers from optical blur and transmission noise caused by adverse imaging environments. To mitigate the above limitations, we present a Lightweight Criss-Cross Swin Transformer (CCST) for Blind Remote Sensing Image Super-Resolution. Concretely, we introduce criss-cross encoding into the self-attention mechanism to establish criss-cross feature correlations of each pixel in an RSI. Along with the shifted window mechanism, repeated criss-cross encoding indirectly builds global feature dependencies. Compared to the conventional self-attention mechanism that spans the entire window, the proposed criss-cross self-attention mechanism drastically reduces computational overhead and feature redundancy while capturing more global information by using large-size windows. Furthermore, to enhance the ability of the RSISR model to extract local features while expanding its receptive field, we incorporate a gated feed-forward tiny network to integrate multi-scale features and introduce gated coefficient learning to adaptively filter redundant counterparts. Experimental results show that our CCST achieves superior performance with lower computational demand compared to state-of-the-art Transformer-based SR algorithms and traditional RSISR methods in multi-degradation scenarios.

Convolutional neural networks (CNNs) have recently achieved outstanding performance for hyperspectral (HS) and multispectral (MS) image fusion. However, CNNs cannot explore the long-range dependence for HS and MS image fusion because of their local receptive fields. To overcome this limitation, a transformer is proposed to leverage the long-range dependence from the network inputs. Because of the ability of long-range modeling, the transformer overcomes the sole CNN on many tasks, whereas its use for HS and MS image fusion is still unexplored. In this article, we propose a spectral-spatial transformer (SST) to show the potentiality of transformers for HS and MS image fusion. We devise first two branches to extract spectral and spatial features in the HS and MS images by SST blocks, which can explore the spectral and spatial long-range dependence, respectively. Afterward, spectral and spatial features are fused feeding the result back to spectral and spatial branches for information interaction. Finally, the high-resolution (HR) HS image is reconstructed by dense links from all the fused features to make full use of them. The experimental analysis demonstrates the high performance of the proposed approach compared with some state-of-the-art (SOTA) methods.

The Euclidean direction search (EDS) algorithm can reduce the complexity by avoiding the matrix inversion operation. However, it may fail to work in impulsive environments. To address this problem, a novel EDS based upon the maximum correntropy criterion (EDS-MCC) algorithm is proposed, which provides computational savings and robustness for combating impulsive noise. Additionally, the EDS-MCC algorithm is analyzed to obtain the theoretical performance by utilizing the energy conservation argument (ECA) and the Taylor expansion method. Simulations are exhibited to show the robustness of the EDS-MCC algorithm and verify the accuracy of the theoretical analysis.

Reference-based super-resolution (RefSR), which aims to introduce an additional high-resolution (HR) reference (Ref) image to improve the reconstruction performance of low-resolution (LR) image, has achieved great success. Existing RefSR methods rely on the texture information of the reference image to compensate for the missing information. However, the differences of scale and orientation are unavoidable when obtaining useful information from the Ref image. In addition, it is difficult to achieve a good match due to the ill-posed between the LR image and Ref image. To address these challenges, we propose a new matching module, named progressive spatial adaptation module (PSAM). PSAM is a progressive alignment model to effectively overcome the ill-pose between the LR image and the Ref image. Further, we propose a spatial correction module (SCM) to correct for scale and orientation. Meanwhile, we introduce a gradient map to further correct the matched features. In addition, we propose a new loss function MC-Loss to ensure the success of correction. Experiments show that the matching method using PSAM to directly replace the existing RefSR is significantly better than the original matching method in terms of both quantitative and qualitative results.

Deep learning (DL)-based multisource information processing plays an essential role in the Internet of Medical Things (IoMT). In this field, medical image fusion integrates scan results from different devices, supporting healthcare systems to make a more informed diagnosis. This study proposes a DL-based network to fuse multimodal medical images. In our method, the lattice unit (LU) is designed to improve the representation capability of the fusion network. Moreover, to acquire hierarchical features from images, the progressive module (PM) treats the network’s shallow and deep layers differently. The shallow layers represent the structure of the source image; the deep layers correspond to details. Different loss functions are utilized for these two kinds of information to retain fused images’ salient structures and functional information. Experiments show that the proposed algorithm performs well on visual quality and objective evaluation, which provides a reliable reference for medical diagnosis. In addition, this method is lightweight and speedy compared to existing algorithms, facilitating further placement in the IoMT’s specific devices.

Deep learning (DL) methods have achieved impressive performance for pansharpening in recent years. However, because of poor generalization, most DL methods achieve unsatisfactory performance for data acquired by sensors not considered during the training phase and decreased performance for samples at full resolution. To solve this issue, we propose a data augmentation framework for pansharpening neural networks. Specifically, we introduce first a random spatial degradation based on anisotropic Gaussian-shaped modulation transfer functions (MTFs) to increase the generalization with respect to different spatial models and sensors. Then, considering that various sensors have different ground sampling distances (GSDs), we randomly rescale the GSD of the training samples to improve the generalization with respect to spatial resolution. Thanks to this module, the generalization to tests from different sensors and samples at full resolution can easily be achieved. Experimental results demonstrate the effectiveness of the proposed approach with better performance when data for training are decoupled with the ones for testing and comparable performance when training and testing are coupled (i.e., data acquired by the same sensor are considered in the two phases). Besides, performance at full resolution for pansharpening neural networks is improved by the proposed approach. The proposed approach has been integrated into existing pansharpening neural networks showing satisfactory performance for widely used sensors, including, GaoFen-1, QuickBird, WorldView-2, WorldView-3, IKONOS, Spot-7, GeoEye, and PHR1A.

Partial discharge (PD) location techniques are a useful tool for condition monitoring of electrical apparatus in power systems. However, the noisy PD measurements may significantly degrade the performance of location algorithms. This article deals with the PD location problem by using adaptive filtering techniques. Heretofore, scarce literature focuses on addressing the PD location based on such method. A novel adaptive algorithm, termed as total least-squares (TLS)-Matérn kernel (TLS-MK), is proposed. Benefiting from the merits of the Matérn kernel, the TLS model can effectively suppress the noise from the direct and reflected waves of the PD source. Meanwhile, the TLS-MK algorithm is used to estimate the time difference, which is used in the PD location. Moreover, the convergence behavior of the TLS-MK algorithm is analyzed. Simulations and experiments show that the proposed algorithm can enhance the location accuracy as compared to state-of-the-art methods for various PD signals.

Recently, deep learning (DL)-based industrial applications have attracted broad attention due to their advanced performance. However, the limited computational resource in portable devices always makes big DL models inapplicable in the industry. DL-based single-image super-resolution also encounters this problem because of its large computations. Besides, most lightweight convolutional-neural-network-based methods utilize features insufficiently, which restricts their capability for industrial reconstruction. To alleviate this problem, we present a progress interaction-learning network (PILN) to refine features at different levels: at the global level, we employ a progressive interaction-learning strategy to integrate hierarchical features in temporal and spatial dimensions; at the mediate level, enhanced interaction-learning units, adopting the enhanced interactive study, significantly boost the reconstruction performance; at the local level, employing pixelwise learning, residual cells are raised to search for an optimal information flow by weight distribution. Extensive experiments demonstrate that the PILN outperforms other state-of-the-art methods.

The partial discharge (PD) classification for electric power equipment based on machine learning algorithms often leads to insufficient generalization ability and low recognition accuracy. To solve the problem, this article develops an improved Wasserstein generative adversarial network with gradient penalty (WGAN-GP) based data augmentation model. The improved WGAN-GP model can generate data samples to supplement the low-data input set in PD source classification. First, an improved WGAN-GP model with conditional generation is trained and various new data samples are generated. Then, the new data samples are utilized to expand the raw dataset. Finally, the expanded dataset is trained to get a new PD classifier. Experimental results demonstrate that the proposed model can generate new high-quality data samples more stably. Moreover, the proposed method can suppress the overfitting risk caused by low data or imbalanced data distributions and the classification accuracy is effectively improved.

Increasingly serious health problems have made the usage of computed tomography surge. Therefore, algorithms for processing CT images are becoming more and more abundant. These algorithms can lessen the harm of cumulative radiation in CT technology for the patient while eliminating the noise of image caused by dose reduction. However, the mainstream CNN-based algorithms are inefficient when dealing with features in broad regions. Inspired by the large receptive field of transformer framework, this paper designs an end-to-end low-dose CT (LDCT) denoising network based on the transformer. The overall network contains a main branch and dual side branches. Specifically, the overlapping-free window-based self-attention transformer block is adopted on the main branch to realize image denoising. On the dual side branches, we propose double enhancement module to enrich edge, texture, and context information of LDCT images. Meanwhile, the receptive field of network is further enlarged after processing, which is helpful for building model's long-range dependencies. The outputs of the side branches are concatenated for enhancing information and generating high-quality CT images. In addition, to better train the network, we introduce a compound loss function including mean squared error (MSE), multi-scale perceptual (MSP), and Sobel-L1 (SL) to make the denoised image closer to the targeted norm-dose CT (NDCT) image. Lastly, we conducted experiments on two clinical datasets including abdomen, head, and chest LDCT images with 25%, 25%, and 10% of the full dose, respectively. The experimental results demonstrated that the proposed DEformer achieved better denoising performance than the existing algorithms.

Convolutional neural networks (CNNs) have achieved impressive performance for hyperspectral (HS) and multispectral (MS) image fusion in recent years. They extract features by local filters, which is limited to explore long-range dependency in input images. However, long-range dependence is an import cue for HS and MS image fusion, as it contributes to exploration of spatial self-similarity and spectral dependence. To take advantage of long-range dependence, we propose a spectral-spatial transformer (SST) for MS and HS image fusion. The experimental results demonstrate the high performance of the proposed approach compared to some state-of-the-art methods.

We propose a novel M-estimate conjugate gradient (CG) algorithm, termed Tukey’s biweight M-estimate CG (TbMCG), for system identification in impulsive noise environments. In particular, the TbMCG algorithm can achieve a faster convergence while retaining a reduced computational complexity as compared to the recursive least-squares (RLS) algorithm. Specifically, the Tukey’s biweight M-estimate incorporates a constraint into the CG filter to tackle impulsive noise environments. Moreover, the convergence behavior of the TbMCG algorithm is analyzed. Simulation results confirm the excellent performance of the proposed TbMCG algorithm for system identification and active noise control applications.

Although the performance of pansharpening has been significantly improved by advanced deep-learning (DL) technologies in recent years, most DL-based methods fail to process multispectral (MS) images with arbitrary numbers of bands by a single model. Consequently, it is inevitable to train separate models for MS images with different numbers of bands, which is time- and storage-consuming as well as inefficient in practice. To tackle the above problem, we propose a bidirectional recurrent pansharpening network (named ArbRPN) for MS images with arbitrary numbers of bands. Our ArbRPN can dynamically reconstruct high-resolution (HR) MS images with different numbers of bands by adaptively changing the number of recurrence to the number of bands of the low-resolution (LR) MS images. Leveraging on the ability of the ArbRPN to process MS images with any number of bands, one can even customize the bands to be pansharpened. Moreover, to achieve superior performance, spectral discrepancy and dependence are considered in the ArbRPN. Details from the panchromatic (PAN) image are adaptively injected into the fused product according to the captured spectral dependence. Furthermore, training strategies of existing DL-based pansharpening methods can only group MS images with a constant number of bands into mini-batches. Therefore, we present a mask-based training method (called mask-training) to solve this problem. Benefiting from the mask-training, our ArbRPN can achieve superior performance and robustness during pansharpening. Extensive experiments show the superior performance of our ArbRPN with respect to the state-of-the-art (SOTA) methods applied to MS images with different numbers of bands. The code of our ArbRPN is available onhttps://github.com/Lihui-Chen/ArbRPN.git.

Summary Human behavior would lead to a significant impact on the environment. By monitoring the environment, we can indirectly monitor human behavior. Remote sensing (RS) technology provides a large number of multispectral (MS) images. When combining the Internet of things (IoT) technology, those images can be used for human behavioral monitoring. However, due to the limitation of the optical sensors embedded in satellites, the spatial resolution of MS image is relatively low, which poses a huge problem for further understanding these images. Pansharpening, also known as multisensor image fusion, aims to sharp an MS image to a high‐resolution multisensor image (HMS) by integrating a corresponding high‐resolution panchromatic (PAN) image. By doing so, the redundancy among big data can be effectively reduced. Traditional Intensity‐Hue‐Saturation (IHS)–based methods often suffer from spectral distortion. To address this problem, a novel pansharpening method is proposed in this paper. Different from those traditional IHS methods, the proposed method first decomposes MS and PAN into high‐frequency‐component (HFC) and low‐frequency‐component (LFC), respectively. Then, the guided filter (GF) is utilized to enhance the spectral information on the detail map. Furthermore, the detail map is refined according to the adaptive coefficients for each band of MS. By performing experiments, we demonstrate the proposed method can obtain satisfying results in both visual quality and object assessment among existing methods.

Abstract In recent years, deep convolutional neural networks (CNNs) have achieved a lot of outstanding results in super‐resolution with superior ability. However, the majority of CNNs only use a series of convolution kernels with the same size to extract features. This will cause limited receptive fields. In this work, we propose a parallel convolution attention network (PCAN) to extract features in an effective way. Specifically, a pair of parallel convolutions (PCs) with different kernel sizes is used in one layer in our network, which can extract features within different receptive fields, thereby making full use of the multiscale information. Meanwhile, we apply a channel‐spatial attention (CSA) module in each parallel convolution block to calculate and fuse channel attention and spatial attention. The obtained attention maps emphasize useful features. Experimental results demonstrate the superiority of our PCAN in comparison with the state‐of‐the‐art methods.

Abstract Imaging systems with different imaging sensors are widely applied to surveillance field, military field, and medicine field. Particularly, infrared imaging sensors can acquire thermal radiations emitted by different objects but lack textural details, and visible imaging sensors can capture abundant textural information but suffer from loss of scene information under poor weather conditions. The fusion of infrared and visible images can synthesize a new image with complementary information of the source images. In this paper, we present a deep learning method with encoder–decoder architecture for infrared and visible image fusion. Firstly, multiscale channel attention blocks are introduced to extract features at different scales, which can preserve more meaningful information and enhance the important information. Secondly, we utilize the improved fusion strategy based on visual saliency to fuse feature maps. Lastly, the fusion result is restored via reconstruction network. In comparison with other state‐of‐the‐art approaches, our experimental results achieve appealing performance on visual effects and objective assessments.

Summary Medical image fusion is a technology of combining multi‐modal images to generate a composite image, which is favorable to improve the capability of doctors in diagnosis and treatment of the disease. In order to achieve good performance, a fusion method by combining Laplacian pyramid (LP) and convolutional sparse representation (CSR) is proposed. In the proposed fusion method, LP transform is performed on each pair of pre‐registered computed tomography image and magnetic resonance image to obtain their detail layers and base layer. Then, the base layer is fused with a CSR‐based approach, whereas the detail layers are merged using the popular “max‐absolute” rule. Finally, the fused image is reconstructed by performing the inverse LP transform over the fused base layer and detail layers. The advantages of our method are that the texture detail information contained in source images can be fully extracted and the overall contrast of the final fused image will not be decreased. Experimental results demonstrate the superiority of the proposed method.

Summary Doctors always desire high‐resolution medical images to have accurate diagnosis. Super‐resolution (SR) is a technology that can improve the resolution of medical images. Convolutional neural network (CNN)–based SR methods have achieved desired performance in natural images. In this paper, we apply a deep dense SR (DDSR) convolutional neural networks model to two types of medical images, including Computerized Tomography (CT) images and Magnetic Resonance imaging (MRI) images. This network densely connects every hidden layer to learn high‐level features, which was first proposed for object recognition. A set of medical images is used for experiments. We compare the performance of DDSR with three state‐of‐the‐art SR network models, including SR Convolutional Neural Network (SRCNN), Fast SR Convolutional Neural Network (FSRCNN), and Very Deep SR Convolutional Neural Network (VDSR). Both the objective indices and subjective evaluations are used for comparison. The results show that the proposed network has better performances both on CT and MRI images.

Recently, Industry 4.0 has attracted much attention. It has close relations with the Internet of Things (IoT). On the other hand, convolutional neural networks (CNNs) have shown promising performance in many foundational services of the IoT applications. For the IoT applications with high-speed data streams and the requirement of time-sensitive actions, fast processing is demanded on small-scale platforms or even on IoT devices themselves. Therefore, it is inappropriate to employ cumbersome CNNs in IoT applications, making the study of model compression necessary. In knowledge transfer, it is common to employ a deep, well-trained network, called teacher, to guide a shallow, untrained network, called student, to have better performance. Previous works have made many attempts to transfer single-scale knowledge from teacher to student, leading to degradation of generalization ability. In this article, we introduce multiscale representations to knowledge transfer, which facilitates the generalization ability of student. We divide student and teacher into several stages. Student learns from multiscale knowledge provided by teacher at the end of each stage. Extensive experiments demonstrate the effectiveness of our proposed method both on image classification and on single image super-resolution. The huge performance gap between student and teacher is significantly narrowed down by our proposed method, making student suitable for IoT applications.

Recent advances in image super-resolution (SR) explored the power of deep learning to achieve a better reconstruction performance. However, the feedback mechanism, which commonly exists in human visual system, has not been fully exploited in existing deep learning based image SR methods. In this paper, we propose an image super-resolution feedback network (SRFBN) to refine low-level representations with high-level information. Specifically, we use hidden states in a recurrent neural network (RNN) with constraints to achieve such feedback manner. A feedback block is designed to handle the feedback connections and to generate powerful high-level representations. The proposed SRFBN comes with a strong early reconstruction ability and can create the final high-resolution image step by step. In addition, we introduce a curriculum learning strategy to make the network well suitable for more complicated tasks, where the low-resolution images are corrupted by multiple types of degradation. Extensive experimental results demonstrate the superiority of the proposed SRFBN in comparison with the state-of-the-art methods. Code is avaliable at https://github.com/Paper99/SRFBN_CVPR19.

Multisensor image fusion or pansharpening aims to sharpen a multispectral (MS) image by integrating the detail map derived from a panchromatic (Pan) image. The intensity-hue-saturation (IHS)-based methods are well adopted in pansharpening applications. However, the pansharpened MS images by IHS-based methods usually suffer from serious spectral distortions and local artifacts due to the mismatch between the estimated detail map and its ground truth. To overcome these defects, we propose a weighted least squares (WLS)-filter-based method in this letter. Different from existing IHS-based methods, the proposed method eliminates the influence of the low-frequency components of the Pan and MS images with the WLS filter. Moreover, the derived detail map is further refined based on the spectral signatures for different bands of the MS image. We test the proposed method on various satellites data; the experimental results demonstrate that the proposed method performs well in both spectral and spatial qualities.

In this paper, a sensing image super-resolution (SR) reconstruction method is proposed. Sparse dictionary dealing with remote sensing image SR problem is introduced in this work. The sparse dictionary is based on a sparsity model where the dictionary atoms have sparse representation over a basic dictionary. The sparse dictionary consists of two parts: basic dictionary and atom representation matrix. The sparse dictionary leads to compact representation and it is both adaptive and efficient. Furthermore, compared with conventional SR methods, two dictionary pairs, i.e. primitive sparse dictionary pair and residual sparse dictionary pair, are proposed. The primitive sparse dictionary pair is learned to reconstruct initial high-resolution (HR) remote sensing image from a single low-resolution (LR) input. However, the initial HR remote sensing image loses some details compare with the corresponding original HR image completely. Therefore, residual sparse dictionary pair is learned to reconstruct residual information. The proposed method is tested on remote sensing images, and the experimental results indicate that the proposed algorithm can provide substantial improvement in resolution of remote sensing images, and the results are superior in quality to the results produced by other methods.

Automatic container code recognition (ACCR) has become an indispensable aspect of current intelligent container management systems. In real applications, an ACCR module sometimes faces the problem of missing characters, i.e., not all the 11 container code characters (CCCs) appear in the input image. However, a few of the present methods can process container code images with missing characters. Therefore, a method is proposed to extract the CCCs for both the situation wherein all the 11 CCCs appear in an image and the situation wherein some CCCs are missing. In this method, hidden Markov model (HMM)-based segmentation confidence is proposed to describe the probability of the segmented characters belonging to the container code. Based on the segmentation confidence, the segmented characters are determined whether they belong to the container code or not, and if there are some characters missing, the positions of these characters can be estimated. Various container code images have been used to test the proposed method. The results of the tests show that the method is effective.