Low-rank tensor recovery methods within the tensor singular value decomposition (t-SVD) framework have demonstrated considerable success by leveraging the inherent low-dimensional structures of multi-dimensional data. However, previous approaches in this framework often rely on linear transforms or, in some cases, nonlinear transforms constructed with fully connected networks (FCNs). These methods typically promote a global low-rank structure, which may not fully exploit the nature of multiple subspaces in real-world data. In this work, we propose a nonlinear transform to capture long-range dependencies and diverse patterns across multiple subspaces of the data within the t-SVD framework. This approach provides a richer and more nuanced representation compared to the localized processing typically seen in FCN-based transforms. In the transform domain, we construct a low-rank self-representation layer that fully exploits the multi-subspace structure inherent in tensor data. Instead of merely enforcing overall low-rankness, our method minimizes the nuclear norm of a self-representation tensor, allowing for a more precise and joint characterization of multiple subspaces. This results in a more accurate representation of the data's intrinsic low-dimensional structures, leading to superior recovery performance. This new framework, termed the DEep Low-rank Tensor representAtion (DELTA), is evaluated across several typical multi-dimensional data recovery applications, including tensor completion, robust tensor completion, and spectral snapshot imaging. Experiments on various real-world multi-dimensional data illustrate the superior performance of our DELTA.

Completing multidimensional color images is a fundamental challenge in image processing and computer vision. However, some tensor-based methods often treat RGB channels as independent modes, thereby neglecting their intrinsic correlations. To address this limitation, we represent RGB values as pure quaternions and organize them into a quaternion tensor for holistic modeling that preserves chromatic relationships. To better capture the nonlinear characteristics inherent in visual data and to improve the compactness of low-rank representations, we propose a nonlinear transformation within the quaternion domain. This design enables more expressive modeling compared to conventional linear approaches. In addition, we introduce two novel regularization terms that jointly encode global low-rankness and local smoothness, with the nonlinear transformation further enhancing the exploitation of structural priors. The overall model is optimized via a nonlinear alternating direction method of multipliers (ADMM), with theoretical guarantees of convergence. Extensive experiments on several datasets demonstrate that the proposed method significantly outperforms state-of-the-art low-rank tensor and quaternion tensor recovery techniques in multidimensional color image completion tasks.

Deep learning has been widely applied to various aspects of computer vision, but the emergence of adversarial attacks raises concerns about its reliability. Adversarial training (AT) is one of the most effective defense methods, which incorporates adversarial examples into the training data. However, AT is typically employed in a discriminative learning manner, i.e., learning the mapping (conditional probability) from samples to labels, it essentially reinforces this mapping without considering the underlying data distribution. It is notable that adversarial examples often deviate from the distribution of normal (clean) samples. Therefore, building upon existing adversarial defense schemes, we propose to further exploit the distribution of normal samples, partly from the generative learning perspective, resulting in a novel robustness enhancement paradigm. We train a simple autoencoder (AE) autoregressively on normal samples to learn their prior distribution, effectively serving as an image manifold. This AE is then used as a manifold projection operator to incorporate the distribution information of normal samples. Specifically, we organically integrate the pretrained AE into the training process of both AT and adversarial distillation (AD), a method aiming at improving the robustness of small models with low capacity. Since the AE captures the distribution of normal samples, it can adaptively pull adversarial examples closer to the normal sample manifold, weakening the attack strength of adversarial samples and easing the learning of mappings from adversarial samples to correct labels. From the Pearson correlation coefficient (PCC) between the statistics on normal and adversarial examples, it’s validated that the AE indeed pulls adversarial samples closer to normal samples. Extensive experiments illustrate that our proposed adversarial defense paradigm significantly improves the robustness compared with previous state-of-the-art AT and AD methods.

Spectral super-resolution (SSR) from RGB images, which involves reconstructing hyperspectral images (HSIs) from color images, has recently received great attention. While convolutional neural network (CNN)-based methods have demonstrated strong performance, they often overlook the self-similarity across the spectral dimension of HSIs. Transformer-based approaches have addressed this limitation by leveraging self-attention mechanisms to capture spectral correlations. However, these methods encounter computational and memory overheads that scale quadratically with the size of the HSIs. To overcome these challenges, we introduce a novel Spectral-wise Low-Rank Attention (SLORA) mechanism that captures inter-spectral consistency in a low-dimensional space, thereby reducing both computational costs and model complexity. Additionally, we propose a flow-based refinement module to enhance generalization and performance on unseen HSIs. Experimental results from the NTIRE 2022 spectral reconstruction challenge and the spectral snapshot compression imaging task datasets validate the superiority of our method over state-of-the-art approaches.

Recently, the fully-connected tensor network (FCTN) decomposition has shown a powerful capability of depicting intrinsic correlations between any pair of tensor modes. But there exists a challenging question in FCTN decomposition-based methods, i.e., the accurate determination of the complicated FCTN-rank, which contains${N(N-1)}/{2}$elements for$N$th-order tensors. In this paper, we design a structural sparsity regularization for the FCTN decomposition, which estimates the complicated FCTN-rank by adaptively pruning near-zero groups in FCTN factor. Based on this regularization, we propose a noisy tensor completion (NTC) model, aiming at the recovery of a tensor from its partial and noisy observation. Besides, we design a proximal alternating minimization (PAM)-based algorithm to solve the model. In theorem, we prove a guarantee for the global convergence of the developed algorithm. To further accelerate our method for large-scale data sets, we customize the randomized block sampling strategy for general tensor network decomposition methods by updating factors from small samples. Experiments demonstrate that our strategy can accurately estimate the FCTN-rank and achieve better reconstruction performances, and our methods outperform the state-of-the-art methods in the reconstruction of different types of real-world tensors.

Tensor decompositions have been successfully applied to multidimensional data recovery. However, classical tensor decompositions are not suitable for emerging spatio-irregular multidimensional data (i.e., spatio-irregular tensor), whose spatial domain is non-rectangular, e.g., spatial transcriptomics data from bioinformatics and semantic units from computer vision. By using preprocessing (e.g., zero-padding or element-wise 0-1 weighting), the spatio-irregular tensor can be converted to a spatio-regular tensor and then classical tensor decompositions can be applied, but this strategy inevitably introduces bias information, leading to artifacts. How to design a tensor-based method suitable for emerging spatio-irregular tensors is an imperative challenge. To address this challenge, we propose a learnable transform-assisted tensor singular value decomposition (LTA-TSVD) for spatio-irregular tensor recovery, which allows us to leverage the intrinsic structure behind the spatio-irregular tensor. Specifically, we design a learnable transform to project the original spatio-irregular tensor into its latent spatio-regular tensor, and then the latent low-rank structure is captured by classical TSVD on the resulting regular tensor. Empowered by LTA-TSVD, we develop spatio-irregular low-rank tensor completion (SIR-LRTC) and spatio-irregular tensor robust principal component analysis (SIR-TRPCA) models for the spatio-irregular tensor imputation and denoising respectively, and we design corresponding solving algorithms with theoretical convergence. Extensive experiments including the spatial transcriptomics data imputation and hyperspectral image denoising show SIR-LRTC and SIR-TRPCA are superior performance to competing approaches and benefit downstream applications.

Tensor decomposition-based models have received increasing attention in hyperspectral image (HSI) denoising. However, tensor decompositions (e.g., Tucker decomposition and tensor singular value decomposition) in these HSI denoising models ignore exploiting the multiple components of the HSI, resulting in unsatisfactory denoising performance. To fully exploit the multiple components of the HSI, we develop a sparsity regularized rank-(L,M,N) block term decomposition (SR-BTD). In SR-BTD, the clean HSI is decomposed as the sum of multiple components, where each component is a sparse core tensor multiplied by matrices along each mode. The sparse regularization on each core tensor can benefit determining the low-rankness of each component with the unknown rank-(L,M,N) in the real world, leading to more accurately exploiting each component. Equipped with SR-BTD, we establish the HSI denoising model and design a hierarchical alternating least squares-based algorithm to efficiently solve the resulting model. Extensive experiments on simulated and real HSI denoising tasks demonstrate SR-BTD is superior to the competing tensor decompositions in terms of numerical results and visual quality.

Since the information across all bands of the cloud-contaminated region is missing, thick cloud removal for remote sensing images (RSIs) is still a challenging problem. Recently, the availability of rich spatial–spectral–temporal information for multitemporal RSIs provides the possibility for addressing the thick cloud removal problem. However, existing methods explore the holistic redundancy of multitemporal RSIs and neglect the important semantic clue of multitemporal images. In this letter, we propose a superpixel-oriented thick cloud removal (STORM) model for multitemporal images, where the multitemporal superpixel as the generic unit allows us to exploit redundancy with semantic clue in a low-rank optimization problem. To harness the resultant irregular fourth-order tensor (i.e., multitemporal superpixels) in the optimization problem, we cleverly introduce the weighted tensor to transform the irregular tensor into the regular tensor, which naturally leads to a standard low-rank tensor optimization problem. To tackle the tensor optimization problem, we develop a proximal alternating minimization (PAM)-based algorithm. Extensive simulated and real experiments on multitemporal RSIs acquired by Sentinel-2 and Landsat-8 satellites demonstrate the superior performance of the proposed method over the comparison methods.

Denoising of hyperspectral image (HSI) is challenging, especially when dealing with large-scale data. Model-based methods show promise in HSI denoising due to their good generalization, but they suffer from computational complexity due to complex priors [like nonlocal self-similarity (NSS)] and iterations, resulting in low efficiency for large-scale HSI processing. To address these challenges, we propose a fast large-scale HSI denoising (FallHyDe) method based on noniterative low-rank (LR) subspace representation to enjoy high denoising efficiency, effectiveness, and flexibility simultaneously. By leveraging the global spectral property of HSI, FallHyDe efficiently estimates spectral subspace and spatial representation coefficients (SRCs) from the observed noisy HSI, reducing computation complexity caused by the high spectral dimension during processing. In addition, we innovatively explore the presence of high signal-to-noise ratio bands (HSNRBs) in real HSI, enabling fast SRC estimation through a least squares problem without relying on complex priors and iterations. FallHyDe requires neither iteration nor parameter tuning, enabling our method to process large-scale HSI denoising quickly and flexibly. Experimental results on both simulated and real HSI datasets demonstrate that our proposed method not only achieves competitive results in quality but also speeds up the restoration by more than ten times than the representative fast HSI denoising methods. The code is available athttps://chenyong1993.github.io/yongchen.github.io/.

Recently, the transform-based tensor nuclear norm (TNN) methods have shown promising performance and drawn increasing attention in tensor completion (TC) problems. The main idea of these methods is to exploit the low-rank structure of frontal slices of the tensor under the transform. However, the transforms in TNN methods usually treat all modes equally and do not consider the different traits of different modes (i.e., spatial and spectral/temporal modes). To address this problem, we suggest a new low-rank tensor representation based on the coupled nonlinear transform (called CoNoT) for a better low-rank approximation. Concretely, spatial and spectral/temporal transforms in the CoNoT, respectively, exploit the different traits of different modes and are coupled together to boost the implicit low-rank structure. Here, we use the convolutional neural network (CNN) as the CoNoT, which can be learned solely from an observed multidimensional image in an unsupervised manner. Based on this low-rank tensor representation, we build a new multidimensional image completion model. Moreover, we also propose an enhanced version (called Ms-CoNoT) to further exploit the spatial multiscale nature of real-world data. Extensive experiments on real-world data substantiate the superiority of the proposed models against many state-of-the-art methods both qualitatively and quantitatively.

Hyperspectral image super-resolution (HISR) is about fusing a low-resolution hyperspectral image (LR-HSI) and a high-resolution multispectral image (HR-MSI) to generate a high-resolution hyperspectral image (HR-HSI). Recently, convolutional neural network (CNN)-based techniques have been extensively investigated for HISR yielding competitive outcomes. However, existing CNN-based methods often require a huge amount of network parameters leading to a heavy computational burden, thus, limiting the generalization ability. In this article, we fully consider the characteristic of the HISR, proposing a general CNN fusion framework with high-resolution guidance, called GuidedNet. This framework consists of two branches, including 1) the high-resolution guidance branch (HGB) that can decompose the high-resolution guidance image into several scales and 2) the feature reconstruction branch (FRB) that takes the low-resolution image and the multiscaled high-resolution guidance images from the HGB to reconstruct the high-resolution fused image. GuidedNet can effectively predict the high-resolution residual details that are added to the upsampled HSI to simultaneously improve spatial quality and preserve spectral information. The proposed framework is implemented using recursive and progressive strategies, which can promote high performance with a significant network parameter reduction, even ensuring network stability by supervising several intermediate outputs. Additionally, the proposed approach is also suitable for other resolution enhancement tasks, such as remote sensing pansharpening and single-image super-resolution (SISR). Extensive experiments on simulated and real datasets demonstrate that the proposed framework generates state-of-the-art outcomes for several applications (i.e., HISR, pansharpening, and SISR). Finally, an ablation study and more discussions assessing, for example, the network generalization, the low computational cost, and the fewer network parameters, are provided to the readers. The code link is: https://github.com/Evangelion09/GuidedNet.

In this article, we develop a new alternating projection method to compute nonnegative low rank matrix approximation for nonnegative matrices. In the nonnegative low rank matrix approximation method, the projection onto the manifold of fixed rank matrices can be expensive as the singular value decomposition is required. We propose to use the tangent space of the point in the manifold to approximate the projection onto the manifold in order to reduce the computational cost. We show that the sequence generated by the alternating projections onto the tangent spaces of the fixed rank matrices manifold and the nonnegative matrix manifold, converge linearly to a point in the intersection of the two manifolds where the convergent point is sufficiently close to optimal solutions. This convergence result based inexact projection onto the manifold is new and is not studied in the literature. Numerical examples in data clustering, pattern recognition and hyperspectral data analysis are given to demonstrate that the performance of the proposed method is better than that of nonnegative matrix factorization methods in terms of computational time and accuracy.

In this article, we propose a novel tensor learning and coding model for third-order data completion. The aim of our model is to learn a data-adaptive dictionary from given observations and determine the coding coefficients of third-order tensor tubes. In the completion process, we minimize the low-rankness of each tensor slice containing the coding coefficients. By comparison with the traditional predefined transform basis, the advantages of the proposed model are that: 1) the dictionary can be learned based on the given data observations so that the basis can be more adaptively and accurately constructed and 2) the low-rankness of the coding coefficients can allow the linear combination of dictionary features more effectively. Also we develop a multiblock proximal alternating minimization algorithm for solving such tensor learning and coding model and show that the sequence generated by the algorithm can globally converge to a critical point. Extensive experimental results for real datasets such as videos, hyperspectral images, and traffic data are reported to demonstrate these advantages and show that the performance of the proposed tensor learning and coding method is significantly better than the other tensor completion methods in terms of several evaluation metrics.

Pansharpening is a critical yet challenging low-level vision task that aims to obtain a higher-resolution image by fusing a multispectral (MS) image and a panchromatic (PAN) image. While most pansharpening methods are based on convolutional neural network (CNN) architectures with standard convolution operations, few attempts have been made with context-adaptive/dynamic convolution, which delivers impressive results on high-level vision tasks. In this paper, we propose a novel strategy to generate local-context adaptive (LCA) convolution kernels and introduce a new global harmonic (GH) bias mechanism, exploiting image local specificity as well as integrating global information, dubbed LAGConv. The proposed LAGConv can replace the standard convolution that is context-agnostic to fully perceive the particularity of each pixel for the task of remote sensing pansharpening. Furthermore, by applying the LAGConv, we provide an image fusion network architecture, which is more effective than conventional CNN-based pansharpening approaches. The superiority of the proposed method is demonstrated by extensive experiments implemented on a wide range of datasets compared with state-of-the-art pansharpening methods. Besides, more discussions testify that the proposed LAGConv outperforms recent adaptive convolution techniques for pansharpening.

Inverse problems in multi-dimensional imaging, e.g., completion, denoising, and compressive sensing, are challenging owing to the big volume of the data and the inherent illposedness. To tackle these issues, this work unsuper-visedly learns a hierarchical low-rank tensor factorization (HLRTF) by solely using an observed multi-dimensional image. Specifically, we embed a deep neural network (DNN) into the tensor singular value decompositionframe-work and develop the HLRTF, which captures the underlying low-rank structures of multi-dimensional images with compact representation abilities. This DNN herein serves as a nonlinear transform from a vector to another to help obtain a better low-rank representation. Our HLRTF infers the parameters of the DNN and the underlying low-rank structure of the original data from its observation via the gradient descent using a non-reference loss function in an unsupervised manner. To address the vanishing gradient in extreme scenarios, e.g., structural missing pixels, we introduce a parametric total variation regularization to constrain the DNN parameters and the tensor factor parameters with theoretical analysis. We apply our HLRTF for typical inverse problems in multi-dimensional imaging including completion, denoising, and snapshot spectral imaging, which demonstrates its generality and wide applicability. Extensive results illustrate the superiority of our method as compared with state-of-the-art methods.

Tensor completion aims at estimating missing values from an incomplete observation, playing a fundamental role for many applications. This work proposes a novel low-rank tensor completion model, in which the inherent low-rank prior and external degradation accordant data-driven prior are simultaneously utilized. Specifically, the tensor nuclear norm (TNN) is adopted to characterize the overall low-dimensionality of the tensor data. Meanwhile, an implicit regularizer is formulated and its related subproblem is solved via a deep convolutional neural network (CNN) under the plug-and-play framework. This CNN, pretrained for the inpainting task on a mass of natural images, is expected to express the external data-driven prior and this plugged inpainter is consistent with the original degradation process. Then, an efficient alternating direction method of multipliers (ADMM) is designed to solve the proposed optimization model. Extensive experiments are conducted on different types of tensor imaging data with the comparison with state-of-the-art methods, illustrating the effectiveness and the remarkable generalization ability of our method.

Image deraining is an important task for subsequent multimedia applications in rainy weather. Traditional deep learning-based methods rely on the quantity and diversity of training data, which is hard to cover all complex real-world rain scenarios. In this work, we propose the first Unsupervised Controllable Network (UConNet) to flexibly tackle different rain scenarios by adaptively controlling the network at the inference stage. Specifically, our unsupervised network takes the physics-based regularizations as the unsupervised loss function. Then, we sensibly derive the relationship between trade-off parameters of the loss function and the weightings of feature maps. Based on this relationship, our learned UConNet can be flexibly customized for different rain scenarios by controlling the weightings of feature maps at the inference stage. Alternatively, these weightings can also be efficiently determined by a learned weightings recommendation network. Extensive experiments for image and video deraining show that our method achieves very promising effectiveness, efficiency, and generalization abilities as compared with state-of-the-art methods.

The general tensor-based methods can recover missing values of multidimensional images by exploiting the low-rankness on the pixel level. However, especially when considerable pixels of an image are missing, the low-rankness is not reliable on the pixel level, resulting in some details losing in their results, which hinders the performance of subsequent image applications (e.g., image recognition and segmentation). In this article, we suggest a novel multiscale feature (MSF) tensorization by exploiting the MSFs of multidimensional images, which not only helps to recover the missing values on a higher level, that is, the feature level but also benefits subsequent image applications. By exploiting the low-rankness of the resulting MSF tensor constructed by the new tensorization, we propose the convex and nonconvex MSF tensor train rank minimization (MSF-TT) to conjointly recover the MSF tensor and the corresponding original tensor in a unified framework. We develop the alternating directional method of multipliers (ADMMs) to solve the convex MSF-TT and the proximal alternating minimization (PAM) to solve the nonconvex MSF-TT. Moreover, we establish the theoretical guarantee of convergence for the PAM algorithm. Numerical examples of real-world multidimensional images show that the proposed MSF-TT outperforms other compared approaches in image recovery and the recovered MSF tensor can benefit the subsequent image recognition.

This article proposes a new denoising method for hyperspectral images (HSIs) corrupted by mixtures (in a statistical sense) of stripe noise, Gaussian noise, and impulsive noise. The proposed method has three distinctive features: 1) it exploits the intrinsic characteristics of HSIs, namely, low-rank and self-similarity; 2) the observation noise is assumed to be additive and modeled by a mixture of Gaussian (MoG) densities; 3) the inference is performed with an expectation maximization (EM) algorithm, which, in addition to the clean HSI, also estimates the mixture parameters (posterior probability of each mode and variances). Comparisons of the proposed method with state-of-the-art algorithms provide experimental evidence of the effectiveness of the proposed denoising algorithm. A MATLAB demo of this work will be available athttps://github.com/TaiXiangJiangfor the sake of reproducibility.

Recently, transform-based tensor nuclear norm (TNN) minimization methods have received increasing attention for recovering third-order tensors in multi-dimensional imaging problems. The main idea of these methods is to perform the linear transform along the third mode of third-order tensors and then minimize the nuclear norm of frontal slices of the transformed tensor. The main aim of this paper is to propose a nonlinear multilayer neural network to learn a nonlinear transform by solely using the observed tensor in a self-supervised manner. The proposed network makes use of the low-rank representation of the transformed tensor and data-fitting between the observed tensor and the reconstructed tensor to learn the nonlinear transform. Extensive experimental results on different data and different tasks including tensor completion, background subtraction, robust tensor completion, and snapshot compressive imaging demonstrate the superior performance of the proposed method over state-of-the-art methods.

Completing missing entries in multidimensional visual data is a typical ill-posed problem that requires appropriate exploitation of prior information of the underlying data. Commonly used priors can be roughly categorized into three classes: global tensor low-rankness, local properties, and nonlocal self-similarity (NSS); most existing works utilize one or two of them to implement completion. Naturally, there arises an interesting question: can one concurrently make use of multiple priors in a unified way, such that they can collaborate with each other to achieve better performance? This work gives a positive answer by formulating a novel tensor completion framework which can simultaneously take advantage of the global-local-nonlocal priors. In the proposed framework, the tensor train (TT) rank is adopted to characterize the global correlation; meanwhile, two Plug-and-Play (PnP) denoisers, including a convolutional neural network (CNN) denoiser and the color block-matching and 3 D filtering (CBM3D) denoiser, are incorporated to preserve local details and exploit NSS, respectively. Then, we design a proximal alternating minimization algorithm to efficiently solve this model under the PnP framework. Under mild conditions, we establish the convergence guarantee of the proposed algorithm. Extensive experiments show that these priors organically benefit from each other to achieve state-of-the-art performance both quantitatively and qualitatively.

Hyperspectral images (HSIs) are of crucial importance in order to better understand features from a large number of spectral channels. Restricted by its inner imaging mechanism, the spatial resolution is often limited for HSIs. To alleviate this issue, in this work, we propose a simple and efficient architecture of deep convolutional neural networks to fuse a low-resolution HSI (LR-HSI) and a high-resolution multispectral image (HR-MSI), yielding a high-resolution HSI (HR-HSI). The network is designed to preserve both spatial and spectral information thanks to a new architecture based on: 1) the use of the LR-HSI at the HR-MSI's scale to get an output with satisfied spectral preservation and 2) the application of the attention and pixelShuffle modules to extract information, aiming to output high-quality spatial details. Finally, a plain mean squared error loss function is used to measure the performance during the training. Extensive experiments demonstrate that the proposed network architecture achieves the best performance (both qualitatively and quantitatively) compared with recent state-of-the-art HSI super-resolution approaches. Moreover, other significant advantages can be pointed out by the use of the proposed approach, such as a better network generalization ability, a limited computational burden, and the robustness with respect to the number of training samples. Please find the source code and pretrained models from https://liangjiandeng.github.io/Projects_Res/HSRnet_2021tnnls.html.

The popular tensor train (TT) and tensor ring (TR) decompositions have achieved promising results in science and engineering. However, TT and TR decompositions only establish an operation between adjacent two factors and are highly sensitive to the permutation of tensor modes, leading to an inadequate and inflexible representation. In this paper, we propose a generalized tensor decomposition, which decomposes an Nth-order tensor into a set of Nth-order factors and establishes an operation between any two factors. Since it can be graphically interpreted as a fully-connected network, we named it fully-connected tensor network (FCTN) decomposition. The superiorities of the FCTN decomposition lie in the outstanding capability for characterizing adequately the intrinsic correlations between any two modes of tensors and the essential invariance for transposition. Furthermore, we employ the FCTN decomposition to one representative task, i.e., tensor completion, and develop an efficient solving algorithm based on proximal alternating minimization. Theoretically, we prove the convergence of the developed algorithm, i.e., the sequence obtained by it globally converges to a critical point. Experimental results substantiate that the proposed method compares favorably to the state-of-the-art methods based on other tensor decompositions.

For multi-temporal remote sensing (RS) images, the distribution of surface materials is constant concerning time and the same material shows different spectral features at different times. Decomposing the image at each time into an abundance tensor and temporal features, there is a strong similarity between abundance tensors of all time. Based on this observation, we suggest a blind thick cloud/shadow removal model, which exploits the sparsity of the cloud component and the similarity between abundance tensors, achieving both cloud detection and multi-temporal information restoration. Moreover, a mask refinement strategy is designed to pursue the optimal cloud/shadow mask. We develop an efficient algorithm to solve the proposed model based on the augmented Lagrange multiplier method. The results of simulated experiments in different scenarios verify the superiority of the proposed method for thick cloud/shadow removal.

Recently, unsupervised deep learning-based methods have shown an empirical success in hyperspectral images (HSIs) denoising, profiting from the strong representation ability of convolutional neural networks. However, these methods only can describe the local structure of the spatial dimension, which is restricted to very limited local receptive fields. To overcome this difficulty, a novel HSIs denoising model based on the deep image prior (DIP) framework is proposed by adding a tensor ring (TR) decomposition, which can enlarge the receptive field of the spatial dimension and capture global spectral correlation simultaneously. Unlike the previous DIP framework that directly minimizes the objective function, we develop an algorithm based on proximal alternating minimization to decouple the model into the classic DIP framework and TR cores least-squares problems, which are easy to solve. Experimental results verify that the proposed DIP- TR compares favorably with compared methods in terms of quality metrics and visual performance.

Removing rain streaks in videos has recently received much attention. Existing hand-crafted priors-based methods suffer from limited representation abilities, and supervised deep learning methods need high-quality training data. This paper proposes a novel video rain streaks removal method by reconciling hand-crafted and self-supervised deep priors. The hand-crafted priors include the learned gradient prior, the sparse prior, and the temporal local smooth prior. Meanwhile, a deep convolutional neural network is employed to self-supervisedly capture the deep prior of the clean video without any training data. Our method organically integrates hand-crated priors and self-supervised deep priors to achieve both high generalization abilities and representation abilities. Thus, our method can faithfully remove directional rain streaks in real world videos. To address the resulting model, we introduce an alternating direction multiplier method algorithm. Extensive experiments validate the superiority of our method over state-of-the-art methods.

In hyperspectral image (HSI) denoising, subspace-based denoising methods can reduce the computational complexity of the denoising algorithm. However, the existing matrix subspaces, which are generated by the unfolding matrix of the HSI tensor, cannot completely represent a tensor since the unfolding operation will destroy the tensor structure. To overcome this, we design a novel basis tensor that is directly learned from the original tensor and present a tensor subspace representation (TenSR), which is a more authentic representation for delivering the intrinsic structure of the tensor than a matrix subspace representation. Equipped with the TenSR, we then propose a TenSR-based HSI denoising (TenSRDe) model, which simultaneously considers the low-tubal rankness of the HSI tensor and the nonlocal self-similarity of the coefficient tensor. Moreover, we develop an efficient proximal alternating minimization (PAM) algorithm to solve the proposed nonconvex model and theoretically prove that the algorithm globally converges to a critical point. Experiments implemented on simulated and real data sets substantiate the denoising effect and efficiency of the proposed method.

This paper addresses the tensor completion problem, which aims to recover missing information of multi-dimensional images. How to represent a low-rank structure embedded in the underlying data is the key issue in tensor completion. In this work, we suggest a novel low-rank tensor representation based on coupled transform, which fully exploits the spatial multi-scale nature and redundancy in spatial and spectral/temporal dimensions, leading to a better low tensor multi-rank approximation. More precisely, this representation is achieved by using two-dimensional framelet transform for the two spatial dimensions, one/two-dimensional Fourier transform for the temporal/spectral dimension, and then Karhunen-Loéve transform (via singular value decomposition) for the transformed tensor. Based on this low-rank tensor representation, we formulate a novel low-rank tensor completion model for recovering missing information in multi-dimensional visual data, which leads to a convex optimization problem. To tackle the proposed model, we develop the alternating directional method of multipliers (ADMM) algorithm tailored for the structured optimization problem. Numerical examples on color images, multispectral images, and videos illustrate that the proposed method outperforms many state-of-the-art methods in qualitative and quantitative aspects.

Recently emerged deep learning methods have achieved great success in single image rain streaks removal. However, existing methods ignore an essential factor in the rain streaks generation mechanism, i.e., the motion blur leading to the line pattern appearances. Thus, they generally produce overderaining or underderaining results. In this article, inspired by the generation mechanism, we propose a novel rain streaks removal framework using a kernel-guided convolutional neural network (KGCNN), achieving state-of-the-art performance with a simple network architecture. More precisely, our framework consists of three steps. First, we learn the motion blur kernel by a plain neural network, termed parameter network, from the detail layer of a rainy patch. Then, we stretch the learned motion blur kernel into a degradation map with the same spatial size as the rainy patch. Finally, we use the stretched degradation map together with the detail patches to train a deraining network with a typical ResNet architecture, which produces the rain streaks with the guidance of the learned motion blur kernel. Experiments conducted on extensive synthetic and real data demonstrate the effectiveness of the proposed KGCNN, in terms of rain streaks removal and image detail preservation.

The tensor tubal rank, defined based on the tensor singular value decomposition (t-SVD), has obtained promising results in hyperspectral image (HSI) denoising. However, the framework of the t-SVD lacks flexibility for handling different correlations along different modes of HSIs, leading to suboptimal denoising performance. This article mainly makes three contributions. First, we introduce a new tensor rank named tensor fibered rank by generalizing the t-SVD to the mode-k t-SVD, to achieve a more flexible and accurate HSI characterization. Since directly minimizing the fibered rank is NP-hard, we suggest a three-directional tensor nuclear norm (3DTNN) and a three-directional log-based tensor nuclear norm (3DLogTNN) as its convex and nonconvex relaxation to provide an efficient numerical solution, respectively. Second, we propose a fibered rank minimization model for HSI mixed noise removal, in which the underlying HSI is modeled as a low-fibered-rank component. Third, we develop an efficient alternating direction method of multipliers (ADMMs)-based algorithm to solve the proposed model, especially, each subproblem within ADMM is proven to have a closed-form solution, although 3DLogTNN is nonconvex. Extensive experimental results demonstrate that the proposed method has superior denoising performance, as compared with the state-of-the-art competing methods on low-rank matrix/tensor approximation and noise modeling.

The main aim of this paper is to develop a framelet representation of the tensor nuclear norm for third-order tensor recovery. In the literature, the tensor nuclear norm can be computed by using tensor singular value decomposition based on the discrete Fourier transform matrix, and tensor completion can be performed by the minimization of the tensor nuclear norm which is the relaxation of the sum of matrix ranks from all Fourier transformed matrix frontal slices. These Fourier transformed matrix frontal slices are obtained by applying the discrete Fourier transform on the tubes of the original tensor. In this paper, we propose to employ the framelet representation of each tube so that a framelet transformed tensor can be constructed. Because of framelet basis redundancy, the representation of each tube is sparsely represented. When the matrix slices of the original tensor are highly correlated, we expect the corresponding sum of matrix ranks from all framelet transformed matrix frontal slices would be small, and the resulting tensor completion can be performed much better. The proposed minimization model is convex and global minimizers can be obtained. Numerical results on several types of multi-dimensional data (videos, multispectral images, and magnetic resonance imaging data) have tested and shown that the proposed method outperformed the other testing methods.

Images captured in rainy conditions are often corrupted by unexpected rain streaks, which severely degrade the performance of subsequent processes in outdoor computer vision systems. In this paper, we exploit the directional smoothness of rain streaks for the single-image rain streaks removal and propose a convex model that uses the directional total variation (DTV) to characterize the smoothness of rain streaks in arbitrary orientations. The proposed model consists of four terms: the fidelity term, the ℓ1norm for the sparsity of rain streaks, and two DTV regularization terms for the directional smoothness and the piecewise smoothness of rain streaks and rain-free backgrounds, respectively. To solve the proposed model, we develop an efficient algorithm based on the alternating direction method of multipliers (ADMM) framework. Extensive experimental results on both synthetic and real rainy images show that our method outperforms the recent state-of-the-art methods visually and quantitatively.

In this paper, we propose a novel low-tubal-rank tensor recovery model, which directly constrains the tubal rank prior for effectively removing the mixed Gaussian and sparse noise in hyperspectral images. The constraints of tubal-rank and sparsity can govern the solution of the denoised tensor in the recovery procedure. To solve the constrained low-tubal-rank model, we develop an iterative algorithm based on bilateral random projections to efficiently solve the proposed model. The advantage of random projections is that the approximation of the low-tubal-rank tensor can be obtained quite accurately in an inexpensive manner. Experimental examples for hyperspectral image denoising are presented to demonstrate the effectiveness and efficiency of the proposed method.

In this paper, we propose a new tensor rank, named fibered rank, by generalizing the tensor singular value decomposition (t-SVD) to the mode-k t-SVD. It factorizes a three-way tensor into two mode-k orthogonal tensors and a mode-k diagonal tensor. To efficiently remove mixed noise and finely preserve the structure information, we propose a novel hyperspectral image denoising model based on the three-directional tensor nuclear norm (3DTNN), which is a convex relaxation of the fibered rank. An efficient alternating direction method of multipliers (ADMM)-based algorithm is developed to solve the proposed model. Experimental results demonstrate the superiority of the proposed method over the compared ones.

Rain streaks removal is an important issue in outdoor vision systems and has recently been investigated extensively. In this paper, we propose a novel video rain streak removal approach FastDeRain, which fully considers the discriminative characteristics of rain streaks and the clean video in the gradient domain. Specifically, on the one hand, rain streaks are sparse and smooth along the direction of the raindrops, whereas on the other hand, clean videos exhibit piecewise smoothness along the rain-perpendicular direction and continuity along the temporal direction. Theses smoothness and continuity results in the sparse distribution in the different directional gradient domain, respectively. Thus, we minimize 1) the ℓ1 norm to enhance the sparsity of the underlying rain streaks, 2) two ℓ1 norm of unidirectional Total Variation (TV) regularizers to guarantee the anisotropic spatial smoothness, and 3) an ℓ1 norm of the time-directional difference operator to characterize the temporal continuity. A split augmented Lagrangian shrinkage algorithm (SALSA) based algorithm is designed to solve the proposed minimization model. Experiments conducted on synthetic and real data demonstrate the effectiveness and efficiency of the proposed method. According to comprehensive quantitative performance measures, our approach outperforms other state-of-the-art methods, especially on account of the running time. The code of FastDeRain can be downloaded at https://github.com/TaiXiangJiang/FastDeRain.

This paper proposes a new denoising method for hyperspectral images (HSIs) corrupted by mixtures (in a statistical sense) of stripe noise, Gaussian noise, and impulsive noise. The proposed method has three distinctive features: 1) it exploits the intrinsic characteristics of HSIs, namely, low-rank and self-similarity; 2) the observation noise is assumed to be additive and modeled by a mixture of Gaussian (MoG) densities; 3) the inference is performed with an expectation maximization (EM) algorithm, which, in addition to the clean HSI, also estimates the mixture parameters (posterior probability of each mode and variances). Comparisons of the proposed method with state-of-the-art algorithms provide experimental evidence of the effectiveness of the proposed denoising algorithm.

Rain streaks removal is an important issue of the outdoor vision system and has been recently investigated extensively. In this paper, we propose a novel tensor based video rain streaks removal approach by fully considering the discriminatively intrinsic characteristics of rain streaks and clean videos, which needs neither rain detection nor time-consuming dictionary learning stage. In specific, on the one hand, rain streaks are sparse and smooth along the raindrops direction, and on the other hand, the clean videos possess smoothness along the rain-perpendicular direction and global and local correlation along time direction. We use the l1 norm to enhance the sparsity of the underlying rain, two unidirectional Total Variation (TV) regularizers to guarantee the different discriminative smoothness, and a tensor nuclear norm and a time directional difference operator to characterize the exclusive correlation of the clean video along time. Alternation direction method of multipliers (ADMM) is employed to solve the proposed concise tensor based convex model. Experiments implemented on synthetic and real data substantiate the effectiveness and efficiency of the proposed method. Under comprehensive quantitative performance measures, our approach outperforms other state-of-the-art methods.