Currently, pretrained models are rapidly scaling in size, which substantially increases the cost of fine-tuning them for downstream tasks. To address this challenge, parameter-efficient fine-tuning (PEFT) methods have been developed to optimize a minimal set of parameters for adaptation. While current PEFT approaches predominantly employ an "additive'' strategy, introducing learnable modules into inputs or architectures, neglect the inherent knowledge embedded within pretrained models, which may be redundant or even conflict with downstream tasks. This limitation leads to increased inference latency and suboptimal transfer performance, particularly in scenarios with significant domain gaps. In this paper, we propose a Subtractive Fine-tuning Paradigm(SFP), which converts multiple redundant operations within the original module into a linear transformation to enhance inference speed and model performance. Specifically, we introduce a compact filter block to replace specific module with interference and redundancy in the original structure to reduce model conflicts. By using a pseudo inverse matrix to construct filter block, ensuring that it can inherit the knowledge of the replacement module, and then freezing the rest of the model, only fine-tuning the filter block is performed to eliminate interference and redundant knowledge, thereby enhancing the model’s adaptability to downstream tasks. Experimental results demonstrate that our SFP outperforms existing PEFT methods in accuracy while decreasing the overall model parameters by 12%. Compared to full fine-tuning, the accuracy has increased by 8.47%(74.04% vs. 65.57%, VTAB).

Implicit neural representation (INR) has emerged as a powerful paradigm for visual data representation. However, classical INR methods represent data in the original space mixed with different frequency components, and several feature encoding parameters (e.g., the frequency parameter $\omega$ω or the rank $R$R) need manual configurations. In this work, we propose a self-evolving cross-frequency INR using the Haar wavelet transform (termed CF-INR), which decouples data into four frequency components and employs INRs in the wavelet space. CF-INR allows the characterization of different frequency components separately, thus enabling higher accuracy for data representation. To more precisely characterize cross-frequency components, we propose a cross-frequency tensor decomposition paradigm for CF-INR with self-evolving parameters, which automatically updates the rank parameter $R$R and the frequency parameter $\omega$ω for each frequency component through self-evolving optimization. This self-evolution paradigm eliminates the laborious manual tuning of these parameters, and learns a customized cross-frequency feature encoding configuration for each dataset. We evaluate CF-INR on a variety of visual data representation and inverse imaging problems, including image regression, inpainting, denoising, and cloud removal. Extensive experiments demonstrate that CF-INR outperforms state-of-the-art methods in each case.

Tensor decompositions are powerful tools for capturing the low-rank structure of dynamic videos. However, existing tensor decompositions primarily consider pixel-wise interactions, thus capturing solely global spatio-temporal correlations and struggling to handle the complex patterns that are inherent to dynamic videos in real-world applications. To overcome this limitation, we propose a dynamic Bhattacharya-Mesner (DyBM) decomposition, which represents the dynamic video as a sum of terms, with each term being the convolution of a BM-rank 1 tensor and a learnable three-dimensional filter. The newly constructed filters enable DyBM decomposition to establish patch-wise interactions in BM-rank 1 tensors, effectively capturing both global and local spatio-temporal correlations in dynamic videos. We further provide a physical interpretation of the factors in DyBM decomposition and offer an in-depth discussion of its relationship to the original BM decomposition. To evaluate the effectiveness of DyBM decomposition, we build a dynamic video recovery model. To solve the model, we develop a corresponding optimization algorithm with a theoretical convergence guarantee. Extensive experiments verify that DyBM decomposition-based method performs more favorably than the state-of-the-art tensor decomposition-based methods especially for dynamic videos.

Recently, the block-term decomposition with rank- $(L_{r}, L_{r}, 1)$ (termed as LL1 decomposition), which decomposes a third-order tensor into the sum of the outer products between vector and matrix factors, has received increasing attention for high-dimensional image reconstruction. However, the fixed low-rank matrix decomposition in LL1 is restricted to third-order tensors, which hinders its development for higher-order tensor data (i.e., order $N \gt 3$ ). To address this, we propose a Block Customized Topology Term Decomposition (BCTD), which represents an $N$ th-order tensor as a sum of outer products of basis vectors and customized $(N-1)$ th-order coefficient tensors with flexible internal topological structures. The proposed BCTD enjoys two advantages: Firstly, it allows tackling higher-order tensors beyond the third-order tensor setting of LL1, which can better preserve the high-dimensional structure of the tensor. Secondly, it allows each term to have a customized topological structure beyond the fixed topological structure (i.e., low-rank matrix decomposition) in LL1, which can better explore the intrinsic high-dimensional low-rank structures of the tensor. To evaluate the performance of the proposed BCTD, we build the corresponding high-dimensional image reconstruction model and provide a theoretical generalization error bound between the recovered tensor of the proposed model and the underlying tensor. To solve the resulting optimization problem, we apply a proximal alternating minimization (PAM)-based algorithm with a theoretical convergence guarantee. Extensive experimental results on high-dimensional image completion and compression tasks using real-world datasets (color videos and light field images) demonstrate the superiority of the proposed model over other baseline models.

Spatio-temporal traffic data imputation is a fundamental component in intelligent transportation systems, which can significantly improve data quality and enhance the accuracy of downstream data mining tasks. Recently, low-rank tensor representation has shown great potential for spatio-temporal traffic data imputation. However, the low-rank assumption focuses on the global structure, neglecting the critical spatial topology and local temporal dependencies inherent in spatio-temporal data. To address these issues, we propose a topology-induced low-rank tensor representation (TILR), which can accurately capture the underlying low-rankness of the spatial multi-scale features induced by topology knowledge. Moreover, to exploit local temporal dependencies, we suggest a learnable convolutional regularization framework, which not only includes some classical convolution-based regularizers but also leads to the discovery of new convolutional regularizers. Equipped with the suggested TILR and convolutional regularizer, we build a unified low-rank tensor model harmonizing spatial topology and temporal dependencies for traffic data imputation, which is expected to deliver promising performance even under extreme and complex missing scenarios. To solve the proposed nonconvex model, we develop an efficient alternating direction method of multipliers (ADMM)-based algorithm and analyze its computational complexity. Extensive experiments demonstrate that the proposed model outperforms state-of-the-art baselines for various missing scenarios. These results reveal the critical synergy between topology-aware low-rank constraint and temporal dynamic modeling for spatio-temporal data imputation.

Existing arbitrary-scale super-resolution (ASSR) methods suffer from quadratic computational complexity w.r.t. image scale due to the reliance on multi-layer perceptrons (MLPs) to query dense spatial coordinate matrices. The inefficiency becomes particularly pronounced when extending to high-dimensional imaging modalities. To address these limitations, we propose a novel functional tensor decomposition (FTD) framework that fundamentally reconfigures the computational paradigm for ASSR. Specifically, we propose 1) a separation mechanism that employs distinct MLPs to query separable spatial coordinate vectors, substantially reducing decoder MLP invocations, and 2) functional tensor Tucker or CP decompositions for efficient factor matrix integration. The FTD framework delivers three key advantages: 1) Superior scalability to high-dimensional imaging modalities, such as hyperspectral images (HSIs), by virtue of the FTD design; 2) Significantly enhanced inference speed across scales; 3) Faster convergence towards a desired training model. Extensive experiments validate FTD's exceptional performance in HSI joint spatial-spectral ASSR, achieving up to 90.04% reduction in inference time and substantial performance improvements. For conventional image ASSR, our method improves both inference speed and convergence efficiency, achieving up to 88.82% inference time reduction and superior few-shot generalization capabilities due to faster convergence.

Remote sensing image (RSI) denoising is an important and fundamental task in RSI processing. Existing denoising methods usually assume that RSI lies in a single matrix or tensor subspace. However, due to the wavelength difference or/and temporal variability, the assumption of a single subspace may not be suitable for RSI. To address this, we propose a tensor multi-subspace representation (TenMSR) for RSI mixed noise removal. To be specific, in this work, we introduce TenMSR to finely characterize the intrinsic tensor multi-subspace structure of RSI. Compared with the single matrix/tensor subspace-based methods, the proposed method can not only precisely describe the wavelength difference or/and temporal variability of RSI but also produce a more compact image distribution in tensor multi-subspace. To mine and preserve the multi-subspace structure, we introduce a nonlinear transform-based 3-D tensor nuclear norm to characterize the tensor low rankness of the multi-subspace representation coefficient. An effective algorithm based on the proximal alternating minimization (PAM) framework is developed to solve the proposed model with theoretical convergence analysis. Extensive experiments show the effectiveness and superiority of the proposed method over existing state-of-the-art single matrix/tensor subspace RSI denoising methods.

Spatial transcriptomics (ST) are emerging technologies that reveal spatial distributions of gene expressions within tissues, serving as important ways to uncover biological insights. However, the irregular spatial profiles and variability of genes make it challenging to integrate spatial information with gene expression under a computational framework. Current algorithms mostly utilize spatial graph neural networks to encode spatial information, which may incur increased computational costs and may not be flexible enough to depict complex spatial configurations. In this study, we introduce a concise yet effective representation framework, STINR, for deciphering ST data. STINR leverages an implicit neural representation (INR) to continuously represent ST data, which efficiently characterizes spatial and slice-wise correlations of ST data by inheriting the implicit smoothness of INR. STINR allows easier integration of multiple slices and multi-omics without any alignment, and serves as a potent tool for various biological tasks including gene imputation, gene denoising, spatial domain detection, and cell-type deconvolution stemed from ST data. In particular, STINR identifies the thinnest cortex layer in the dorsolateral prefrontal cortex which previous methods were unable to achieve, and more accurately identifies tumor regions in the human squamous cell carcinoma, showcasing its practical value for biological discoveries. Code at https://github.com/YisiLuo/STINR.

Abstract Multimodal personality analysis targets accurately detecting personality traits by incorporating related multimodal information. However, existing methods focus on unimodal features while overlooking the bimodal association features crucial for this interdisciplinary task. Therefore, we propose a multimodal personality prediction framework based on an adaptive graph transformer network and multi‐task learning. Firstly, we utilize pre‐trained models to learn specific representations from different modalities. Here, we employ pre‐trained multimodal models' encoders as the backbones of the modality‐specific extraction methods to mine unimodal features. Specifically, we introduce a novel adaptive graph transformer network to mine personality‐related bimodal association features. This network effectively learns higher‐order temporal dependencies based on relational graphs and emphasizes more significant features. Furthermore, we utilize a multimodal channel attention residual fusion module to obtain the fused features, and we propose a multimodal and unimodal joint learning regression head to learn and predict scores for personality traits. We design a multi‐task loss function to enhance the robustness and accuracy of personality prediction. Experimental results on the two benchmark datasets demonstrate the effectiveness of our framework, which outperforms the state‐of‐the‐art methods. The code is available at https://github.com/RongquanWang/PPF-AGTNMTL .

Forecasting urban traffic states is crucial to transportation network monitoring and management, playing an important role in the decision-making process. Despite the substantial progress that has been made in developing accurate, efficient, and reliable algorithms for traffic forecasting, most existing approaches fail to handle sparsity, high-dimensionality, and nonstationarity in traffic time series and seldom consider the temporal dependence between traffic states. To address these issues, this work presents a Hankel temporal matrix factorization (HTMF) model using the Hankel matrix in the lower dimensional spaces under a matrix factorization framework. In particular, we consider an alternating minimization scheme to optimize the factor matrices in matrix factorization and the Hankel matrix in the lower dimensional spaces simultaneously. To perform traffic state forecasting, we introduce two efficient estimation processes on real-time incremental data, including an online imputation (i.e., reconstruct missing values) and an online forecasting (i.e., estimate future data points). Through extensive experiments on the real-world Uber movement speed data set in Seattle, Washington, we empirically demonstrate the superior forecasting performance of HTMF over several baseline models and highlight the advantages of HTMF for addressing sparsity, nonstationarity, and short time series. History: Accepted by Ram Ramesh, Area Editor for Data Science & Machine Learning. Funding: This research was supported by the Institute for Data Valorisation, the Interuniversity Research Centre on Enterprise Networks, Logistics and Transportation, the National Natural Science Foundation of China [Grants 12371456, 72101049, 72232001], the Sichuan Science and Technology Program [Grant 2024NSFJQ0038], and the Fundamental Research Funds for the Central Universities [Grant DUT23RC(3)045].

Nonlocal self-similarity (NSS) is an important prior that has been successfully applied in multi-dimensional data processing tasks, e.g., image and video recovery. However, existing NSS-based methods are solely suitable for meshgrid data such as images and videos, but are not suitable for emerging off-meshgrid data, e.g., point cloud and weather data. In this work, we revisit the NSS from the continuous representation perspective and propose a novel Continuous Representation-based NonLocal method (termed as CRNL), which has two innovative features as compared with classical nonlocal methods. First, based on the continuous representation, our CRNL unifies the measure of self-similarity for on-meshgrid and off-meshgrid data and thus is naturally suitable for both of them. Second, the nonlocal continuous groups can be more compactly and efficiently represented by the coupled low-rank function factorization, which simultaneously exploits the similarity within each group and across different groups, while classical nonlocal methods neglect the similarity across groups. This elaborately designed coupled mechanism allows our method to enjoy favorable performance over conventional NSS methods in terms of both effectiveness and efficiency. Extensive multi-dimensional data processing experiments on-meshgrid (e.g., image inpainting and image denoising) and off-meshgrid (e.g., weather data prediction and point cloud recovery) validate the versatility, effectiveness, and efficiency of our CRNL as compared with state-of-the-art methods.

Abstract. In this work, we study the subspace clustering and recovery of multidimensional images. Existing matrix-based/tensor-based subspace clustering methods successfully consider unilateral information (i.e., the similarity between image samples) to cluster samples into subspaces by using low-rank representation. The key issue of the unilateral representation-based methods is that the number of samples in each subspace should be sufficient for subspace representation. In practice, the clustering performance can be degraded when there is only a small number of observed samples in each subspace. To address the problem of insufficient observed samples, we propose to introduce hidden tensor data to supplement an insufficient number of observed samples. We employ both observed samples and hidden tensor data under low-rank constraints so that a new bilateral tensor low-rank representation (BTLRR) in subspace clustering is formulated. We show that a closed-form solution of block-diagonal tensor structure is obtained in subspace clustering of observed samples and hidden tensor data. Also the proposed BTLRR optimization problem can be solved by using the convex relaxation technique and augmented Lagrangian multiplier algorithm. The proposed BTLRR can fully explore the bilateral information of observations, including not only the similarity between samples but also the relationship among features. Extensive numerical results on multidimensional image data clustering and recovery illustrate that the effectiveness and robustness of the proposed bilateral representation are better than those of state-of-the-art methods (e.g., the popular LRR and TLRR methods).

Abstract. Block term decomposition (BTD), which decomposes a third-order tensor into multiple terms with the multilinear rank [Formula: see text], has garnered increasing attention for high-dimensional image representation. The idea of decomposing the tensor into multiple terms has the potential to reveal the underlying different structures of the original tensor. However, BTD practically struggles to reveal these underlying different structures in the original domain, which locks the potential of the BTD. To address this problem, we propose a dictionary-based BTD (DBTD) for third-order tensors by revisiting the BTD from the convolutional dictionary learning perspective, which can better reveal the underlying different structures of the original tensor. The proposed DBTD decomposes the original tensor into multiple different terms, where each term is represented by the convolution of an adaptive dictionary and the corresponding low-rank coefficient. Herein, the adaptive dictionaries can represent distinct patterns and contribute to the DBTD’s ability to reveal the underlying different structures of the original tensor. Moreover, we establish the essential uniqueness guarantee for the DBTD. Empowered with DBTD, we suggest a high-dimensional image recovery model and develop an efficiently solving algorithm with a convergence guarantee. Numerical results on real-world high-dimensional images demonstrate that the proposed DBTD outperforms other competing decompositions in image recovery and benefits subsequent image applications.

Abstract. Recently, we have witnessed the success of total variation (TV) for many imaging applications. However, traditional TV is defined on the original pixel domain, which limits its potential. In this work, we suggest a new TV regularization defined on the neural domain. Concretely, the discrete data is implicitly and continuously represented by a deep neural network (DNN), and we use the derivatives of DNN outputs with respect to (w.r.t.) input coordinates to capture local correlations of data. As compared with classical TV on the original domain, the proposed TV on the neural domain (termed NeurTV) enjoys the following advantages. First, NeurTV is free of discretization error induced by the discrete difference operator. Second, NeurTV is not limited to meshgrid but is suitable for both meshgrid and non-meshgrid data. Third, NeurTV can more exactly capture local correlations across data for any direction and any order of derivatives attributed to the implicit and continuous nature of neural domain. We theoretically reinterpret NeurTV under the variational approximation framework, which allows us to build the connection between NeurTV and classical TV and inspires us to develop variants (e.g., space-variant NeurTV). Extensive numerical experiments with meshgrid data (e.g., color and hyperspectral images) and non-meshgrid data (e.g., point clouds and spatial transcriptomics) showcase the effectiveness of the proposed methods.

Recently, the classical tensor-tensor product (T-product) has attracted considerable attention for capturing the interactions between tensor factors. However, the mode-3 consistency in the T-product restricts its flexibility and expressive ability. To break the restriction, we suggest a tensor-tensor product for arbitrary mode-3 dimension (termed as Art-product) which enables us to flexibly and expressively capture the interactions between tensor factors. Concretely, by leveraging the exclusive hierarchical nonlinear transforms along the third mode, two tensor factors with inconsistency dimensions are first transformed into the corresponding latent factors with consistency dimensions. The face-wise product is then performed between these latent factors with consistency dimensions. Empowered with this Art-product, we can readily deconstruct and reconstruct new tensor network decomposition from an interaction perspective. As a representative example, we redesign the tensor train decomposition which can benefit from the advantage of the Art-product. Extensive experiments on multi-spectral images, color videos, and light field data sustain the superiority of tensor train decomposition equipped with Art-product over classic tensor decomposition.

Implicit neural representation (INR) has emerged as a powerful representation for data (e.g., multispectral images and videos). Previously, most INR methods directly represent data in the original space. However, since different frequency components are mixed in the original space, it is difficult to capture these frequency components simultaneously and accurately. To alleviate this limitation, we suggest a new frequency-aware implicit neural representation (FA-INR) working in a physically interpretable and learnable frequency space by cleverly introducing an extra frequency dimension, which allows us to readily decouple and modulate different frequency components, leading to a more accurate characterization of different frequency components in a divide-and-conquer manner. Specifically, the FA-INR consists of two important modules, i.e., the frequency module and the integration module. In the frequency module, we propose a new low-rank tensor frequency function to compactly and continuously represent the latent frequency space. In the integration module, different frequency components are adaptively integrated back to the original space. Extensive experiments on various multi-dimensional data, including multispectral images, color videos, and light field data, demonstrate that the proposed FA-INR significantly outperforms the state-of-the-art INR methods, especially for characterizing high-frequency components (e.g., textures and edges).

Hyperspectral anomaly detection (HAD) aims to distinguish anomalies from background in hyperspectral images (HSIs). Recently, low-rank representation (LRR)-based methods have attracted widespread attention, but their pixel-level design overlooks semantic information in HSIs, resulting in unsatisfactory anomaly detection performance. To address this issue, we propose a prompted tensor decomposition (ProTD) for HAD, which cleverly leverages the power of large vision models with the designed prompt to guide the decomposition of HSIs. Specifically, guided by high-level semantic information distilled from a pretrained Segment Anything Model (SAM) with the prompt, we leverage uni-deep Tucker decomposition with Spatio-Spectral Total Variation (SSTV) regularization to represent the background, while using ℓ1norm to capture anomalies. To solve this model, we design an efficient algorithm based on alternating minimization. Extensive experiments on benchmark datasets (including Airport, Urban, and Beach) demonstrate that the proposed ProTD model outperforms state-of-the-art HAD methods, especially those LRR-based methods.

Thick cloud removal is a long-standing and critical challenge in remote sensing (RS) image processing, with the increasing availability of multi-temporal RS images offering new opportunities to address this problem. The main limitation of existing cloud removal methods is that the classical fidelity only considers original pixel domain or handcrafted/pretrained filtered domains, overlooking the individuality filters and the corresponding feature behind each RS image, which leads to evident detail discrepancies. To address this issue, we suggest a data-aware fidelity based on the untrained neural network, which encourages deep data-aware feature matching between the contaminated image and the guidance image. Complementary to the data-aware fidelity, we design the deep self-representation to implicitly impose regularization benefiting from the same untrained neural network. Equipped with the elaborately designed fidelity and regularization, we propose a unified data-aware fidelity and regularization learning (called DAFRL) paradigm for thick cloud removal that flexibly adapts to diverse multi-temporal RS images. Under this paradigm, the fidelity and regularization are empowered by the same untrained neural network, serving distinct functions while collaborating organically. Experimental results on both simulated and real datasets show that the proposed DAFRL effectively preserves fine details and outperforms the compared methods.

In recent years, tensor decomposition-based approaches forhyperspectral anomaly detection(HAD) have gained significant attention in the field of remote sensing. However, existing methods often fail to flexibly and effectively extract both the global correlations and local smoothness of the background components inhyperspectral images(HSIs). To mitigate this critical issue, we put forward a novel HAD method named HAD-EUNTRFR, which incorporates an enhanced unified nonconvex tensor ring (TR) factors regularization. In the HAD-EUNTRFR framework, the raw HSIs are first decomposed into background and anomaly components using the idea of tensor robust principal component analysis. The TR decomposition is then employed to capture the spatial-spectral correlations within the background component. Additionally, we introduce a unified and efficient nonconvex regularizer, induced bytensor singular value decomposition(T-SVD), to simultaneously encode the low-rankness and sparsity of the 3-D gradient TR factors into a unique concise form. The above characterization scheme enables the interpretable gradient TR factors to inherit the low-rankness and smoothness of the original background. To further enhance anomaly detection, we design a generalized nonconvex regularization term to exploit the group sparsity of the anomaly component. Based upon the above, we ultimately propose a scalable and reliable nonconvex HAD model. To solve the resulting doubly nonconvex model, we develop a highly efficient optimization algorithm based on thealternating direction method of multipliers(ADMM) framework. Theoretical results on convergence analysis for the proposed algorithm are derived. Experimental results on several benchmark datasets demonstrate that our proposed method outperforms existingstate-of-the-art(SOTA) approaches in terms of detection accuracy.

Hyperspectral anomaly detection is an important task in remote sensing for identifying pixels with anomalous spectral signatures that deviate from their local background. Recently, low-rank and sparse representation-based methods have garnered significant attention in hyperspectral anomaly detection, which typically employ low-rank representation to characterize the background and sparse representation to capture anomalies. Since the background and anomalies usually exhibit complex characteristics beyond the low-rankness and sparsity, low-rank and sparse representation-based methods typically do not perform satisfactorily for complex scenarios. To address the challenge, we propose an unsupervised hyperspectral anomaly detection method from a continuous perspective, which organically integrates Continuous Background representation and deep Anomaly Representation (CBAR). Specifically, the CBAR model leverages the continuous low-rank tensor function to encapsulate both the low-rankness and smoothness of the background and the deep neural network to capture the complex geometric structure of anomalies. Moreover, to mitigate the overfitting of the background and anomalies to the observed HSI, we introduce two terms as overfit-shield by exploiting the prior knowledge of the background and anomalies. To solve the CBAR model, we develop an efficient alternating minimization algorithm. Extensive experiments on benchmark datasets (including Airport, Urban, Beach, and HYDICE) demonstrate that the proposed CBAR outperforms the state-of-the-art anomaly detection methods both qualitatively and quantitatively. For reproducibility, we will release our source code at: https://github.com/Weihao-Wu/CBAR.

Tensor decompositions have been successfully applied to multidimensional data recovery. However, classical tensor decompositions are not suitable for emerging spatio-irregular multidimensional data (i.e., spatio-irregular tensor), whose spatial domain is non-rectangular, e.g., spatial transcriptomics data from bioinformatics and semantic units from computer vision. By using preprocessing (e.g., zero-padding or element-wise 0-1 weighting), the spatio-irregular tensor can be converted to a spatio-regular tensor and then classical tensor decompositions can be applied, but this strategy inevitably introduces bias information, leading to artifacts. How to design a tensor-based method suitable for emerging spatio-irregular tensors is an imperative challenge. To address this challenge, we propose a learnable transform-assisted tensor singular value decomposition (LTA-TSVD) for spatio-irregular tensor recovery, which allows us to leverage the intrinsic structure behind the spatio-irregular tensor. Specifically, we design a learnable transform to project the original spatio-irregular tensor into its latent spatio-regular tensor, and then the latent low-rank structure is captured by classical TSVD on the resulting regular tensor. Empowered by LTA-TSVD, we develop spatio-irregular low-rank tensor completion (SIR-LRTC) and spatio-irregular tensor robust principal component analysis (SIR-TRPCA) models for the spatio-irregular tensor imputation and denoising respectively, and we design corresponding solving algorithms with theoretical convergence. Extensive experiments including the spatial transcriptomics data imputation and hyperspectral image denoising show SIR-LRTC and SIR-TRPCA are superior performance to competing approaches and benefit downstream applications.

Multi-view subspace clustering (MVSC) separates the data with multiple views into multiple clusters, and each cluster corresponds to one certain subspace. Existing tensor-based MVSC methods construct self-representation subspace coefficient matrices of all views as a tensor, and introduce the tensor nuclear norm (TNN) to capture the complementary information hidden in different views. The key assumption is that the data samples of each subspace must be sufficient for subspace representation. This work proposes a nonconvex latent transformed low-rank tensor representation framework for MVSC. To deal with the insufficient sample problem, we study the latent low-rank representation in the multi-view case to supplement underlying observed samples. Moreover, we propose to use data-driven transformed TNN (TTNN), resulting from the intrinsic structure of multi-view samples, to preserve the consensus and complementary information in the transformed domain. Meanwhile, the proposed unified nonconvex low-rank tensor representation framework can better learn the high correlation among different views. To resolve the proposed nonconvex optimization model, we propose an effective algorithm under the framework of the alternating direction method of multipliers and theoretically prove that the iteration sequences converge to the critical point. Experiments on various datasets showcase outstanding performance.

Recently, the transform-based tensor representation has attracted increasing attention in multimedia data (e.g., images and videos) recovery problems, which consists of two indispensable components, i.e., the transform and the characterization. Previously, the development of transform-based tensor representation has focused mainly on the transform perspective. Although several attempts have considered shallow matrix factorization (e.g., singular value decomposition and nonnegative matrix factorization) for characterizing the frontal slices of the transformed tensor (termed the latent tensor), the faithful characterization perspective has been underexplored. To address this issue, we propose a unifiedDeepTensorRepresentation (DTR) framework by synergistically combining the deep latent generative module and the deep transform module. Especially, the deep latent generative module can faithfully generate the latent tensor as compared with shallow matrix factorization. The new DTR framework not only allows us to better understand the classical shallow representations but also leads us to explore new representations. To examine the representation capability of the proposed DTR, we consider the representative multidimensional data recovery task and suggest an unsupervised DTR-based multidimensional data recovery model. Extensive experiments demonstrate that DTR achieves superior performance compared to the state-of-the-art methods from both quantitative and qualitative aspects, especially for fine detail recovery.

Recently, the tensor nuclear norm (TNN)-based tensor robust principle component analysis (TRPCA) has achieved impressive performance in multidimensional data processing. The underlying assumption in TNN is the low-rankness of frontal slices of the tensor in the transformed domain (e.g., Fourier domain). However, the low-rankness assumption is usually violative for real-world multidimensional data (e.g., video and image) due to their intrinsically nonlinear structure. How to effectively and efficiently exploit the intrinsic structure of multidimensional data remains a challenge. In this article, we first suggest a kernelized TNN (KTNN) by leveraging the nonlinear kernel mapping in the transform domain, which faithfully captures the intrinsic structure (i.e., implicit low-rankness) of multidimensional data and is computed at a lower cost by introducing kernel trick. Armed with KTNN, we propose a tensor robust kernel PCA (TRKPCA) model for handling multidimensional data, which decomposes the observed tensor into an implicit low-rank component and a sparse component. To tackle the nonlinear and nonconvex model, we develop an efficient alternating direction method of multipliers (ADMM)-based algorithm. Extensive experiments on real-world applications collectively verify that TRKPCA achieves superiority over the state-of-the-art RPCA methods.

Dictionary learning (DL) is a pivotal task in machine learning and signal processing, involving extracting representative features from a given dataset. However, conventional DL models are known to be highly sensitive to outliers. To circumvent this issue, we introduce a new and robust DL model based on unbalanced optimal transport (UOT). Compared to DL models based on conventional robust distances and the Wasserstein distance, our model not only captures and leverages the structural information within the data but also demonstrates strong resilience to outliers. By employing the structure of the proposed robust DL model, we develop a novel hybrid block coordinate descent (BCD) algorithm. The proposed algorithm maintains computational tractability by exploiting special block structures of the subproblems. In addition, we establish the convergence of our algorithm without the Lipschitz smooth condition. Through extensive experimentation, we validate our theoretical results and demonstrate the effectiveness of the proposed method on synthetic data, MNIST data, Olivetti faces dataset, and hyperspectral images (HSIs) datasets.

Tensor network (TN) representation is a powerful technique for computer vision and machine learning. TN structure search (TN-SS) aims to search for a customized structure to achieve a compact representation, which is a chal-lenging NP-hard problem. Recent “sampling-evaluation”-based methods require sampling an extensive collection of structures and evaluating them one by one, resulting in pro-hibitively high computational costs. To address this issue, we propose a novel TN paradigm, named SVD-inspired TN decomposition (SVDinsTN), which allows us to efficiently solve the TN-SS problem from a regularized modeling per-spective, eliminating the repeated structure evaluations. To be specific, by inserting a diagonal factor for each edge of the fully-connected TN, SVDinsTN allows us to calculate TN cores and diagonal factors simultaneously, with the factor sparsity revealing a compact TN structure. In theory, we prove a convergence guarantee for the proposed method. Experimental results demonstrate that the proposed method achieves approximately 100 ~ 1000 times acceleration compared to the state-of-the-art TN-SS methods while maintaining a comparable level of representation ability.

This paper proposes a bi-level tensor decomposition (BLTD), properly exploiting the characterization advantages of tensor subspace representation (TSR) and tensor ring decomposition (TRD). More specifically, the first level is related to the decomposition of a third-order tensor into its TSR using the tensor-tensor product (t-product), and the second level performs TRD on the coefficient tensor obtained by TSR. Leveraging the proposed BLTD, we design a bi-level tensor nuclear norm (BLTNN)-based model for hyperspectral images (HSIs) denoising. To solve the model, we develop an efficient alternating direction method of multipliers (ADMM)-based algorithm. Experimental results demonstrate the superior performance of our method compared to existing methods.

The pixel intensity of remote sensing images at different time nodes exhibits significant differences due to factors such as changes in solar illumination angles. Consequently, the previous cloud removal methods, primarily based on the original pixel domain, yield unsatisfactory results. In this paper, considering the sharing of similar features among remote sensing images at different time nodes, we first design a novel feature-domain fidelity that leverages the feature extraction capability of convolution operator, allowing for the precise preservation of intricate details and textures inherent in multi-temporal remote sensing images. Building upon the feature-domain fidelity, we propose a cloud removal model that organically integrates the low fully-connected tensor network rank regularization, which comprehensively captures the spatial-spectral-temporal correlations of multi-temporal remote sensing images. Moreover, we develop an effective algorithm based on proximal alternating minimization to solve the proposed model. Numerical experiments conducted on both simulated and real-world data validate that the proposed method outperforms the compared ones.

Since the information across all bands of the cloud-contaminated region is missing, thick cloud removal for remote sensing images (RSIs) is still a challenging problem. Recently, the availability of rich spatial–spectral–temporal information for multitemporal RSIs provides the possibility for addressing the thick cloud removal problem. However, existing methods explore the holistic redundancy of multitemporal RSIs and neglect the important semantic clue of multitemporal images. In this letter, we propose a superpixel-oriented thick cloud removal (STORM) model for multitemporal images, where the multitemporal superpixel as the generic unit allows us to exploit redundancy with semantic clue in a low-rank optimization problem. To harness the resultant irregular fourth-order tensor (i.e., multitemporal superpixels) in the optimization problem, we cleverly introduce the weighted tensor to transform the irregular tensor into the regular tensor, which naturally leads to a standard low-rank tensor optimization problem. To tackle the tensor optimization problem, we develop a proximal alternating minimization (PAM)-based algorithm. Extensive simulated and real experiments on multitemporal RSIs acquired by Sentinel-2 and Landsat-8 satellites demonstrate the superior performance of the proposed method over the comparison methods.

Most existing learning-based deraining methods are supervisedly trained on synthetic rainy-clean pairs. The domain gap between the synthetic and real rain makes them less generalized to complex real rainy scenes. Moreover, the existing methods mainly utilize the property of the image or rain layers independently, while few of them have considered their mutually exclusive relationship. To solve above dilemma, we explore the intrinsic intra-similarity within each layer and inter-exclusiveness between two layers and propose an unsupervised non-local contrastive learning (NLCL) deraining method. The non-local self-similarity image patches as the positives are tightly pulled together and rain patches as the negatives are remarkably pushed away, and vice versa. On one hand, the intrinsic self-similarity knowledge within positive/negative samples of each layer benefits us to discover more compact representation; on the other hand, the mutually exclusive property between the two layers enriches the discriminative decomposition. Thus, the internal self-similarity within each layer (similarity) and the external exclusive relationship of the two layers (dissimilarity) serving as a generic image prior jointly facilitate us to unsupervisedly differentiate the rain from clean image. We further discover that the intrinsic dimension of the non-local image patches is generally higher than that of the rain patches. This insight motivates us to design an asymmetric contrastive loss that precisely models the compactness discrepancy of the two layers, thereby improving the discriminative decomposition. In addition, recognizing the limited quality of existing real rain datasets, which are often small-scale or obtained from the internet, we collect a large-scale real dataset under various rainy weathers that contains high-resolution rainy images. Extensive experiments conducted on different real rainy datasets demonstrate that the proposed method obtains state-of-the-art performance in real deraining.

Since higher-order tensors are naturally suitable for representing multi-dimensional data in real-world, e.g., color images and videos, low-rank tensor representation has become one of the emerging areas in machine learning and computer vision. However, classical low-rank tensor representations can solely represent multi-dimensional discrete data on meshgrid, which hinders their potential applicability in many scenarios beyond meshgrid. To break this barrier, we propose a low-rank tensor function representation (LRTFR) parameterized by multilayer perceptrons (MLPs), which can continuously represent data beyond meshgrid with powerful representation abilities. Specifically, the suggested tensor function, which maps an arbitrary coordinate to the corresponding value, can continuously represent data in an infinite real space. Parallel to discrete tensors, we develop two fundamental concepts for tensor functions, i.e., the tensor function rank and low-rank tensor function factorization, and utilize MLPs to paramterize factor functions of the tensor function factorization. We theoretically justify that both low-rank and smooth regularizations are harmoniously unified in LRTFR, which leads to high effectiveness and efficiency for data continuous representation. Extensive multi-dimensional data recovery applications arising from image processing (image inpainting and denoising), machine learning (hyperparameter optimization), and computer graphics (point cloud upsampling) substantiate the superiority and versatility of our method as compared with state-of-the-art methods. Especially, the experiments beyond the original meshgrid resolution (hyperparameter optimization) or even beyond meshgrid (point cloud upsampling) validate the favorable performances of our method for continuous representation.

Multi-dimensional data are inevitably corrupted, which hinders subsequent applications (e.g., image segmentation and classification). Recently, due to the powerful ability to characterize the correlation between any two modes of tensors, fully-connected tensor network (FCTN) decomposition has received increasing attention in multi-dimensional data recovery. However, the expressive power of FCTN decomposition in the original pixel domain has yet to be fully leveraged, which can not provide satisfactory results in the recovery of details and textures, especially for low-sampling rates or heavy noise scenarios. In this work, we suggest a feature-based FCTN decomposition model (termed as F-FCTN) for multi-dimensional data recovery, which can faithfully capture the relationship between the spatial-temporal/spectral-feature modes. Compared with the original FCTN decomposition, F-FCTN can more effectively recover the details and textures and be more suitable for the subsequent high-level applications. However, F-FCTN leads to a larger-scale feature tensor as compared with the original tensor, which brings challenges in designing the solving algorithm. To harness the resulting large-scale optimization problem, we develop an efficient leverage score sampling-based proximal alternating minimization (S-PAM) algorithm and theoretically establish its relative error guarantee. Extensive numerical experiments on real-world data illustrate that the proposed method performs favorably against compared methods in data recovery and facilitates subsequent image classification.

Recently, fully-connected tensor network (FCTN) decomposition, which factorizes the target tensor into a series of interconnected factor tensors, has drawn growing focus on multi-dimensional visual data processing. However, the lack of clear physical interpretation for the factor tensors hinders us from introducing handcrafted regularizers to deeply explore the potential of FCTN decomposition. To tackle this issue, we suggest a unimode hierarchical nonlinear (UHN) decomposition for each factor tensor, which can adaptively capture the complex nonlinear structure and implicitly regularize factor tensors. With this UHN decomposition of the factor tensors, we naturally propose a nested fully-connected tensor network (N-FCTN) decomposition. Attributed to the adaptive and implicit regularization inherent in UHN decomposition of factor tensors, the proposed N-FCTN decomposition is expected to perform favorably against the original FCTN decomposition. Based on the proposed N-FCTN decomposition, we build a multi-dimensional visual data recovery model and provide a theoretical error bound between the recovered tensor by our model and the underlying tensor. To address the resulting non-convex and nonlinear optimization problem, we develop an efficient proximal alternating minimization (PAM)-based algorithm and establish its theoretical convergence guarantee. Extensive experimental results on multi-spectral images, color videos, and light field data demonstrate the superior recovery performance of the proposed method compared to the state-of-the-art methods.

Recently, transform-based tensor nuclear norm (TNN) methods have received increasing attention as a powerful tool for multi-dimensional visual data (color images, videos, and multispectral images, etc.) recovery. Especially, the redundant transform-based TNN achieves satisfactory recovery results, where the redundant transform along spectral mode can remarkably enhance the low-rankness of tensors. However, it suffers from expensive computational cost induced by the redundant transform. In this paper, we propose a learnable spatial-spectral transform-based TNN model for multi-dimensional visual data recovery, which not only enjoys better low-rankness capability but also allows us to design fast algorithms accompanying it. More specifically, we first project the large-scale original tensor to the small-scale intrinsic tensor via the learnable semi-orthogonal transforms along the spatial modes. Here, the semi-orthogonal transforms, serving as the key building block, can boost the spatial low-rankness and lead to a small-scale problem, which paves the way for designing fast algorithms. Secondly, to further boost the low-rankness, we apply the learnable redundant transform along the spectral mode to the small-scale intrinsic tensor. To tackle the proposed model, we apply an efficient proximal alternating minimization-based algorithm, which enjoys a theoretical convergence guarantee. Extensive experimental results on real-world data (color images, videos, and multispectral images) demonstrate that the proposed method outperforms state-of-the-art competitors in terms of evaluation metrics and running time.

Denoising of hyperspectral image (HSI) is challenging, especially when dealing with large-scale data. Model-based methods show promise in HSI denoising due to their good generalization, but they suffer from computational complexity due to complex priors [like nonlocal self-similarity (NSS)] and iterations, resulting in low efficiency for large-scale HSI processing. To address these challenges, we propose a fast large-scale HSI denoising (FallHyDe) method based on noniterative low-rank (LR) subspace representation to enjoy high denoising efficiency, effectiveness, and flexibility simultaneously. By leveraging the global spectral property of HSI, FallHyDe efficiently estimates spectral subspace and spatial representation coefficients (SRCs) from the observed noisy HSI, reducing computation complexity caused by the high spectral dimension during processing. In addition, we innovatively explore the presence of high signal-to-noise ratio bands (HSNRBs) in real HSI, enabling fast SRC estimation through a least squares problem without relying on complex priors and iterations. FallHyDe requires neither iteration nor parameter tuning, enabling our method to process large-scale HSI denoising quickly and flexibly. Experimental results on both simulated and real HSI datasets demonstrate that our proposed method not only achieves competitive results in quality but also speeds up the restoration by more than ten times than the representative fast HSI denoising methods. The code is available athttps://chenyong1993.github.io/yongchen.github.io/.

Transformer holds significance in deep learning (DL) research. Node embedding (NE) and positional encoding (PE) are usually two indispensable components in a Transformer. The former can excavate hidden correlations from the data, while the latter can store locational relationships between nodes. Recently, the Transformer has been applied for hyperspectral image (HSI) classification because the model can capture long-range dependencies to aggregate global features for representation learning. In an HSI, adjacent pixels tend to be homogeneous, while the NE does not identify the positional information of pixels. Therefore, PE is crucial for Transformers to understand locational relationships between pixels. However, in this area, most Transformer-based methods randomly generate PEs without considering their physical meaning, which leads to weak representations. This article proposes a new graph generative structure-aware Transformer (GraphGST) to solve the above-mentioned PE problem when implementing HSI classification. In our GraphGST, a new absolute PE (APE) is established to acquire pixels’ absolute positional sequences (APSs) and is integrated into the Transformer architecture. Moreover, a generative mechanism with self-supervised learning is developed to achieve cross-view contrastive learning (CL), aiming to enhance the representation learning of the Transformer. The proposed GraphGST model can capture local-to-global correlations, and the extracted APSs can complement the spectral features of pixels to assist in NE. Several experiments with real HSIs are conducted to evaluate the effectiveness of our GraphGST. The proposed method demonstrates very competitive performance compared with other state-of-the-art (SOTA) approaches. Our source codes will be provided in the following linkhttps://github.com/yuanchaosu/TGRS-graphGST.

Thick cloud contamination in remote sensing (RS) images significantly hinders their utility in subsequent applications. Traditional cloud removal methods predominantly focus on the design of the regularization term, while neglecting the design of the fidelity term. Recently, the proposal of gradient-domain fidelity has highlighted the significance of the fidelity term design, dedicated to maintaining textures in the gradient domain. However, the handcrafted gradient-domain fidelity still has limitations in capturing invariant and delicate features behind multitemporal RS (MTRS) images, leading to unsatisfactory detail preservation. To address the problem, we suggest a deep domain fidelity to capture deep features by leveraging a pretrained deep network, which matches the intrinsic deep feature between the original images and reference images, instead of matching the shallow features in gradient domain fidelity. Empowered with the deep domain fidelity, we propose a thick cloud removal model (called DFTR) for MTRS images, organically integrating the deep domain fidelity term with a low-rank (LR) regularization term (i.e., tensor ring (TR) decomposition), offering fine detail preservation. Extensive simulated and real experiments on MTRS images demonstrate that the proposed method outperforms the compared methods, including the origin domain-based and the gradient domain-based methods, in thick cloud removal, especially for detail preservation.

Anomaly detection in hyperspectral images (HSIs) aims to identify sparse, interesting anomalies against the background, which has become a significant topic in remote sensing. Although the existing tensor-based methods have achieved commendable performance to some extent, there is still room for further improvement. In combination with three key techniques, i.e., gradient map-based modeling, circular tensor ring (TR) unfolding, and nonconvex regularization, this article proposes a novel generalized nonconvex method for hyperspectral anomaly detection (HAD) tasks within the TR framework. For the implementation of our proposed approach, abbreviated as TR-GNHAD, we first develop an effective and reliable HAD model in virtue of two newly unified nonconvex regularizers. The first regularizer is devised under a new prior characterization paradigm, which has a strong ability to encode two insightful prior information underlying the HSI’s background simultaneously, i.e., global low rankness and local smoothness. The other regularizer can well capture the structured sparsity of the abnormal component. Then, we derive an efficient optimization algorithm to solve the proposed model based on the alternating direction method of multipliers (ADMMs) framework. Experiments conducted on 12 HSI datasets illustrate that the proposed approach achieves highly competitive performance in both qualitative and quantitative metrics compared with several state-of-the-art HAD methods.

Coded aperture snapshot spectral imaging (CASSI) is an important technique for capturing three-dimensional (3D) hyperspectral images (HSIs), and involves an inverse problem of reconstructing the 3D HSI from its corresponding coded 2D measurements. Existing model-based and learning-based methods either could not explore the implicit feature of different HSIs or require a large amount of paired data for training, resulting in low reconstruction accuracy or poor generalization performance as well as interpretability. To remedy these deficiencies, this paper proposes a novel HSI reconstruction method, which exploits the global spectral correlation from the HSI itself through a formulation of model-driven low-rank subspace representation and learns the deep prior by a data-driven self-supervised deep learning scheme. Specifically, we firstly develop a model-driven low-rank subspace representation to decompose the HSI as the product of an orthogonal basis and a spatial representation coefficient, then propose a data-driven deep guided spatial-attention network (called DGSAN) to adaptively reconstruct the implicit spatial feature of HSI by learning the deep coefficient prior (DCP), and finally embed these implicit priors into an iterative optimization framework through a self-supervised training way without requiring any training data. Thus, the proposed method shall enhance the reconstruction accuracy, generalization ability, and interpretability. Extensive experiments on several datasets and imaging systems validate the superiority of our method. The source code and data of this article will be made publicly available at https://github.com/ChenYong1993/LRSDN.

Recently, low-rank tensor completion (LRTC) methods by exploiting the global low-rankness of the target tensor have shown great potential for traffic data imputation. However, in real-world transportation networks, traffic data usually suffer from more complicated structural missing patterns than random-missing patterns, e.g., tube-missing patterns due to disruptions in wireless connections or slice-missing mechanism caused by sensor maintenance. As the naturally low-rank structure of traffic data in several missing scenarios, the existing LRTC methods indeed refrain from desirable performance for imputing traffic data. To tackle the complicated missing scenarios, we propose a convolutional low-rank tensor representation (CLRTR). Especially, CLRTR represents each unfolding matrix of the tensor as a sum of convolutions between two-dimensional (2D) filters and the corresponding low-rank coefficients, which allows us to simultaneously reveal the local patterns and the low-rankness of traffic data. Based on the CLRTR, we introduce the corresponding low-rank metric CLRTR-rank. Based on the suggested low-rank metric, we propose a traffic data imputation model that is well-suited to the complicated missing data scenarios. To implement the resultant imputation model, we design the alternating direction method of multipliers (ADMM) based algorithm with a theoretical convergence guarantee. Extensive numerical experiments on several real-world traffic datasets for both traffic data imputation and downstream traffic data prediction highlight the superiority of our model over the existing state-of-the-art matrix/tensor models for extensive missing scenarios.

Snapshot compressive imaging (SCI) is a promising technique that captures a 3-D hyperspectral image (HSI) by a 2-D detector in a compressed manner. The ill-posed inverse process of reconstructing the HSI from their corresponding 2-D measurements is challenging. However, current approaches either neglect the underlying characteristics, such as high spectral correlation, or demand abundant training datasets, resulting in an inadequate balance among performance, generalizability, and interpretability. To address these challenges, in this article, we propose a novel approach called LR2DP that integrates the model-driven low-rank prior and data-driven deep priors for SCI reconstruction. This approach not only captures the spectral correlation and deep spatial features of HSI but also takes advantage of both model-based and learning-based methods without requiring any extra training datasets. Specifically, to preserve the strong spectral correlation of the HSI effectively, we propose that the HSI lies in a low-rank subspace, thereby transforming the problem of reconstructing the HSI into estimating the spectral basis and spatial representation coefficient. Inspired by the mutual promotion of unsupervised deep image prior (DIP) and trained deep denoising prior (DDP), we integrate the unsupervised network and pre-trained deep denoiser into the plug-and-play (PnP) regime to estimate the representation coefficient together, aiming to explore the internal target image prior (learned by DIP) and the external training image prior (depicted by pre-trained DDP) of the HSI. An effective half-quadratic splitting (HQS) technique is employed to optimize the proposed HSI reconstruction model. Extensive experiments on both simulated and real datasets demonstrate the superiority of the proposed method over the state-of-the-art approaches.

Recently, the transform-based tensor nuclear norm (TNN) methods have shown promising performance and drawn increasing attention in tensor completion (TC) problems. The main idea of these methods is to exploit the low-rank structure of frontal slices of the tensor under the transform. However, the transforms in TNN methods usually treat all modes equally and do not consider the different traits of different modes (i.e., spatial and spectral/temporal modes). To address this problem, we suggest a new low-rank tensor representation based on the coupled nonlinear transform (called CoNoT) for a better low-rank approximation. Concretely, spatial and spectral/temporal transforms in the CoNoT, respectively, exploit the different traits of different modes and are coupled together to boost the implicit low-rank structure. Here, we use the convolutional neural network (CNN) as the CoNoT, which can be learned solely from an observed multidimensional image in an unsupervised manner. Based on this low-rank tensor representation, we build a new multidimensional image completion model. Moreover, we also propose an enhanced version (called Ms-CoNoT) to further exploit the spatial multiscale nature of real-world data. Extensive experiments on real-world data substantiate the superiority of the proposed models against many state-of-the-art methods both qualitatively and quantitatively.

Non-blind deblurring methods achieve decent performance under the accurate blur kernel assumption. Since the kernel uncertainty (i.e. kernel error) is inevitable in practice, semi-blind deblurring is suggested to handle it by introducing the prior of the kernel (or induced) error. However, how to design a suitable prior for the kernel (or induced) error remains challenging. Hand-crafted prior, incorporating domain knowledge, generally performs well but may lead to poor performance when kernel (or induced) error is complex. Data-driven prior, which excessively depends on the diversity and abundance of training data, is vulnerable to out-of-distribution blurs and images. To address this challenge, we suggest a dataset-free deep residual prior for the kernel induced error (termed as residual) expressed by a customized untrained deep neural network, which allows us to flexibly adapt to different blurs and images in real scenarios. By organically integrating the respective strengths of deep priors and hand-crafted priors, we propose an unsupervised semi-blind deblurring model which recovers the clear image from the blurry image and inaccurate blur kernel. To tackle the formulated model, an efficient alternating minimization algorithm is developed. Extensive experiments demonstrate the favorable performance of the proposed method as compared to model-driven and data-driven methods in terms of image quality and the robustness to different types of kernel error.

Audio-visual zero-shot learning (ZSL) has attracted board attention, as it could classify video data from classes that are not observed during training. However, most of the existing methods are restricted to background scene bias and fewer motion details by employing a single-stream network to process scenes and motion information as a unified entity. In this paper, we address this challenge by proposing a novel dual-stream architecture Motion-Decoupled Spiking Transformer (MDFT) to explicitly decouple the contextual semantic information and highly sparsity dynamic motion information. Specifically, The Recurrent Joint Learning Unit (RJLU) could extract contextual semantic information effectively and understand the environment in which actions occur by capturing joint knowledge between different modalities. By converting RGB images to events, our approach effectively captures motion information while mitigating the influence of background scene biases, leading to more accurate classification results. We utilize the inherent strengths of Spiking Neural Networks (SNNs) to process highly sparsity event data efficiently. Additionally, we introduce a Discrepancy Analysis Block (DAB) to model the audio motion features. To enhance the efficiency of SNNs in extracting dynamic temporal and motion information, we dynamically adjust the threshold of Leaky Integrate-and-Fire (LIF) neurons based on the statistical cues of global motion and contextual semantic information. Our experiments demonstrate the effectiveness of MDFT, which consistently outperforms state-of-the-art methods across mainstream benchmarks. Moreover, we find that motion information serves as a powerful regularization for video networks, where using it improves the accuracy of HM and ZSL by 19.1% and 38.4%, respectively.

Recently, tensor singular value decomposition (t-SVD) has emerged as a promising tool for hyperspectral image (HSI) processing. In the t-SVD, there are two key building blocks: (i) the low-rank enhanced transform and (ii) the accompanying low-rank characterization of transformed frontal slices. Previous t-SVD methods mainly focus on the developments of (i), while neglecting the other important aspect, i.e., the exact characterization of transformed frontal slices. In this letter, we exploit the potentiality in both building blocks by leveraging the Hierarchical nonlinear transform and the Hierarchical matrix factorization to establish a new Tensor Factorization (termed as H2TF). Compared to shallow counter partners, e.g., low-rank matrix factorization or its convex surrogates, H2TF can better capture complex structures of transformed frontal slices due to its hierarchical modeling abilities. We then suggest the H2TF-based HSI denoising model and develop an alternating direction method of multipliers-based algorithm to address the resultant model. Extensive experiments validate the superiority of our method over state-of-the-art HSI denoising methods.

Recently, due to the ability of exploiting perceptual information, superpixel-based methods have received attention for hyperspectral image (HSI) denoising. However, existing superpixel-based denoising methods unfold the irregular 3D superpixels into the matrices along the spectral mode, which inevitably destroys the intrinsic structure of the irregular 3D superpixels. To tackle the irregular 3D superpixels, we introduce the irregular tensor representation for superpixels-guided HSI denoising. More concretely, by introducing the weighted tensor, we suggest a tensor representation of each irregular 3D superpixel, which can preserve the intrinsic structure of the irregular 3D superpixel. Equipped with the irregular tensor representation, we establish a superpixel-guided tensor optimization model for HSI denoising, which simultaneously exploits the perceptual information and low-rank structure within 3D superpixels. To solve the resulting tensor optimization problem, we develop an inexact augmented Lagrange multiplier (IALM) algorithm. Experimental results show that the proposed method outperforms other state-of-the-art HSI denoising methods, particularly matrix-based methods, on both simulated and real data.

Recently, block-term decomposition with rank-(Lr,Lr,1) (termed as LL1 decomposition), which is physically inspired by linear spectral unmixing, has received increasing attention in hyperspectral images (HSIs) denoising. However, due to the intrinsic nonlinear structure of real-world HSIs, the low-rankness of HSIs is usually implicit. Moreover, the essential uniqueness guarantee is usually violated with the low-rank assumption of the abundance maps unsupported in real scenarios, which hampers the successful deployment of LL1 decomposition. Inspired by the nonlinear spectral unmixing, we propose a nonlinear learnable transform-based LL1 decomposition (NT-LL1) for characterizing the implicit low-rank structure of real-world HSIs. More concretely, the nonlinear learnable transform in NT-LL1 decomposition is a composed transform consisting of a linear semi-orthogonal transform and a component-wise nonlinear transform, which collaboratively enhances the low-rankness of the abundance maps. Empowering with the NT-LL1 decomposition, we propose an NT-LL1 decomposition-based model for HSIs denoising. To tackle the resulting model, we develop an efficient proximal alternating minimization-based algorithm with a convergence guarantee. Extensive experimental results including simulated and real data collectively verify the superiority of the proposed method as compared with the competing methods.

Abstract. The transform-based tensor nuclear norm (TNN) methods have shown good recovery results for tensor completion. However, the TNN methods are based on the single-tube transforms in which transforms are applied to each tube independently. The performance of the single-tube transform-based TNN methods is not good for recovery of missing tubes in multidimensional images (e.g., all the observations are missing in a pixel location of multispectral images). The main aim of this paper is to address this issue by proposing and developing a learnable group-tube transform-based TNN (GTNN) method that can effectively explore the correlation of neighboring tubes by leveraging a learnable group-tube transform. The proposed learnable group-tube transform is a separable three-dimensional transform that consists of a one-dimensional spectral/temporal transform (i.e., single-tube transform) and a two-dimensional spatial transform. Such group-tube transform can effectively explore the correlation of neighboring tubes. Based on the elaborately designed low-rank metric GTNN, we suggest a low-rank tensor completion model. To solve this highly nonconvex model, we design an efficient multiblock proximal alternating minimization algorithm and establish the convergence guarantee. A variety of numerical experiments on real-world multidimensional imaging data including traffic speed data, color images, videos, and multispectral images collectively manifest that the GTNN method outperforms some state-of-the-art TNN methods especially when the observations along tubes are missing.

Recently, untrained neural network priors (UNNPs) have received increasing attention for multi-dimensional image recovery. However, previous studies are based on over-parameterized untrained neural networks, which results in unstable behavior. In this letter, we propose an untrained low-rank neural network prior (ULRNNP) for multi-dimensional image recovery, which enjoys the powerful representation ability and stable behavior. More specifically, the elaborately designed nonlinear Tucker decomposition module implicitly imposes low-rank constraints on the feature tensor and can more compactly represent the feature tensor. Attributed to the suggested nonlinear Tucker decomposition module, ULRNNP can simultaneously enjoy strong representation ability and stable behavior. The friendly stable behavior allows us to design a friendly stopping criteria without the reference ground truth image as compared with classic UNNP-based methods. Extensive experiments on different multi-dimensional image datasets validate the superior performance of the proposed ULRNNP over state-of-the-art methods.

Filter pruning is one of the most popular approaches for compressing convolutional neural networks (CNNs). The most critical task in pruning is to evaluate the importance of each convolutional filter, such that the less important filters can be removed while the overall model performance is minimally affected. In each layer, some filters may be linearly dependent on each other, which means that they have replaceable information. Redundant information can be removed without significantly affecting information richness and model performance. In this paper, we propose a novel low-rank guided pruning scheme to obtain skeleton neural networks by alternatively training and pruning CNNs. In each step, training is performed with nuclear-norm regularization to low-rank the filters in each layer, followed by filter pruning to maintain the information richness via the maximally linearly independent subsystem. A novel “smaller-norm-and-linearly-dependent-less-important” pruning criterion is proposed to compress the model. The training and pruning processes can be repeated until the model is fully trained. To investigate the performance, we applied the proposed joint training and pruning scheme to train the CNNs for image classification. We considered three benchmark datasets: MNIST, CIFAR-10 and ILSVRC-2012. The proposed method successfully achieved a higher pruning rate and better classification performance compared to state-of-the-art compression methods.

Thick clouds in remote sensing (RS) images deteriorate the visual quality and hinder subsequent applications. The emerging multitemporal RS images with rich temporal information bring the opportunity for cloud removal. How to effectively exploit the rich temporal information of the multitemporal RS images remains challenging. As multitemporal RS images with the same geographic scene, the spatial gradient of RS images at different time nodes has a resemblance, which can guide the reconstruction of the cloudy region. Motivated by this, we suggest a gradient domain fidelity with respect to the guided gradient for thick cloud removal in multitemporal RS images, which faithfully preserves the fine edges and textures compared to the original pixel domain fidelity. Armed with the gradient domain fidelity, we propose a low-rank tensor ring decomposition model (TRGFid) for the thick cloud removal problem. In the proposed model, the guided gradient of the cloudy region is availably estimated by using the regression method from the cloud-free region of different time nodes. Moreover, we develop an efficient proximal alternating minimization-based algorithm for solving the proposed nonconvex model. Extensive simulated and real experiments show that the proposed method outperforms its competitors, and preserves fine edges and textures.

Deep learning models, such as convolutional neural networks (CNNs), have made significant progress in hyperspectral image (HSI) classification. However, these models require a large number of parameters, which occupy a lot of storage space and suffer from overfitting, thus resulting in performance loss. To solve the above problems, in this article, we propose a new compression network [namely, a Hybrid Fully Connected Tensorized Compression Network (HybridFCTCN)] by considering the high dimensionality of HSI data. First, using the low-rank fully connected tensor network decomposition (FCTND), three novel units, i.e., FCTN-FC, FCTNConv2D, and FCTNConv3D, are designed to compress the weight tensor of standard fully connected (FC) layer and kernel tensor of convolutional layer, reducing their parameters. In the novel units, the intrinsic correlation of the decomposed factors is adequately exploited by the FC structures, which enhances their feature extraction and classification abilities. Then, benefiting from the hybrid network backbone composed of the FCTNConv3D and FCTNConv2D units, HybridFCTCN can extract more discriminative features with fewer parameters, while it has great generalization capability and robustness, enabling better HSI classification. Finally, the rank of above-designed units is defined, and its determination is discussed to facilitate the application of the proposed model. Extensive experiments on three widely used HSI datasets reveal that the proposed model achieves state-of-the-art classification performance for different training sample sizes with a very small number of parameters.

Cloud removal is an important task in the remotely sensed images (RSIs) processing, which is beneficial for downstream applications, such as unmixing, fusion, and target detection. Multi-temporal remotely sensed images (MRSIs), which contains the abundant spatial-spectral-temporal (SST) information, potentially bring the new opportunities for cloud removal. However, how to effectively and efficiently explore the rich information of MRSIs remains a challenge. Inspired by the low-rankness of MRSIs, we propose an Unsupervised Domain Factorization Network (UnDFN) for thick cloud removal, which allows us to effectively and efficiently exploit the rich SST information of MRSIs. In UnDFN framework, we first factorize RSI for each time node of MRSIs into its corresponding spatial factor and spectral factor. Due to the powerful expressive ability, the untrained neural networks are leveraged to faithfully capture the spatial and spectral factors. Especially, motivated by the low-rankness of the concatenated spatial factors of all time nodes, a low-rank spatial factor module is elaborately designed to effectively and efficiently capture the spatial factors of all time nodes as compared with separately using networks to capture spatial factors for each time node. Extensive experiments on simulated and real MRSIs of different satellites (including Sentinel-2 and Landsat-8) substantiate that the proposed UnDFN achieves state-of-the-art performance in thick cloud removal compared to other methods.

Recently, the transform-based tensor nuclear norm (TNN) framework has yielded promising results for hyperspectral image (HSI) denoising as compared with previous original-domain tensor-based models. However, the TNN framework only exploits the low-rankness of each band of HSIs (tensors) under a single spectral transform. The correlation between all bands under the transform (i.e., the global low-rankness of the transformed tensor) and the sparsity of the transformed HSI, which are beneficial for HSI denoising, is usually neglected in the TNN framework. In this article, we propose to reconcile sparse and low-tensor-ring (TR)-rank priors in the learned transformed domain (called T-RSTR model) for HSI denoising. In T-RSTR, the transform-based low-TR-rank and sparse regularizers are designed to characterize the global low-rankness and sparsity of the transformed tensors, respectively, and then the transform-based low-TR-rank and sparse regularizers are organically integrated and benefit from each other for substantially boosting denoising performance. To tackle the T-RSTR model, we elaborately design a proximal alternating minimization-based algorithm with the theoretical convergence. Extensive numerical results demonstrate that T-RSTR is superior to the competing methods.

In this article, we propose a novel tensor learning and coding model for third-order data completion. The aim of our model is to learn a data-adaptive dictionary from given observations and determine the coding coefficients of third-order tensor tubes. In the completion process, we minimize the low-rankness of each tensor slice containing the coding coefficients. By comparison with the traditional predefined transform basis, the advantages of the proposed model are that: 1) the dictionary can be learned based on the given data observations so that the basis can be more adaptively and accurately constructed and 2) the low-rankness of the coding coefficients can allow the linear combination of dictionary features more effectively. Also we develop a multiblock proximal alternating minimization algorithm for solving such tensor learning and coding model and show that the sequence generated by the algorithm can globally converge to a critical point. Extensive experimental results for real datasets such as videos, hyperspectral images, and traffic data are reported to demonstrate these advantages and show that the performance of the proposed tensor learning and coding method is significantly better than the other tensor completion methods in terms of several evaluation metrics.

Inverse problems in multi-dimensional imaging, e.g., completion, denoising, and compressive sensing, are challenging owing to the big volume of the data and the inherent illposedness. To tackle these issues, this work unsuper-visedly learns a hierarchical low-rank tensor factorization (HLRTF) by solely using an observed multi-dimensional image. Specifically, we embed a deep neural network (DNN) into the tensor singular value decompositionframe-work and develop the HLRTF, which captures the underlying low-rank structures of multi-dimensional images with compact representation abilities. This DNN herein serves as a nonlinear transform from a vector to another to help obtain a better low-rank representation. Our HLRTF infers the parameters of the DNN and the underlying low-rank structure of the original data from its observation via the gradient descent using a non-reference loss function in an unsupervised manner. To address the vanishing gradient in extreme scenarios, e.g., structural missing pixels, we introduce a parametric total variation regularization to constrain the DNN parameters and the tensor factor parameters with theoretical analysis. We apply our HLRTF for typical inverse problems in multi-dimensional imaging including completion, denoising, and snapshot spectral imaging, which demonstrates its generality and wide applicability. Extensive results illustrate the superiority of our method as compared with state-of-the-art methods.

Image deraining is a typical low-level image restoration task, which aims at decomposing the rainy image into two distinguishable layers: clean image layer and rain layer. Most of the existing learning-based deraining methods are supervisedly trained on synthetic rainy-clean pairs. The domain gap between the synthetic and real rains makes them less generalized to different real rainy scenes. Moreover, the existing methods mainly utilize the property of the two layers independently, while few of them have considered the mutually exclusive relationship between the two layers. In this work, we propose a novel non-local contrastive learning (NLCL) method for unsupervised image deraining. Consequently, we not only utilize the intrinsic self-similarity property within samples, but also the mutually exclusive property between the two layers, so as to better differ the rain layer from the clean image. Specifically, the non-local self-similarity image layer patches as the positives are pulled together and similar rain layer patches as the negatives are pushed away. Thus the similar positive/negative samples that are close in the original space benefit us to enrich more discriminative representation. Apart from the self-similarity sampling strategy, we analyze how to choose an appropriate feature encoder in NLCL. Extensive experiments on different real rainy datasets demonstrate that the proposed method obtains state-of-the-art performance in real deraining.

Tensor completion aims at estimating missing values from an incomplete observation, playing a fundamental role for many applications. This work proposes a novel low-rank tensor completion model, in which the inherent low-rank prior and external degradation accordant data-driven prior are simultaneously utilized. Specifically, the tensor nuclear norm (TNN) is adopted to characterize the overall low-dimensionality of the tensor data. Meanwhile, an implicit regularizer is formulated and its related subproblem is solved via a deep convolutional neural network (CNN) under the plug-and-play framework. This CNN, pretrained for the inpainting task on a mass of natural images, is expected to express the external data-driven prior and this plugged inpainter is consistent with the original degradation process. Then, an efficient alternating direction method of multipliers (ADMM) is designed to solve the proposed optimization model. Extensive experiments are conducted on different types of tensor imaging data with the comparison with state-of-the-art methods, illustrating the effectiveness and the remarkable generalization ability of our method.

Image deraining is an important task for subsequent multimedia applications in rainy weather. Traditional deep learning-based methods rely on the quantity and diversity of training data, which is hard to cover all complex real-world rain scenarios. In this work, we propose the first Unsupervised Controllable Network (UConNet) to flexibly tackle different rain scenarios by adaptively controlling the network at the inference stage. Specifically, our unsupervised network takes the physics-based regularizations as the unsupervised loss function. Then, we sensibly derive the relationship between trade-off parameters of the loss function and the weightings of feature maps. Based on this relationship, our learned UConNet can be flexibly customized for different rain scenarios by controlling the weightings of feature maps at the inference stage. Alternatively, these weightings can also be efficiently determined by a learned weightings recommendation network. Extensive experiments for image and video deraining show that our method achieves very promising effectiveness, efficiency, and generalization abilities as compared with state-of-the-art methods.

Multispectral image (MSI) inpainting plays an important role in real applications. Recently, fully connected tensor network (FCTN) decomposition has been shown the remarkable ability to fully characterize global correlation. Considering global correlation and nonlocal self-similarity (NSS) of MSIs, this letter introduces FCTN decomposition to the whole MSI and its NSS groups and proposes a novel nonlocal patch-based FCTN (NL-FCTN) decomposition for MSI inpainting. More specially, the NL-FCTN decomposition-based method, which increases tensor order by stacking similar small-sized patches to NSS groups, cleverly leverages the remarkable ability of FCTN decomposition to deal with higher-order tensors. Besides, we propose an efficient proximal alternating minimization (PAM)-based algorithm to solve the proposed NL-FCTN decomposition-based model with a theoretical convergence guarantee. Extensive experiments on MSIs demonstrate that the proposed method achieves the state-of-the-art inpainting performance among all compared methods.

The existing low-rank tensor completion methods develop many tensor decompositions and corresponding tensor ranks in order to reconstruct the missing information by exploiting the inherent low-rank structure under the assumption that the data is low-rank under one of the kinds of decompositions. However, the assumption is easily violated for real-world data, e.g., color images and multispectral images, as the low-rank structure of these data is not significant. To better take advantage of the global correlation relationship, we propose a kernel low-rank tensor completion model, where original data is mapped into the feature space using a kernel mapping. Although the original data is high-rank, it is low-rank in the feature space owing to the kernel mapping. Therefore, the proposed model could take advantage of the implicitly low-rank structure in the feature space and estimate the missing entries well. Considering it is not easy to explicitly kernelize the tensor, we reformulate the model as the inner product form and introduce the kernel trick for efficiently solving the resulting model. Extensive experiments on color images and multispectral images show that the proposed method outperforms the state-of-the-art low-rank tensor completion methods.

Recently, low-rank prior (LRP) has achieved success in tensor completion, thanks to its ability in capturing global correlations. However, since the presence of dynamic foreground breaks the low-rank assumption, LRP sometimes fails to work for complex videos. How to effectively capture the implicit low-rankness is a tricky challenge in complex video completion. To address this challenge, we propose a video completion model, which organically combines the respective merits of deterministic LRP and deep image prior. Specifically, we first decompose a complex video into the dynamic foreground and static background, and then characterize each component individually. For the dynamic foreground, instead of the hand-crafted prior, we consider the deep foreground prior expressed by U-Net architecture in an unsupervised manner, while we consider the tensor low-rank prior for the static background. Moreover, we develop an efficient alternating direction method of multipliers algorithm under the plug-and-play framework to tackle the resulting model. Different from the classic plug-and-play framework with a pre-trained and fixed network, our algorithm can adaptively update the parameters of network during iterations, which enables the network to have a more flexible expression ability. Experimental results verify that the proposed method can do better than some state-of-the-art tensor completion methods in complex video completion.

The general tensor-based methods can recover missing values of multidimensional images by exploiting the low-rankness on the pixel level. However, especially when considerable pixels of an image are missing, the low-rankness is not reliable on the pixel level, resulting in some details losing in their results, which hinders the performance of subsequent image applications (e.g., image recognition and segmentation). In this article, we suggest a novel multiscale feature (MSF) tensorization by exploiting the MSFs of multidimensional images, which not only helps to recover the missing values on a higher level, that is, the feature level but also benefits subsequent image applications. By exploiting the low-rankness of the resulting MSF tensor constructed by the new tensorization, we propose the convex and nonconvex MSF tensor train rank minimization (MSF-TT) to conjointly recover the MSF tensor and the corresponding original tensor in a unified framework. We develop the alternating directional method of multipliers (ADMMs) to solve the convex MSF-TT and the proximal alternating minimization (PAM) to solve the nonconvex MSF-TT. Moreover, we establish the theoretical guarantee of convergence for the PAM algorithm. Numerical examples of real-world multidimensional images show that the proposed MSF-TT outperforms other compared approaches in image recovery and the recovered MSF tensor can benefit the subsequent image recognition.

Hyperspectral image (HSI) mixed noise removal is a fundamental problem and an important preprocessing step in remote sensing fields. The low-rank approximation-based methods have been verified effective to encode the global spectral correlation for HSI denoising. However, due to the large scale and complexity of real HSI, previous low-rank HSI denoising techniques encounter several problems, including coarse rank approximation (such as nuclear norm), the high computational cost of singular value decomposition (SVD) (such as Schatten$p$-norm), and adaptive rank selection (such as low-rank factorization). In this article, two novel factor group sparsity-regularized nonconvex low-rank approximation (FGSLR) methods are introduced for HSI denoising, which can simultaneously overcome the mentioned issues of previous works. The FGSLR methods capture the spectral correlation via low-rank factorization, meanwhile utilizing factor group sparsity regularization to further enhance the low-rank property. It is SVD-free and robust to rank selection. Moreover, FGSLR is equivalent to Schatten$p$-norm approximation (Theorem 1), and thus FGSLR is tighter than the nuclear norm in terms of rank approximation. To preserve the spatial information of HSI in the denoising process, the total variation regularization is also incorporated into the proposed FGSLR models. Specifically, the proximal alternating minimization is designed to solve the proposed FGSLR models. Experimental results have demonstrated that the proposed FGSLR methods significantly outperform existing low-rank approximation-based HSI denoising methods.

Hyperspectral image (HSI) denoising has been regarded as an effective and economical preprocessing step in data subsequent applications. Recent nonlocal low-rank approximation on each full band patch group has demonstrated their superiority for HSI denoising. These methods, however, directly design the low-rank regularization to the grouped patch image itself (i.e., original domain), which ignores the spatial information of the grouped patch image and cannot explores the potential structure. To address these issues, this paper proposes a nonlocal group sparsifying transform learning method (dubbed TLNLGS) for HSI denoising. Motivated by the global spectral correlation in the HSI, we firstly impose a certain low-dimensional subspace hypothesis over the HSI to prevent the heavy computation burden with the spectral band increases, and then explore a discriminatively intrinsic nonlocal group sparse prior of the reduced image by transform model. The learned group sparse prior can not only excavate the nonlocal self-similarity as recent nonlocal low-rank approximation methods but also preserve the local spatial smooth structure of the image. Moreover, compared with the fixed transform domain (e.g., gradient and discrete cosine transformation domains), the transform learning scheme can improve the sparse representation ability. An efficient block coordinate descent (BCD) algorithm is developed to solve the proposed model. Extensive experiments, including simulated and real HSI datasets, indicate the superiority of the proposed TLNLGS method over the state-of-the-art HSI denoising approaches.

Fusing a pair of low-spatial-resolution hyperspectral image (LR-HSI) and high-spatial-resolution multispectral image (HR-MSI) has been regarded as an effective and economical strategy to achieve HR-HSI, which is essential to many applications. Among existing fusion models, the tensor ring (TR) decomposition-based model has attracted rising attention due to its superiority in approximating high-dimensional data compared to other traditional matrix/tensor decomposition models. Unlike directly estimating HR-HSI in traditional models, the TR fusion model translates the fusion procedure into an estimate of the TR factor of HR-HSI, which can efficiently capture the spatial–spectral correlation of HR-HSI. Although the spatial–spectral correlation has been preserved well by TR decomposition, the spatial–spectral continuity of HR-HSI is ignored in existing TR decomposition models, sometimes resulting in poor quality of reconstructed images. In this article, we introduce a factor smoothed regularization for TR decomposition to capture the spatial–spectral continuity of HR-HSI. As a result, our proposed model is calledfactor smoothed TR decompositionmodel, dubbedFSTRD. In order to solve the suggested model, we develop an efficient proximal alternating minimization algorithm. A series of experiments on four synthetic datasets and one real-world dataset show that the quality of reconstructed images can be significantly improved by the introduced factor smoothed regularization, and thus, the suggested method yields the best performance by comparing it to state-of-the-art methods.

The existing cloud removal methods either need a cloud mask as prior knowledge or detect clouds before cloud removal processing, i.e., the detection and removal processes are separate. In this article, we propose a box-constrained (BC) smooth low-rank plus group sparse model to simultaneously detect and remove clouds, by formulating the degraded data as the summation of image and cloud components. For the cloud component, we propose a group sparse function along the spectral dimension. This is motivated by our observations that: 1) one tube is contaminated by clouds if any pixels of this tube are contaminated by clouds; 2) the positions of tubes, which are taken at different times, contaminated by clouds are different. For the image component, we propose to use a tensor rank based on the tensor singular value decomposition. The tensor rank characterizes the global property of the image component and could not keep the cloud-free information unchanged. To address the problem, we introduce a BC on the image component to force its cloud-free information to be equal to the corresponding values of observed data. Owing to the BC, the proposed model integrates the cloud detection and removal processes so that the two processes could promote each other and result in a promising result. To solve the proposed model, we develop an efficient algorithm that can generate the cloud mask, image component, and cloud component alternately. Extensive experiments on synthetic and real data show that the proposed method is competitive compared with the completion and other cloud removal methods.

This article proposes a new denoising method for hyperspectral images (HSIs) corrupted by mixtures (in a statistical sense) of stripe noise, Gaussian noise, and impulsive noise. The proposed method has three distinctive features: 1) it exploits the intrinsic characteristics of HSIs, namely, low-rank and self-similarity; 2) the observation noise is assumed to be additive and modeled by a mixture of Gaussian (MoG) densities; 3) the inference is performed with an expectation maximization (EM) algorithm, which, in addition to the clean HSI, also estimates the mixture parameters (posterior probability of each mode and variances). Comparisons of the proposed method with state-of-the-art algorithms provide experimental evidence of the effectiveness of the proposed denoising algorithm. A MATLAB demo of this work will be available athttps://github.com/TaiXiangJiangfor the sake of reproducibility.

The existing nonblind cloud and cloud shadow (cloud/shadow) removal methods for remote sensing (RS) images are based on the assumption that cloud/shadow masks are accurately given. Since the masks are usually manually labeled or detected by cloud detection methods, whose accuracy cannot be well guaranteed, the cloud/shadow removal effect may be affected. In this article, we suggest a robust thick cloud/shadow removal (RTCR) method that meets the problem with an inaccurate mask. To faithfully reconstruct the multitemporal information, a coupled tensor factorization is used to explore the relationship between the abundances of the multitemporal images in the same scene. Moreover, an efficient algorithm is developed to solve the proposed model based on the augmented Lagrange multiplier method. The experimental results under accurate masks and inaccurate masks demonstrate its robustness and superiority for thick cloud/shadow removal.

Hyperspectral images (HSIs) are often contaminated by several types of noise, which significantly limits the accuracy of subsequent applications. Recently, low-rank modeling based on tensor singular value decomposition (T-SVD) has achieved great success in HSI restoration. Most of them use the convex and nonconvex surrogates of the tensor rank, which cannot well approximate the tensor singular values and obtain suboptimal restored results. We suggest a novel HSI restoration model by introducing a fibered rank constrained tensor restoration framework with an embedded plug-and-play (PnP)-based regularization (FRCTR-PnP). More precisely, instead of using the convex and nonconvex surrogates to approximate the fibered rank, the proposed model directly constrains the tensor fibered rank of the solution, leading to a better approximation to the original image. Since exploiting the low-fibered-rankness of HSI is mainly to capture the global structure, we further employ an implicit PnP-based regularization to preserve the image details. Particularly, the above two building blocks are complementary to each other, rather than isolated and uncorrelated. Based on the alternating direction multiplier method (ADMM), we propose an efficient algorithm to tackle the proposed model. For robustness, we develop a three-directional randomized T-SVD (3DRT-SVD), which preserves the intrinsic structure of the clean HSI and removes partial noise by projecting the HSI onto a low-dimensional essential subspace. Extensive experimental results including simulated and real data demonstrate that the proposed method achieves superior performance over compared methods in terms of quantitative evaluation and visual inspection.

Image denoising is often empowered by accurate prior information. In recent years, data-driven neural network priors have shown promising performance for RGB natural image denoising. Compared to classic handcrafted priors (e.g., sparsity and total variation), the “deep priors” are learned using a large number of training samples, which can accurately model the complex image generating process. However, data-driven priors are hard to acquire for hyperspectral images (HSIs) due to the lack of training data. A remedy is to use the so-called unsupervised deep image prior (DIP). Under the unsupervised DIP framework, it is hypothesized and empirically demonstrated that proper neural network structures are reasonable priors of certain types of images, and the network weights can be learned without training data. Nonetheless, the most effective unsupervised DIP structures were proposed for natural images instead of HSIs. The performance of unsupervised DIP-based HSI denoising is limited by a couple of serious challenges, namely network structure design and network complexity. This work puts forth an unsupervised DIP framework that is based on the classic spatiospectral decomposition of HSIs. Utilizing the so-called linear mixture model of HSIs, two types of unsupervised DIPs, that is, U-Net-like network and fully connected networks, are employed to model the abundance maps and endmembers contained in the HSIs, respectively. This way, empirically validated unsupervised DIP structures for natural images can be easily incorporated for HSI denoising. Besides, the decomposition also substantially reduces network complexity. An efficient alternating optimization algorithm is proposed to handle the formulated denoising problem. Simulated and real data experiments are employed to showcase the effectiveness of the proposed approach.

Hyperspectral image (HSI) denoising is a fundamental task in remote sensing image processing, which is helpful for HSI subsequent applications, such as unmixing and classification. Thanks to the powerful representation ability of untrained deep neural networks, deep image prior (DIP)-based methods achieve tremendous successes in image processing (e.g., denoising and inpainting). However, DIP-based methods neglect the tensor low-rank prior of the underlying HSI which will be beneficial to capturing the global structure of the underlying HSI. To address this issue, we propose a novel model for HSI denoising, which can simultaneously take respective advantages of the tensor low-rank prior and the deep spatial-spectral prior. The tensor low-rank prior leads to a better global structure and the deep spatial-spectral prior is complementary to preserve better local details. On the one hand, we adopt low-rank tensor ring (TR) decomposition to characterize the tensor low-rank prior and capture the global structure of the underlying HSI. On the other hand, we employ untrained deep neural networks to flexibly represent the deep spatial-spectral prior and capture the local details of the underlying HSI. To solve the proposed model, we develop an efficient alternating minimization algorithm. Experimental results on simulated and real data validate the advantages of the proposed model in HSI denoising. Compared with state-of-the-art HSI denoising methods, the proposed method preserves better local details and the global structure of the underlying HSI.

The image nonlocal self-similarity (NSS) prior refers to the fact that a local patch often has many nonlocal similar patches to it across the image and has been widely applied in many recently proposed machining learning algorithms for image processing. However, there is no theoretical analysis on its working principle in the literature. In this paper, we discover a potential causality between NSS and low-rank property of color images, which is also available to grey images. A new patch group based NSS prior scheme is proposed to learn explicit NSS models of natural color images. The numerical low-rank property of patched matrices is also rigorously proved. The NSS-based QMC algorithm computes an optimal low-rank approximation to the high-rank color image, resulting in high PSNR and SSIM measures and particularly the better visual quality. A new tensor NSS-based QMC method is also presented to solve the color video inpainting problem based on quaternion tensor representation. The numerical experiments on color images and videos indicate the advantages of NSS-based QMC over the state-of-the-art methods.

Recently, transform-based tensor nuclear norm (TNN) minimization methods have received increasing attention for recovering third-order tensors in multi-dimensional imaging problems. The main idea of these methods is to perform the linear transform along the third mode of third-order tensors and then minimize the nuclear norm of frontal slices of the transformed tensor. The main aim of this paper is to propose a nonlinear multilayer neural network to learn a nonlinear transform by solely using the observed tensor in a self-supervised manner. The proposed network makes use of the low-rank representation of the transformed tensor and data-fitting between the observed tensor and the reconstructed tensor to learn the nonlinear transform. Extensive experimental results on different data and different tasks including tensor completion, background subtraction, robust tensor completion, and snapshot compressive imaging demonstrate the superior performance of the proposed method over state-of-the-art methods.

Completing missing entries in multidimensional visual data is a typical ill-posed problem that requires appropriate exploitation of prior information of the underlying data. Commonly used priors can be roughly categorized into three classes: global tensor low-rankness, local properties, and nonlocal self-similarity (NSS); most existing works utilize one or two of them to implement completion. Naturally, there arises an interesting question: can one concurrently make use of multiple priors in a unified way, such that they can collaborate with each other to achieve better performance? This work gives a positive answer by formulating a novel tensor completion framework which can simultaneously take advantage of the global-local-nonlocal priors. In the proposed framework, the tensor train (TT) rank is adopted to characterize the global correlation; meanwhile, two Plug-and-Play (PnP) denoisers, including a convolutional neural network (CNN) denoiser and the color block-matching and 3 D filtering (CBM3D) denoiser, are incorporated to preserve local details and exploit NSS, respectively. Then, we design a proximal alternating minimization algorithm to efficiently solve this model under the PnP framework. Under mild conditions, we establish the convergence guarantee of the proposed algorithm. Extensive experiments show that these priors organically benefit from each other to achieve state-of-the-art performance both quantitatively and qualitatively.

The popular tensor train (TT) and tensor ring (TR) decompositions have achieved promising results in science and engineering. However, TT and TR decompositions only establish an operation between adjacent two factors and are highly sensitive to the permutation of tensor modes, leading to an inadequate and inflexible representation. In this paper, we propose a generalized tensor decomposition, which decomposes an Nth-order tensor into a set of Nth-order factors and establishes an operation between any two factors. Since it can be graphically interpreted as a fully-connected network, we named it fully-connected tensor network (FCTN) decomposition. The superiorities of the FCTN decomposition lie in the outstanding capability for characterizing adequately the intrinsic correlations between any two modes of tensors and the essential invariance for transposition. Furthermore, we employ the FCTN decomposition to one representative task, i.e., tensor completion, and develop an efficient solving algorithm based on proximal alternating minimization. Theoretically, we prove the convergence of the developed algorithm, i.e., the sequence obtained by it globally converges to a critical point. Experimental results substantiate that the proposed method compares favorably to the state-of-the-art methods based on other tensor decompositions.

The hyperspectral unmixing (HU) aims at estimating the spectral signatures of endmembers (or materials) and their corresponding abundance maps of the hyperspectral image (HSI). In this work, we treat the HSI as an 3D cube and propose a new tensor-based HU method. We decompose an HSI data as the sum of several multilinear rank-(Lr, Lr, 1) terms (or LL1 model). Based on the fact that the latent factors of LL1 model are physical meaningful (i.e., abundance maps and spectral signatures), we build a nonnegative tensor decomposition optimization model with the low-rank constraint and impose an implicit regularizer to exploit the nonlocal self-similarity prior of abundance maps-whose related subproblem can be easily solved under the plug-and-play framework. We develop an alternating direction method of multipliers algorithm to solve the proposed model. Numerical experiments demonstrate the effectiveness of our algorithm.

For multi-temporal remote sensing (RS) images, the distribution of surface materials is constant concerning time and the same material shows different spectral features at different times. Decomposing the image at each time into an abundance tensor and temporal features, there is a strong similarity between abundance tensors of all time. Based on this observation, we suggest a blind thick cloud/shadow removal model, which exploits the sparsity of the cloud component and the similarity between abundance tensors, achieving both cloud detection and multi-temporal information restoration. Moreover, a mask refinement strategy is designed to pursue the optimal cloud/shadow mask. We develop an efficient algorithm to solve the proposed model based on the augmented Lagrange multiplier method. The results of simulated experiments in different scenarios verify the superiority of the proposed method for thick cloud/shadow removal.

Recently, unsupervised deep learning-based methods have shown an empirical success in hyperspectral images (HSIs) denoising, profiting from the strong representation ability of convolutional neural networks. However, these methods only can describe the local structure of the spatial dimension, which is restricted to very limited local receptive fields. To overcome this difficulty, a novel HSIs denoising model based on the deep image prior (DIP) framework is proposed by adding a tensor ring (TR) decomposition, which can enlarge the receptive field of the spatial dimension and capture global spectral correlation simultaneously. Unlike the previous DIP framework that directly minimizes the objective function, we develop an algorithm based on proximal alternating minimization to decouple the model into the classic DIP framework and TR cores least-squares problems, which are easy to solve. Experimental results verify that the proposed DIP- TR compares favorably with compared methods in terms of quality metrics and visual performance.

The deep convolutional neural network has achieved significant progress for single image rain streak removal. However, most of the data-driven learning methods are full-supervised or semi-supervised, unexpectedly suffering from significant performance drop when dealing with the real rain. These data-driven learning methods are representative yet generalize poor for real rain. The opposite holds true for the model-driven unsupervised optimization methods. To overcome these problems, we propose a unified unsupervised learning framework which inherits the generalization and representation merits for real rain removal. Specifically, we first discover a simple yet important domain knowledge that directional rain streak is anisotropic while the natural clean image is isotropic, and formulate the structural discrepancy into the energy function of the optimization model. Consequently, we design an optimization model driven deep CNN in which the unsupervised loss function of the optimization model is enforced on the proposed network for better generalization. In addition, the architecture of the network mimics the main role of the optimization models with better feature representation. On one hand, we take advantage of the deep network to improve the representation. On the other hand, we utilize the unsupervised loss of the optimization model for better generalization. Overall, the unsupervised learning framework achieves good generalization and representation: unsupervised training (loss) with only a few real rainy images (input) and physical meaning network (architecture). Extensive experiments on synthetic and real-world rain datasets show the superiority of the proposed method.

Removing rain streaks in videos has recently received much attention. Existing hand-crafted priors-based methods suffer from limited representation abilities, and supervised deep learning methods need high-quality training data. This paper proposes a novel video rain streaks removal method by reconciling hand-crafted and self-supervised deep priors. The hand-crafted priors include the learned gradient prior, the sparse prior, and the temporal local smooth prior. Meanwhile, a deep convolutional neural network is employed to self-supervisedly capture the deep prior of the clean video without any training data. Our method organically integrates hand-crated priors and self-supervised deep priors to achieve both high generalization abilities and representation abilities. Thus, our method can faithfully remove directional rain streaks in real world videos. To address the resulting model, we introduce an alternating direction multiplier method algorithm. Extensive experiments validate the superiority of our method over state-of-the-art methods.

Sparse unmixing is an important technique for analyzing and processing hyperspectral images (HSIs). Simultaneously exploiting spatial correlation and sparsity improves substantially abundance estimation accuracy. In this article, we propose to exploit nonlocal spatial information in the HSI for the sparse unmixing problem. Specifically, we first group similar patches in the HSI, and then unmix each group by imposing simultaneous a low-rank constraint and joint sparsity in the corresponding third-order abundance tensor. To this end, we build an unmixing model with a mixed regularization term consisting of the sum of the weighted tensor trace norm and the weighted tensor$\ell _{2,1}$-norm of the abundance tensor. The proposed model is solved under the alternating direction method of multipliers framework. We term the developed algorithm as the nonlocal tensor-based sparse unmixing algorithm. The effectiveness of the proposed algorithm is illustrated in experiments with both simulated and real hyperspectral data sets.

In hyperspectral image (HSI) denoising, subspace-based denoising methods can reduce the computational complexity of the denoising algorithm. However, the existing matrix subspaces, which are generated by the unfolding matrix of the HSI tensor, cannot completely represent a tensor since the unfolding operation will destroy the tensor structure. To overcome this, we design a novel basis tensor that is directly learned from the original tensor and present a tensor subspace representation (TenSR), which is a more authentic representation for delivering the intrinsic structure of the tensor than a matrix subspace representation. Equipped with the TenSR, we then propose a TenSR-based HSI denoising (TenSRDe) model, which simultaneously considers the low-tubal rankness of the HSI tensor and the nonlocal self-similarity of the coefficient tensor. Moreover, we develop an efficient proximal alternating minimization (PAM) algorithm to solve the proposed nonconvex model and theoretically prove that the algorithm globally converges to a critical point. Experiments implemented on simulated and real data sets substantiate the denoising effect and efficiency of the proposed method.

This paper addresses the tensor completion problem, which aims to recover missing information of multi-dimensional images. How to represent a low-rank structure embedded in the underlying data is the key issue in tensor completion. In this work, we suggest a novel low-rank tensor representation based on coupled transform, which fully exploits the spatial multi-scale nature and redundancy in spatial and spectral/temporal dimensions, leading to a better low tensor multi-rank approximation. More precisely, this representation is achieved by using two-dimensional framelet transform for the two spatial dimensions, one/two-dimensional Fourier transform for the temporal/spectral dimension, and then Karhunen-Loéve transform (via singular value decomposition) for the transformed tensor. Based on this low-rank tensor representation, we formulate a novel low-rank tensor completion model for recovering missing information in multi-dimensional visual data, which leads to a convex optimization problem. To tackle the proposed model, we develop the alternating directional method of multipliers (ADMM) algorithm tailored for the structured optimization problem. Numerical examples on color images, multispectral images, and videos illustrate that the proposed method outperforms many state-of-the-art methods in qualitative and quantitative aspects.

Recently emerged deep learning methods have achieved great success in single image rain streaks removal. However, existing methods ignore an essential factor in the rain streaks generation mechanism, i.e., the motion blur leading to the line pattern appearances. Thus, they generally produce overderaining or underderaining results. In this article, inspired by the generation mechanism, we propose a novel rain streaks removal framework using a kernel-guided convolutional neural network (KGCNN), achieving state-of-the-art performance with a simple network architecture. More precisely, our framework consists of three steps. First, we learn the motion blur kernel by a plain neural network, termed parameter network, from the detail layer of a rainy patch. Then, we stretch the learned motion blur kernel into a degradation map with the same spatial size as the rainy patch. Finally, we use the stretched degradation map together with the detail patches to train a deraining network with a typical ResNet architecture, which produces the rain streaks with the guidance of the learned motion blur kernel. Experiments conducted on extensive synthetic and real data demonstrate the effectiveness of the proposed KGCNN, in terms of rain streaks removal and image detail preservation.

Hyperspectral imaging, providing abundant spatial and spectral information simultaneously, has attracted a lot of interest in recent years. Unfortunately, due to the hardware limitations, the hyperspectral image (HSI) is vulnerable to various degradations, such as noises (random noise), blurs (Gaussian and uniform blur), and downsampled (both spectral and spatial downsample), each corresponding to the HSI denoising, deblurring, and super-resolution tasks, respectively. Previous HSI restoration methods are designed for one specific task only. Besides, most of them start from the 1-D vector or 2-D matrix models and cannot fully exploit the structurally spectral-spatial correlation in 3-D HSI. To overcome these limitations, in this article, we propose a unified low-rank tensor recovery model for comprehensive HSI restoration tasks, in which nonlocal similarity within spectral-spatial cubic and spectral correlation are simultaneously captured by third-order tensors. Furthermore, to improve the capability and flexibility, we formulate it as a weighted low-rank tensor recovery (WLRTR) model by treating the singular values differently. We demonstrate the reweighed strategy, which has been extensively studied in the matrix, also greatly benefits the tensor modeling. We also consider the stripe noise in HSI as the sparse error by extending WLRTR to robust principal component analysis (WLRTR-RPCA). Extensive experiments demonstrate the proposed WLRTR models consistently outperform state-of-the-art methods in typical HSI low-level vision tasks, including denoising, destriping, deblurring, and super-resolution.

Stripe noise from different remote sensing imaging systems varies considerably in terms of response, length, angle, and periodicity. Due to the complex distributions of different stripes, the destriping results of previous methods may be oversmoothed or contain residual stripe. To overcome this key problem, we provide a comprehensive analysis of existing destriping methods and propose a deep convolutional neural network (CNN) for handling various kinds of stripes. Moreover, previous methods individually model the stripe or the image priors, which may lose the relationship between them. In this article, a two-stream CNN is designed to simultaneously model the stripe and image, which better facilitates distinguishing them from each other. Moreover, we incorporate the wavelet into our CNN model for better directional feature representation. Therefore, the CNN learns the discriminative representation from the external data set, while the wavelet models the internal directionality of the stripe, in which both the internal and external priors are beneficial to the destriping task. In addition, the wavelet extracts the multiscale information with a larger receptive field for global contextual information modeling; thus, we can better distinguish the stripe from the similar image line pattern structures. The proposed method has been extensively evaluated on a number of data sets and outperforms the state-of-the-art methods by substantially a large margin in terms of quantitative and qualitative assessments, speed, and robustness.

Hyperspectral image (HSI) denoising is a fundamental problem in remote sensing and image processing. Recently, nonlocal low-rank tensor approximation-based denoising methods have attracted much attention due to their advantage of being capable of fully exploiting the nonlocal self-similarity and global spectral correlation. Existing nonlocal low-rank tensor approximation methods were mainly based on two common decomposition [Tucker or CANDECOMP/PARAFAC (CP)] methods and achieved the state-of-the-art results, but they are subject to certain issues and do not produce the best approximation for a tensor. For example, the number of parameters for Tucker decomposition increases exponentially according to its dimensions, and CP decomposition cannot better preserve the intrinsic correlation of the HSI. In this article, a novel nonlocal tensor-ring (TR) approximation is proposed for HSI denoising by using TR decomposition to explore the nonlocal self-similarity and global spectral correlation simultaneously. TR decomposition approximates a high-order tensor as a sequence of cyclically contracted third-order tensors, which has strong ability to explore these two intrinsic priors and to improve the HSI denoising results. Moreover, an efficient proximal alternating minimization algorithm is developed to optimize the proposed TR decomposition model efficiently. Extensive experiments on three simulated data sets under several noise levels and two real data sets verify that the proposed TR model provides better HSI denoising results than several state-of-the-art methods in terms of quantitative and visual performance evaluations.

Hyperspectral (HS) imaging has shown its superiority in many real applications. However, it is usually difficult to obtain high-resolution (HR) HS images through existing imaging techniques due to the hardware limitations. To improve the spatial resolution of HS images, this article proposes an effective HS-multispectral (HS-MS) image fusion method by combining the ideas of nonlocal low-rank tensor modeling and spectral unmixing. To be more precise, instead of unfolding the HS image into a matrix as done in the literature, we directly represent it as a tensor, then a designed nonlocal Tucker decomposition is used to model its underlying spatial-spectral correlation and the spatial self-similarity. The MS image serves mainly as a data constraint to maintain spatial consistency. To further reduce the spectral distortions in spatial enhancement, endmembers, and abundances from the spectral are used for spectral regularization. An efficient algorithm based on the alternating direction method of multipliers (ADMM) is developed to solve the resulting model. Extensive experiments on four HS image data sets demonstrate the superiority of the proposed method over several state-of-the-art HS-MS image fusion methods.

As a preprocessing step, hyperspectral image (HSI) restoration plays a critical role in many subsequent applications. Recently, based on the framework of subspace representation and low-rank matrix/tensor factorization (LRMF/LRTF), many single-factor-regularized methods add various regularizations on the spatial factor to characterize its spatial prior knowledge. However, these methods neglect the common characteristics among different bands and the spectral continuity of HSIs. To tackle this issue, this article establishes a bridge between the factor-based regularization and the HSI priors and proposes a double-factor-regularized LRTF model for HSI mixed noise removal. The proposed model employs LRTF to characterize the spectral global low rankness, introduces a weighted group sparsity constraint on the spatial difference images (SpatDIs) of the spatial factor to promote the group sparsity in the SpatDIs of HSIs, and suggests a continuity constraint on the spectral factor to promote the spectral continuity of HSIs. Moreover, we develop a proximal alternating minimization-based algorithm to solve the proposed model. Extensive experiments conducted on the simulated and real HSIs demonstrate that the proposed method has superior performance on mixed noise removal compared with the state-of-the-art methods based on subspace representation, noise modeling, and LRMF/LRTF.

The tensor tubal rank, defined based on the tensor singular value decomposition (t-SVD), has obtained promising results in hyperspectral image (HSI) denoising. However, the framework of the t-SVD lacks flexibility for handling different correlations along different modes of HSIs, leading to suboptimal denoising performance. This article mainly makes three contributions. First, we introduce a new tensor rank named tensor fibered rank by generalizing the t-SVD to the mode-k t-SVD, to achieve a more flexible and accurate HSI characterization. Since directly minimizing the fibered rank is NP-hard, we suggest a three-directional tensor nuclear norm (3DTNN) and a three-directional log-based tensor nuclear norm (3DLogTNN) as its convex and nonconvex relaxation to provide an efficient numerical solution, respectively. Second, we propose a fibered rank minimization model for HSI mixed noise removal, in which the underlying HSI is modeled as a low-fibered-rank component. Third, we develop an efficient alternating direction method of multipliers (ADMMs)-based algorithm to solve the proposed model, especially, each subproblem within ADMM is proven to have a closed-form solution, although 3DLogTNN is nonconvex. Extensive experimental results demonstrate that the proposed method has superior denoising performance, as compared with the state-of-the-art competing methods on low-rank matrix/tensor approximation and noise modeling.

Hyperspectral image compressive sensing reconstruction (HSI-CSR) can largely reduce the high expense and low efficiency of transmitting HSI to ground stations by storing a few compressive measurements, but how to precisely reconstruct the HSI from a few compressive measurements is a challenging issue. It has been proven that considering the global spectral correlation, spatial structure, and nonlocal self-similarity priors of HSI can achieve satisfactory reconstruction performances. However, most of the existing methods cannot simultaneously capture the mentioned priors and directly design the regularization term to the HSI. In this article, we propose a novel subspace-based nonlocal tensor ring decomposition method (SNLTR) for HSI-CSR. Instead of designing the regularization of the low-rank approximation to the HSI, we assume that the HSI lies in a low-dimensional subspace. Moreover, to explore the nonlocal self-similarity and preserve the spatial structure of HSI, we introduce a nonlocal tensor ring decomposition strategy to constrain the related coefficient image, which can decrease the computational cost compared to the methods that directly employ the nonlocal regularization to HSI. Finally, a well-known alternating minimization method is designed to efficiently solve the proposed SNLTR. Extensive experimental results demonstrate that our SNLTR method can significantly outperform existing approaches for HSI-CSR.

The main aim of this paper is to develop a framelet representation of the tensor nuclear norm for third-order tensor recovery. In the literature, the tensor nuclear norm can be computed by using tensor singular value decomposition based on the discrete Fourier transform matrix, and tensor completion can be performed by the minimization of the tensor nuclear norm which is the relaxation of the sum of matrix ranks from all Fourier transformed matrix frontal slices. These Fourier transformed matrix frontal slices are obtained by applying the discrete Fourier transform on the tubes of the original tensor. In this paper, we propose to employ the framelet representation of each tube so that a framelet transformed tensor can be constructed. Because of framelet basis redundancy, the representation of each tube is sparsely represented. When the matrix slices of the original tensor are highly correlated, we expect the corresponding sum of matrix ranks from all framelet transformed matrix frontal slices would be small, and the resulting tensor completion can be performed much better. The proposed minimization model is convex and global minimizers can be obtained. Numerical results on several types of multi-dimensional data (videos, multispectral images, and magnetic resonance imaging data) have tested and shown that the proposed method outperformed the other testing methods.

Images captured in rainy conditions are often corrupted by unexpected rain streaks, which severely degrade the performance of subsequent processes in outdoor computer vision systems. In this paper, we exploit the directional smoothness of rain streaks for the single-image rain streaks removal and propose a convex model that uses the directional total variation (DTV) to characterize the smoothness of rain streaks in arbitrary orientations. The proposed model consists of four terms: the fidelity term, the ℓ1norm for the sparsity of rain streaks, and two DTV regularization terms for the directional smoothness and the piecewise smoothness of rain streaks and rain-free backgrounds, respectively. To solve the proposed model, we develop an efficient algorithm based on the alternating direction method of multipliers (ADMM) framework. Extensive experimental results on both synthetic and real rainy images show that our method outperforms the recent state-of-the-art methods visually and quantitatively.

In this paper, we propose a novel low-tubal-rank tensor recovery model, which directly constrains the tubal rank prior for effectively removing the mixed Gaussian and sparse noise in hyperspectral images. The constraints of tubal-rank and sparsity can govern the solution of the denoised tensor in the recovery procedure. To solve the constrained low-tubal-rank model, we develop an iterative algorithm based on bilateral random projections to efficiently solve the proposed model. The advantage of random projections is that the approximation of the low-tubal-rank tensor can be obtained quite accurately in an inexpensive manner. Experimental examples for hyperspectral image denoising are presented to demonstrate the effectiveness and efficiency of the proposed method.

In this paper, we propose a new tensor rank, named fibered rank, by generalizing the tensor singular value decomposition (t-SVD) to the mode-k t-SVD. It factorizes a three-way tensor into two mode-k orthogonal tensors and a mode-k diagonal tensor. To efficiently remove mixed noise and finely preserve the structure information, we propose a novel hyperspectral image denoising model based on the three-directional tensor nuclear norm (3DTNN), which is a convex relaxation of the fibered rank. An efficient alternating direction method of multipliers (ADMM)-based algorithm is developed to solve the proposed model. Experimental results demonstrate the superiority of the proposed method over the compared ones.

Hyperspectral unmixing has attracted much attention in recent years. Single sparse unmixing assumes that a pixel in a hyperspectral image consists of a relatively small number of spectral signatures from large, ever-growing, and available spectral libraries. Joint-sparsity (or row-sparsity) model typically enforces all pixels in a neighborhood to share the same set of spectral signatures. The two sparse models are widely used in the literature. In this paper, we propose a joint-sparsity-blocks model for abundance estimation problem. Namely, the abundance matrix of size m × n is partitioned to have one row block and s column blocks and each column block itself is joint-sparse. It generalizes both the single (i.e., s = n) and the joint (i.e., s = 1) sparsities. Moreover, concatenating the proposed joint-sparsity-blocks structure and low rankness assumption on the abundance coefficients, we develop a new algorithm called joint-sparseblocks and low-rank unmixing. In particular, for the joint-sparseblocks regression problem, we develop a two-level reweighting strategy to enhance the sparsity along the rows within each block. Simulated and real-data experiments demonstrate the effectiveness of the proposed algorithm.

Rain streaks removal is an important issue in outdoor vision systems and has recently been investigated extensively. In this paper, we propose a novel video rain streak removal approach FastDeRain, which fully considers the discriminative characteristics of rain streaks and the clean video in the gradient domain. Specifically, on the one hand, rain streaks are sparse and smooth along the direction of the raindrops, whereas on the other hand, clean videos exhibit piecewise smoothness along the rain-perpendicular direction and continuity along the temporal direction. Theses smoothness and continuity results in the sparse distribution in the different directional gradient domain, respectively. Thus, we minimize 1) the ℓ1 norm to enhance the sparsity of the underlying rain streaks, 2) two ℓ1 norm of unidirectional Total Variation (TV) regularizers to guarantee the anisotropic spatial smoothness, and 3) an ℓ1 norm of the time-directional difference operator to characterize the temporal continuity. A split augmented Lagrangian shrinkage algorithm (SALSA) based algorithm is designed to solve the proposed minimization model. Experiments conducted on synthetic and real data demonstrate the effectiveness and efficiency of the proposed method. According to comprehensive quantitative performance measures, our approach outperforms other state-of-the-art methods, especially on account of the running time. The code of FastDeRain can be downloaded at https://github.com/TaiXiangJiang/FastDeRain.

Rain streaks removal is an important issue of the outdoor vision system and has been recently investigated extensively. In this paper, we propose a novel tensor based video rain streaks removal approach by fully considering the discriminatively intrinsic characteristics of rain streaks and clean videos, which needs neither rain detection nor time-consuming dictionary learning stage. In specific, on the one hand, rain streaks are sparse and smooth along the raindrops direction, and on the other hand, the clean videos possess smoothness along the rain-perpendicular direction and global and local correlation along time direction. We use the l1 norm to enhance the sparsity of the underlying rain, two unidirectional Total Variation (TV) regularizers to guarantee the different discriminative smoothness, and a tensor nuclear norm and a time directional difference operator to characterize the exclusive correlation of the clean video along time. Alternation direction method of multipliers (ADMM) is employed to solve the proposed concise tensor based convex model. Experiments implemented on synthetic and real data substantiate the effectiveness and efficiency of the proposed method. Under comprehensive quantitative performance measures, our approach outperforms other state-of-the-art methods.

We propose a novel image inpainting model that can effectively estimate missing pixels in an observed image. The latent image is characterized by a group-based low-rank prior, which assumes that a group of vectorized similar image patches can be well approximated by a low-rank matrix. We enforce the low-rankness of each group by penalizing a truncated difference of the l1and the l2norms of its singular values, which achieves a close approximation to the matrix rank. We apply a difference of convex algorithm (DCA) to solve the proposed model efficiently. Our method is validated on filling missing blocks and randomly missing pixels, with superior performance over the state-of-the-art.

In this paper, we develop a sparsity based model for the fusion of a high spatial-resolution image and a multispectral image. The given model is based on the combination of a dynamic gradient sparsity (DGS) and an anisotropic spectral-spatial total variation (ASSTV). We design an alternating direction method of multipliers (ADMM) based algorithm to solve the proposed model. In contrast to existing approaches, the proposed method can generate more spatial details as well as preserve favorable spectral information. Experimental results demonstrate that the proposed approach outperforms several state-of-the-art image fusion methods both quantitatively and visually, in terms of both pansharpening application of remote sensing images and fusion application of natural color images.

In this paper, we focus on the despeckling of synthetic aperture radar (SAR) images by variational methods which introduce nonlocal regularization functionals. To achieve this goal, two models are investigated from different aspects. The first model is derived for the logarithmically transformed (homomorphic) domain of the SAR data, and the other is derived for the original (nonhomomorphic) domain. The statistical properties of the speckle and the log-transformed speckle are analyzed, and the similarity measurements between pixels in the homomorphic domain and nonhomomorphic domain are then derived for constructing the corresponding nonlocal regularization functionals. Meanwhile, in the proposed models, we develop a strategy to adaptively choose the regularization parameters based on both the local heterogeneity information and the noise level of the images, aiming at getting a better balance between the goodness of fit of the original data and the amount of smoothing. A quasi-Newton iteration method is employed to quickly minimize the proposed adaptive nonlocal functionals. Experiments conducted on both simulated images and real SAR images confirm the good performances of the proposed methods, both in reducing speckle and preserving image quality.

This paper proposes a framelet-based convex optimization model for multiplicative noise and blur removal problem. The main idea is to employ framelet expansion to represent the original image and use the variable decomposition to solve the problem. Because of the nature of multiplicative noise, we decompose the observed data into the original image variable and the noise variable to obtain the resulting model. The original image variable is represented by framelet, and it is determined by using$l_{1}$-norm in the selection and shrinkage of framelet coefficients. The noise variable is measured by using the mean and the variance of the underlying probability distribution. This framelet setting can be applied to analysis, synthesis, and balanced approaches, and the resulting optimization models are convex, such that they can be solved very efficiently by the alternating direction of a multiplier method. An another contribution of this paper is to propose to select the regularization parameter by using the$l_{1}$-based L-curve method for these framelet based models. Numerical examples are presented to illustrate the effectiveness of these models and show that the performance of the proposed method is better than that by the existing methods.

In this paper, we propose an automatic and efficient method to enhance edge sharpness of images. Starting from an image with blur edges, we improve the edges using transformed Heaviside functions for better visualization. In addition, we provide an efficient method to directly compute the scaling and shifting factors of the transformed Heaviside functions, so that blur edges can be improved accurately. Experimental results show that the proposed method is fast and can get sharper image edges than some recent state-of-the-art edge enhancement methods. We also apply the edge sharpening method to image super-resolution and obtained promising results.

In this paper, we address the total variation (TV)-based nonlinear image restoration problems. In nonlinear image restoration problems, an original image is corrupted by a spatially-invariant blur, the build-in nonlinearity in imaging system, and the additive Gaussian white noise. We study the objective function consisting of the nonlinear least squares data-fitting term and the TV regularization term of the restored image. By making use of the structure of the objective function, an efficient alternating direction method of multipliers can be developed for solving the proposed model. The convergence of the numerical scheme is also studied. Numerical examples, including nonlinear image restoration and high-dynamic range imaging are reported to demonstrate the effectiveness of the proposed model and the efficiency of the proposed numerical scheme.

The main contribution of this paper is to propose a new convex optimization model for multiplicative noise and blur removal. The main idea is to rewrite a blur and multiplicative noise equation such that both the image variable and the noise variable are decoupled. The resulting objective function involves the total variation regularization term, the term of variance of the inverse of noise, the $\ell_1$-norm of the data-fitting term among the observed image, and noise and image variables. Such a convex minimization model can be solved efficiently by using many numerical methods in the literature. Numerical examples are presented to demonstrate the effectiveness of the proposed model. Experimental results show that the proposed model can handle blur and multiplicative noise (Gamma, Gaussian, or Rayleigh distribution) removal quite well.

The main aim of this paper is to study total variation (TV) regularization in deblurring and sparse unmixing of hyperspectral images. In the model, we also incorporate blurring operators for dealing with blurring effects, particularly blurring operators for hyperspectral imaging whose point spread functions are generally system dependent and formed from axial optical aberrations in the acquisition system. An alternating direction method is developed to solve the resulting optimization problem efficiently. According to the structure of the TV regularization and sparse unmixing in the model, the convergence of the alternating direction method can be guaranteed. Experimental results are reported to demonstrate the effectiveness of the TV and sparsity model and the efficiency of the proposed numerical scheme, and the method is compared to the recent Sparse Unmixing via variable Splitting Augmented Lagrangian and TV method by Iordache