Incomplete multi-view clustering (IMVC) has recently attracted increasing attention and achieved notable progress in computer vision. However, existing IMVC approaches still face several critical challenges. First, most methods fail to capture the inherent nonlinear structures of real-world data. Second, they fail to sufficiently exploit low-rankness and smoothness of multiple views. To overcome these limitations, we propose a novel method, termed Incomplete Multi-View Kernel Subspace Clustering with Tensor-Correlated Total Variation Regularization (KSC-TCTV), which integrates the ability of kernels to capture nonlinear separability with the strength of tensors in characterizing high-order correlations. Specifically, KSC-TCTV first effectively models nonlinear structures by embedding data into a kernel Hilbert space. And then, we introduce a log-based tensor-correlated total variation (Logt-CTV) regularizer in the kernel space, which jointly enforces global low-rankness for inter-view dependency modeling and local smoothness for intra-view structure preservation. The Logt-CTV employs logarithmic non- convex relaxation to mitigate the estimation bias. Experiments on several public benchmark datasets demonstrate that KSC-TCTV outperforms the state-of-the-art IMVC methods.

The effective integration and classification of hyperspectral images (HSIs) and light detection and ranging (LiDAR) data is of great significance in Earth observation missions, which are confronted with challenges such as insufficient information utilization and feature heterogeneity. This article proposes a multimodal quaternion representation network (MMQRN) for multisource remote sensing (RS) data classification. Specifically, we first propose the multimodal quaternion representation (MMQR), which employs the orthogonal imaginary components of quaternions to model the complex nonlinear interactions among complementary features, thereby enabling their comprehensive fusion and utilization. Subsequently, we design a multimodal feature cross-fusion (MFCF) framework to integrate multisource, multimodal, and multilevel features adequately. Finally, we leverage the ability to capture long-term dependencies of transformers to design a quaternion convolutional transformer network (QCTN) for modeling global and local spatial-spectral information, respectively. Experiments conducted on three multisource RS datasets demonstrate the superior performance of the proposed MMQRN relative to other state-of-the-art classification methods.

Implicit neural representation (INR) has emerged as a powerful representation for data (e.g., multispectral images and videos). Previously, most INR methods directly represent data in the original space. However, since different frequency components are mixed in the original space, it is difficult to capture these frequency components simultaneously and accurately. To alleviate this limitation, we suggest a new frequency-aware implicit neural representation (FA-INR) working in a physically interpretable and learnable frequency space by cleverly introducing an extra frequency dimension, which allows us to readily decouple and modulate different frequency components, leading to a more accurate characterization of different frequency components in a divide-and-conquer manner. Specifically, the FA-INR consists of two important modules, i.e., the frequency module and the integration module. In the frequency module, we propose a new low-rank tensor frequency function to compactly and continuously represent the latent frequency space. In the integration module, different frequency components are adaptively integrated back to the original space. Extensive experiments on various multi-dimensional data, including multispectral images, color videos, and light field data, demonstrate that the proposed FA-INR significantly outperforms the state-of-the-art INR methods, especially for characterizing high-frequency components (e.g., textures and edges).

Hyperspectral image (HSI) denoising is a critical preprocessing step in remote sensing. Recently, the denoising diffusion probabilistic models (DDPMs) have emerged as the powerful generative models. However, applying DDPM to HSI denoising task remains challenging owing to the scarcity and acquisition difficulty of HSIs. Thus, how to effectively incorporate physical priors into the DDPM to improve denoising performance remains an underexplored issue. To this end, we propose an Unmixing-Guided Learnable Abundance-Latent Diffusion for HSI Denoising (ULADiff), which is a from-scratch, task-specific diffusion framework that incorporates physically interpretable priors and conditional information into the DDPM. ULADiff comprises three key components, including a Spectral Unmixing Transformer (SUT) network, an abundance-based diffusion model, and a reconstruction module. Specifically, we employ a learnable block-based SUT module in a self-supervised manner to decompose noisy HSIs into the abundance maps and endmembers. The SUT module enables the diffusion model to operate in a lower-dimensional abundance domain that better captures the underlying structure of HSIs. Then, we incorporate the first eigenimage, the reconstructed image via Singular Value Decomposition, as a physically meaningful condition to facilitate controllable generation. Furthermore, we propose a reconstruction module that enforces a spatial-spectral consistency prior by simultaneously imposing total variation regularization on the endmembers and a sparsity constraint on the abundance maps. This design preserves the intrinsic structures of the HSI and improves reconstruction quality. Comprehensive evaluations on synthetic and real-world datasets demonstrate that ULADiff outperforms state-of-the-art methods in both quantitative performance and visual fidelity.

Recently, the transform-based tensor representation has attracted increasing attention in multimedia data (e.g., images and videos) recovery problems, which consists of two indispensable components, i.e., the transform and the characterization. Previously, the development of transform-based tensor representation has focused mainly on the transform perspective. Although several attempts have considered shallow matrix factorization (e.g., singular value decomposition and nonnegative matrix factorization) for characterizing the frontal slices of the transformed tensor (termed the latent tensor), the faithful characterization perspective has been underexplored. To address this issue, we propose a unifiedDeepTensorRepresentation (DTR) framework by synergistically combining the deep latent generative module and the deep transform module. Especially, the deep latent generative module can faithfully generate the latent tensor as compared with shallow matrix factorization. The new DTR framework not only allows us to better understand the classical shallow representations but also leads us to explore new representations. To examine the representation capability of the proposed DTR, we consider the representative multidimensional data recovery task and suggest an unsupervised DTR-based multidimensional data recovery model. Extensive experiments demonstrate that DTR achieves superior performance compared to the state-of-the-art methods from both quantitative and qualitative aspects, especially for fine detail recovery.

The transform-based tensor nuclear norm (TNN) methods have shown promise in hyperspectral images (HSIs) mixed noise removal. However, these transform-based methods only consider the low-rankness of the transformed tensor. The sparsity of the transformed tensor, which is beneficial for HSIs denoising, is neglected in the transform-based TNN methods. In this paper, we propose a novel HSIs denoising model based on the group-tube transform induced collaborative sparsity and low-rankness (GTCSLR). Here, GTC-SLR organically integrates the sparsity and low-rankness, capitalizing on their synergies to enhance denoising performance. We develop a proximal alternating minimization (PAM)-based algorithm to efficiently address the resulting optimization problem. Experimental results demonstrate the superiority of our method over state-of-the-art HSIs denoising methods, as evidenced by quality metrics and visual assessments.

Recently, subspace representation-based methods have shown notable success in hyperspectral image (HSI) denoising by exploiting the spectral correlation of HSIs. However, these methods cannot fully explore the spatial correlation within HSIs, resulting in unsatisfactory results. To address this limitation, we propose a novel generalized Kronecker decomposition-based subspace representation (GKDSR) method, which can simultaneously characterize the spatial and spectral correlation of HSIs. Specifically, we first decompose an HSI as the spectral subspace and coefficients. Then we impose the face-wise generalized Kronecker decomposition to the coefficients to fully characterize the spatial correlation of the HSI. Armed with the proposed GKDSR method, we build a GKDSR-based HSI denoising model, which can not only achieve a promising spectral fidelity but also preserve spatial fine details. To solve the proposed HSI denoising model, we develop an efficient alternating minimization-based algorithm. Experimental results demonstrate the significant superiority of the proposed method compared with competing methods in terms of spectral fidelity and preservation of spatial details.

Recently, the transform-based tensor nuclear norm (TNN) methods have shown promising performance and drawn increasing attention in tensor completion (TC) problems. The main idea of these methods is to exploit the low-rank structure of frontal slices of the tensor under the transform. However, the transforms in TNN methods usually treat all modes equally and do not consider the different traits of different modes (i.e., spatial and spectral/temporal modes). To address this problem, we suggest a new low-rank tensor representation based on the coupled nonlinear transform (called CoNoT) for a better low-rank approximation. Concretely, spatial and spectral/temporal transforms in the CoNoT, respectively, exploit the different traits of different modes and are coupled together to boost the implicit low-rank structure. Here, we use the convolutional neural network (CNN) as the CoNoT, which can be learned solely from an observed multidimensional image in an unsupervised manner. Based on this low-rank tensor representation, we build a new multidimensional image completion model. Moreover, we also propose an enhanced version (called Ms-CoNoT) to further exploit the spatial multiscale nature of real-world data. Extensive experiments on real-world data substantiate the superiority of the proposed models against many state-of-the-art methods both qualitatively and quantitatively.

Recently, tensor singular value decomposition (t-SVD) has emerged as a promising tool for hyperspectral image (HSI) processing. In the t-SVD, there are two key building blocks: (i) the low-rank enhanced transform and (ii) the accompanying low-rank characterization of transformed frontal slices. Previous t-SVD methods mainly focus on the developments of (i), while neglecting the other important aspect, i.e., the exact characterization of transformed frontal slices. In this letter, we exploit the potentiality in both building blocks by leveraging the Hierarchical nonlinear transform and the Hierarchical matrix factorization to establish a new Tensor Factorization (termed as H2TF). Compared to shallow counter partners, e.g., low-rank matrix factorization or its convex surrogates, H2TF can better capture complex structures of transformed frontal slices due to its hierarchical modeling abilities. We then suggest the H2TF-based HSI denoising model and develop an alternating direction method of multipliers-based algorithm to address the resultant model. Extensive experiments validate the superiority of our method over state-of-the-art HSI denoising methods.

Recently, untrained neural network priors (UNNPs) have received increasing attention for multi-dimensional image recovery. However, previous studies are based on over-parameterized untrained neural networks, which results in unstable behavior. In this letter, we propose an untrained low-rank neural network prior (ULRNNP) for multi-dimensional image recovery, which enjoys the powerful representation ability and stable behavior. More specifically, the elaborately designed nonlinear Tucker decomposition module implicitly imposes low-rank constraints on the feature tensor and can more compactly represent the feature tensor. Attributed to the suggested nonlinear Tucker decomposition module, ULRNNP can simultaneously enjoy strong representation ability and stable behavior. The friendly stable behavior allows us to design a friendly stopping criteria without the reference ground truth image as compared with classic UNNP-based methods. Extensive experiments on different multi-dimensional image datasets validate the superior performance of the proposed ULRNNP over state-of-the-art methods.

Cloud removal is an important task in the remotely sensed images (RSIs) processing, which is beneficial for downstream applications, such as unmixing, fusion, and target detection. Multi-temporal remotely sensed images (MRSIs), which contains the abundant spatial-spectral-temporal (SST) information, potentially bring the new opportunities for cloud removal. However, how to effectively and efficiently explore the rich information of MRSIs remains a challenge. Inspired by the low-rankness of MRSIs, we propose an Unsupervised Domain Factorization Network (UnDFN) for thick cloud removal, which allows us to effectively and efficiently exploit the rich SST information of MRSIs. In UnDFN framework, we first factorize RSI for each time node of MRSIs into its corresponding spatial factor and spectral factor. Due to the powerful expressive ability, the untrained neural networks are leveraged to faithfully capture the spatial and spectral factors. Especially, motivated by the low-rankness of the concatenated spatial factors of all time nodes, a low-rank spatial factor module is elaborately designed to effectively and efficiently capture the spatial factors of all time nodes as compared with separately using networks to capture spatial factors for each time node. Extensive experiments on simulated and real MRSIs of different satellites (including Sentinel-2 and Landsat-8) substantiate that the proposed UnDFN achieves state-of-the-art performance in thick cloud removal compared to other methods.

Recently, low-rank prior (LRP) has achieved success in tensor completion, thanks to its ability in capturing global correlations. However, since the presence of dynamic foreground breaks the low-rank assumption, LRP sometimes fails to work for complex videos. How to effectively capture the implicit low-rankness is a tricky challenge in complex video completion. To address this challenge, we propose a video completion model, which organically combines the respective merits of deterministic LRP and deep image prior. Specifically, we first decompose a complex video into the dynamic foreground and static background, and then characterize each component individually. For the dynamic foreground, instead of the hand-crafted prior, we consider the deep foreground prior expressed by U-Net architecture in an unsupervised manner, while we consider the tensor low-rank prior for the static background. Moreover, we develop an efficient alternating direction method of multipliers algorithm under the plug-and-play framework to tackle the resulting model. Different from the classic plug-and-play framework with a pre-trained and fixed network, our algorithm can adaptively update the parameters of network during iterations, which enables the network to have a more flexible expression ability. Experimental results verify that the proposed method can do better than some state-of-the-art tensor completion methods in complex video completion.

Image denoising is often empowered by accurate prior information. In recent years, data-driven neural network priors have shown promising performance for RGB natural image denoising. Compared to classic handcrafted priors (e.g., sparsity and total variation), the “deep priors” are learned using a large number of training samples, which can accurately model the complex image generating process. However, data-driven priors are hard to acquire for hyperspectral images (HSIs) due to the lack of training data. A remedy is to use the so-called unsupervised deep image prior (DIP). Under the unsupervised DIP framework, it is hypothesized and empirically demonstrated that proper neural network structures are reasonable priors of certain types of images, and the network weights can be learned without training data. Nonetheless, the most effective unsupervised DIP structures were proposed for natural images instead of HSIs. The performance of unsupervised DIP-based HSI denoising is limited by a couple of serious challenges, namely network structure design and network complexity. This work puts forth an unsupervised DIP framework that is based on the classic spatiospectral decomposition of HSIs. Utilizing the so-called linear mixture model of HSIs, two types of unsupervised DIPs, that is, U-Net-like network and fully connected networks, are employed to model the abundance maps and endmembers contained in the HSIs, respectively. This way, empirically validated unsupervised DIP structures for natural images can be easily incorporated for HSI denoising. Besides, the decomposition also substantially reduces network complexity. An efficient alternating optimization algorithm is proposed to handle the formulated denoising problem. Simulated and real data experiments are employed to showcase the effectiveness of the proposed approach.

Hyperspectral image (HSI) denoising is a fundamental task in remote sensing image processing, which is helpful for HSI subsequent applications, such as unmixing and classification. Thanks to the powerful representation ability of untrained deep neural networks, deep image prior (DIP)-based methods achieve tremendous successes in image processing (e.g., denoising and inpainting). However, DIP-based methods neglect the tensor low-rank prior of the underlying HSI which will be beneficial to capturing the global structure of the underlying HSI. To address this issue, we propose a novel model for HSI denoising, which can simultaneously take respective advantages of the tensor low-rank prior and the deep spatial-spectral prior. The tensor low-rank prior leads to a better global structure and the deep spatial-spectral prior is complementary to preserve better local details. On the one hand, we adopt low-rank tensor ring (TR) decomposition to characterize the tensor low-rank prior and capture the global structure of the underlying HSI. On the other hand, we employ untrained deep neural networks to flexibly represent the deep spatial-spectral prior and capture the local details of the underlying HSI. To solve the proposed model, we develop an efficient alternating minimization algorithm. Experimental results on simulated and real data validate the advantages of the proposed model in HSI denoising. Compared with state-of-the-art HSI denoising methods, the proposed method preserves better local details and the global structure of the underlying HSI.

Recently, unsupervised deep learning-based methods have shown an empirical success in hyperspectral images (HSIs) denoising, profiting from the strong representation ability of convolutional neural networks. However, these methods only can describe the local structure of the spatial dimension, which is restricted to very limited local receptive fields. To overcome this difficulty, a novel HSIs denoising model based on the deep image prior (DIP) framework is proposed by adding a tensor ring (TR) decomposition, which can enlarge the receptive field of the spatial dimension and capture global spectral correlation simultaneously. Unlike the previous DIP framework that directly minimizes the objective function, we develop an algorithm based on proximal alternating minimization to decouple the model into the classic DIP framework and TR cores least-squares problems, which are easy to solve. Experimental results verify that the proposed DIP- TR compares favorably with compared methods in terms of quality metrics and visual performance.

This paper addresses the tensor completion problem, which aims to recover missing information of multi-dimensional images. How to represent a low-rank structure embedded in the underlying data is the key issue in tensor completion. In this work, we suggest a novel low-rank tensor representation based on coupled transform, which fully exploits the spatial multi-scale nature and redundancy in spatial and spectral/temporal dimensions, leading to a better low tensor multi-rank approximation. More precisely, this representation is achieved by using two-dimensional framelet transform for the two spatial dimensions, one/two-dimensional Fourier transform for the temporal/spectral dimension, and then Karhunen-Loéve transform (via singular value decomposition) for the transformed tensor. Based on this low-rank tensor representation, we formulate a novel low-rank tensor completion model for recovering missing information in multi-dimensional visual data, which leads to a convex optimization problem. To tackle the proposed model, we develop the alternating directional method of multipliers (ADMM) algorithm tailored for the structured optimization problem. Numerical examples on color images, multispectral images, and videos illustrate that the proposed method outperforms many state-of-the-art methods in qualitative and quantitative aspects.