Implicit neural representation (INR) has emerged as a powerful paradigm for visual data representation. However, classical INR methods represent data in the original space mixed with different frequency components, and several feature encoding parameters (e.g., the frequency parameter $\omega$ω or the rank $R$R) need manual configurations. In this work, we propose a self-evolving cross-frequency INR using the Haar wavelet transform (termed CF-INR), which decouples data into four frequency components and employs INRs in the wavelet space. CF-INR allows the characterization of different frequency components separately, thus enabling higher accuracy for data representation. To more precisely characterize cross-frequency components, we propose a cross-frequency tensor decomposition paradigm for CF-INR with self-evolving parameters, which automatically updates the rank parameter $R$R and the frequency parameter $\omega$ω for each frequency component through self-evolving optimization. This self-evolution paradigm eliminates the laborious manual tuning of these parameters, and learns a customized cross-frequency feature encoding configuration for each dataset. We evaluate CF-INR on a variety of visual data representation and inverse imaging problems, including image regression, inpainting, denoising, and cloud removal. Extensive experiments demonstrate that CF-INR outperforms state-of-the-art methods in each case.

Existing arbitrary-scale super-resolution (ASSR) methods suffer from quadratic computational complexity w.r.t. image scale due to the reliance on multi-layer perceptrons (MLPs) to query dense spatial coordinate matrices. The inefficiency becomes particularly pronounced when extending to high-dimensional imaging modalities. To address these limitations, we propose a novel functional tensor decomposition (FTD) framework that fundamentally reconfigures the computational paradigm for ASSR. Specifically, we propose 1) a separation mechanism that employs distinct MLPs to query separable spatial coordinate vectors, substantially reducing decoder MLP invocations, and 2) functional tensor Tucker or CP decompositions for efficient factor matrix integration. The FTD framework delivers three key advantages: 1) Superior scalability to high-dimensional imaging modalities, such as hyperspectral images (HSIs), by virtue of the FTD design; 2) Significantly enhanced inference speed across scales; 3) Faster convergence towards a desired training model. Extensive experiments validate FTD's exceptional performance in HSI joint spatial-spectral ASSR, achieving up to 90.04% reduction in inference time and substantial performance improvements. For conventional image ASSR, our method improves both inference speed and convergence efficiency, achieving up to 88.82% inference time reduction and superior few-shot generalization capabilities due to faster convergence.

Many real-world data are inherently multi-dimensional, e.g., color images, videos, and hyperspectral images. How to effectively and compactly represent these multi-dimensional data within a unified framework is an important pursuit. Previous methods focus on tensor factorizations, convolutional networks, or diffusion models for multi-dimensional data representation, which may not fully utilize inherent data structures and may lead to redundant parameters. In this work, we propose a Deep Rank-One Tensor Functional Factorization (DRO-TFF), which internally utilizes more comprehensive data priors facilitated by much fewer parameters. Concretely, our DRO-TFF consists of three organically integrated blocks: compact rank-one factorizations in the spatial domain, a deep transform to capture underlying low-dimensional structures, and smooth factors parameterized by implicit neural representations. Through a series of theoretical analysis, we show the rich data priors encoded in the DRO-TFF structure, e.g., Lipschitz smoothness and low-rankness. Extensive experiments on multi-dimensional data recovery problems, such as image and video inpainting, image denoising, and hyperspectral mixed noise removal, showcase the effectiveness of the proposed method.

Spatial transcriptomics (ST) are emerging technologies that reveal spatial distributions of gene expressions within tissues, serving as important ways to uncover biological insights. However, the irregular spatial profiles and variability of genes make it challenging to integrate spatial information with gene expression under a computational framework. Current algorithms mostly utilize spatial graph neural networks to encode spatial information, which may incur increased computational costs and may not be flexible enough to depict complex spatial configurations. In this study, we introduce a concise yet effective representation framework, STINR, for deciphering ST data. STINR leverages an implicit neural representation (INR) to continuously represent ST data, which efficiently characterizes spatial and slice-wise correlations of ST data by inheriting the implicit smoothness of INR. STINR allows easier integration of multiple slices and multi-omics without any alignment, and serves as a potent tool for various biological tasks including gene imputation, gene denoising, spatial domain detection, and cell-type deconvolution stemed from ST data. In particular, STINR identifies the thinnest cortex layer in the dorsolateral prefrontal cortex which previous methods were unable to achieve, and more accurately identifies tumor regions in the human squamous cell carcinoma, showcasing its practical value for biological discoveries. Code at https://github.com/YisiLuo/STINR.

The nuclear norm (NN) has been widely explored in matrix recovery problems, such as Robust PCA and matrix completion, leveraging the inherent global low-rank structure of the data. In this study, we introduce a new modified nuclear norm (MNN) framework, where the MNN family norms are defined by adopting suitable transformations and performing the NN on the transformed matrix. The MNN framework offers two main advantages: (1) it jointly captures both local information and global low-rankness without requiring trade-off parameter tuning; (2) under mild assumptions on the transformation, we provide theoretical recovery guarantees for both Robust PCA and MC tasks—an achievement not shared by existing methods that combine local and global information. Thanks to its general and flexible design, MNN can accommodate various proven transformations, enabling a unified and effective approach to structured low-rank recovery. Extensive experiments demonstrate the effectiveness of our method. Code and supplementary material are available at https://github.com/andrew-pengjj/modified_nuclear_norm.

Online tensor decompositions are powerful and proven techniques that address the challenges in processing high-velocity streaming tensor data, such as traffic flow and weather system. The main aim of this work is to propose a novel online functional tensor decomposition (OFTD) framework, which represents a spatial-temporal continuous function using the CP tensor decomposition parameterized by coordinate-based implicit neural representations (INRs). The INRs allow for natural characterization of continually expanded streaming data by simply adding new coordinates into the network. Particularly, our method transforms the classical online tensor decomposition algorithm into a more dynamic continual learning paradigm of updating the INR weights to fit the new data without forgetting the previous tensor knowledge. To this end, we introduce a long-tail memory replay method that adapts to the local continuity property of INR. Extensive experiments for streaming tensor completion using traffic, weather, user-item, and video data verify the effectiveness of the OFTD approach for streaming data analysis. This endeavor serves as a pivotal inspiration for future research to connect classical online tensor tools with continual learning paradigms to better explore knowledge underlying streaming tensor data.

Nonlocal self-similarity (NSS) is an important prior that has been successfully applied in multi-dimensional data processing tasks, e.g., image and video recovery. However, existing NSS-based methods are solely suitable for meshgrid data such as images and videos, but are not suitable for emerging off-meshgrid data, e.g., point cloud and weather data. In this work, we revisit the NSS from the continuous representation perspective and propose a novel Continuous Representation-based NonLocal method (termed as CRNL), which has two innovative features as compared with classical nonlocal methods. First, based on the continuous representation, our CRNL unifies the measure of self-similarity for on-meshgrid and off-meshgrid data and thus is naturally suitable for both of them. Second, the nonlocal continuous groups can be more compactly and efficiently represented by the coupled low-rank function factorization, which simultaneously exploits the similarity within each group and across different groups, while classical nonlocal methods neglect the similarity across groups. This elaborately designed coupled mechanism allows our method to enjoy favorable performance over conventional NSS methods in terms of both effectiveness and efficiency. Extensive multi-dimensional data processing experiments on-meshgrid (e.g., image inpainting and image denoising) and off-meshgrid (e.g., weather data prediction and point cloud recovery) validate the versatility, effectiveness, and efficiency of our CRNL as compared with state-of-the-art methods.

Abstract. Recently, we have witnessed the success of total variation (TV) for many imaging applications. However, traditional TV is defined on the original pixel domain, which limits its potential. In this work, we suggest a new TV regularization defined on the neural domain. Concretely, the discrete data is implicitly and continuously represented by a deep neural network (DNN), and we use the derivatives of DNN outputs with respect to (w.r.t.) input coordinates to capture local correlations of data. As compared with classical TV on the original domain, the proposed TV on the neural domain (termed NeurTV) enjoys the following advantages. First, NeurTV is free of discretization error induced by the discrete difference operator. Second, NeurTV is not limited to meshgrid but is suitable for both meshgrid and non-meshgrid data. Third, NeurTV can more exactly capture local correlations across data for any direction and any order of derivatives attributed to the implicit and continuous nature of neural domain. We theoretically reinterpret NeurTV under the variational approximation framework, which allows us to build the connection between NeurTV and classical TV and inspires us to develop variants (e.g., space-variant NeurTV). Extensive numerical experiments with meshgrid data (e.g., color and hyperspectral images) and non-meshgrid data (e.g., point clouds and spatial transcriptomics) showcase the effectiveness of the proposed methods.

Implicit neural representation (INR) has emerged as a powerful representation for data (e.g., multispectral images and videos). Previously, most INR methods directly represent data in the original space. However, since different frequency components are mixed in the original space, it is difficult to capture these frequency components simultaneously and accurately. To alleviate this limitation, we suggest a new frequency-aware implicit neural representation (FA-INR) working in a physically interpretable and learnable frequency space by cleverly introducing an extra frequency dimension, which allows us to readily decouple and modulate different frequency components, leading to a more accurate characterization of different frequency components in a divide-and-conquer manner. Specifically, the FA-INR consists of two important modules, i.e., the frequency module and the integration module. In the frequency module, we propose a new low-rank tensor frequency function to compactly and continuously represent the latent frequency space. In the integration module, different frequency components are adaptively integrated back to the original space. Extensive experiments on various multi-dimensional data, including multispectral images, color videos, and light field data, demonstrate that the proposed FA-INR significantly outperforms the state-of-the-art INR methods, especially for characterizing high-frequency components (e.g., textures and edges).

Full waveform inversion (FWI) is pivotal for exploring subsurface structures and physical parameters. However, classical FWI methods often experience challenges like cycle skipping and non-linearity, necessitating accurate initial velocity models. Pure data-driven approaches based on deep learning are constrained by limited real labeled data and inadequate generalization, hindering practical applications. To address these issues, we propose an unsupervised physics-informed machine learning FWI with low-rank implicit neural representation (termed LR-IFWI), which utilizes a low-rank matrix factorization parameterized by coordinate-based neural networks, continuously and compactly representing the velocity model by implicitly encoding low-rank properties and smoothness. Our method has three crucial advantages: (i) LR-IFWI considerably improves inversion accuracy and shows steadier inversion convergence behavior; (ii) LR-IFWI can reduce the number of iterations for comparable inversion accuracy, improving efficiency attributed to the compact low-rank representation; (iii) LR-IFWI has better robustness, alleviating the dependence on the initial models and improving noise resistance due to the physical constraints and low-rank neural representation. Numerical tests on the two-dimensional Marmousi model demonstrate that LR-IFWI achieves efficient and accurate inversion with fewer iterations and greater precision when utilizing smooth, linear, and random initial velocity models. Further experiments with noisy seismic data, missing low-frequency components, and more challenging velocity models, such as 2D SEG/EAGE Salt and Overthrust models, highlight its robustness and generalization.

For infrared small target detection tasks, deep unfolding techniques have demonstrated effectiveness and practical value. However, existing methods generally emphasize the low-rankness of background and the sparsity of targets within the robust principal component analysis (RPCA) framework, which may overlook the intrinsic gradient prior information existed in background. To address the challenges of complex background estimation and accurate small target detection, we propose a gradient prior-guided deep unfolding network, termed the correlated total variation network (CTVNet). First, we introduce a correlated total variation regularization to simultaneously characterize the low-rankness and local smoothness of the background, and transform it into the estimation of gradient maps. Subsequently, we employ a multi-scale feature fusion network to thoroughly extract gradient priors, replacing the complex and limited analytical computation of gradient correlations. Finally, we unfold the designed iterative algorithm using alternating direction method of multipliers (ADMM) into a learnable network, where each module corresponds to a specific operator within the iterative process, and all parameters are learnable. By training the network end-to-end, the learnable modules can be automatically optimized to better separate the background and the target. Extensive experimental results demonstrate that our proposed method achieves competitive performance compared to several state-of-the-art algorithms while exhibiting superior performance and generalization capabilities on both in-distribution and out-of-distribution data. Our code is available at https://github.com/AuroraPei/CTVNet.

Seismic data inevitably suffers from random noise due to environmental contributors, which seriously affects subsequent processing and analysis. Deep learning has been a successful tool for seismic data random noise attenuation. Due to the scarcity of clean labels in real scenarios, researchers have attached more attention to self-supervised methods without paired training data. However, most self-supervised methods are costly in computations and thus are inefficient for practical large data volume implementation. In this paper, we propose a novel self-supervised method for seismic random noise attenuation by designing a Kolmogorov-Arnold network (KAN)-empowered neural low-rank representation (NLRR) method. Specifically, the proposed method adopts a compact tensor factorization parameterized by implicit neural representations to efficiently encode both low-rank and smooth priors of seismic data into the model. Moreover, we introduce generalized KANs by using multiple sinusoidal activation functions with different frequencies, serving as factor functions of NLRR to empower its representation ability. Extensive experiments on synthetic and field seismic data demonstrate the clear superiority of our method in terms of efficiency and efficacy over several traditional and deep learning-based methods for random noise attenuation. Specifically, our method reduces over 90% execution time against existing self-supervised methods while still achieving evidently better denoising results.

Recently, the transform-based tensor representation has attracted increasing attention in multimedia data (e.g., images and videos) recovery problems, which consists of two indispensable components, i.e., the transform and the characterization. Previously, the development of transform-based tensor representation has focused mainly on the transform perspective. Although several attempts have considered shallow matrix factorization (e.g., singular value decomposition and nonnegative matrix factorization) for characterizing the frontal slices of the transformed tensor (termed the latent tensor), the faithful characterization perspective has been underexplored. To address this issue, we propose a unifiedDeepTensorRepresentation (DTR) framework by synergistically combining the deep latent generative module and the deep transform module. Especially, the deep latent generative module can faithfully generate the latent tensor as compared with shallow matrix factorization. The new DTR framework not only allows us to better understand the classical shallow representations but also leads us to explore new representations. To examine the representation capability of the proposed DTR, we consider the representative multidimensional data recovery task and suggest an unsupervised DTR-based multidimensional data recovery model. Extensive experiments demonstrate that DTR achieves superior performance compared to the state-of-the-art methods from both quantitative and qualitative aspects, especially for fine detail recovery.

Since higher-order tensors are naturally suitable for representing multi-dimensional data in real-world, e.g., color images and videos, low-rank tensor representation has become one of the emerging areas in machine learning and computer vision. However, classical low-rank tensor representations can solely represent multi-dimensional discrete data on meshgrid, which hinders their potential applicability in many scenarios beyond meshgrid. To break this barrier, we propose a low-rank tensor function representation (LRTFR) parameterized by multilayer perceptrons (MLPs), which can continuously represent data beyond meshgrid with powerful representation abilities. Specifically, the suggested tensor function, which maps an arbitrary coordinate to the corresponding value, can continuously represent data in an infinite real space. Parallel to discrete tensors, we develop two fundamental concepts for tensor functions, i.e., the tensor function rank and low-rank tensor function factorization, and utilize MLPs to paramterize factor functions of the tensor function factorization. We theoretically justify that both low-rank and smooth regularizations are harmoniously unified in LRTFR, which leads to high effectiveness and efficiency for data continuous representation. Extensive multi-dimensional data recovery applications arising from image processing (image inpainting and denoising), machine learning (hyperparameter optimization), and computer graphics (point cloud upsampling) substantiate the superiority and versatility of our method as compared with state-of-the-art methods. Especially, the experiments beyond the original meshgrid resolution (hyperparameter optimization) or even beyond meshgrid (point cloud upsampling) validate the favorable performances of our method for continuous representation.

Hyperspectral image (HSI) denoising is an essential preprocessing step for downstream applications. Fully characterizing the spatial-spectral priors of HSI is crucial for HSI denoising tasks. In recent years, denoising methods based on low-rank subspaces have garnered attention. Within the low-rank decomposition framework (LRDF), the restoration of HSIs can be formulated as a problem of restoring two subspace factors. Since the rank of the HSI data has been predetermined by LRDF, subspace-based methods have already characterized the spectral low-rankness information. Next, subspace-based methods only need to encode spatial priors for HSIs. Existing subspace-based methods either rely on a manual-designed regularization or a pre-trained deep neural network. The former fails to fully capture the intrinsic priors of the HSI, while the latter may encounter generalization issues. Inspired by the unsupervised deep image prior (DIP) technique, this article proposes a double subspace deep prior (DSDP) model to track the mentioned issues. In this model, the two subspace factors are parallelly represented by two deep neural networks. By incorporating popular attention modules into classical convolutional neural networks, the well-designed subspace factor neural network can effectively capture the deep prior of the two subspace factors separately from each HSI in an unsupervised manner. Additionally, the total variation (TV) regularizer is added to constrain the generation of the subspace factor neural network, and further to ensure the effectiveness and robustness of the parameter learning process. Extensive experiments demonstrate that our method outperforms a series of competing methods.

Recently, the transform-based tensor nuclear norm (TNN) methods have shown promising performance and drawn increasing attention in tensor completion (TC) problems. The main idea of these methods is to exploit the low-rank structure of frontal slices of the tensor under the transform. However, the transforms in TNN methods usually treat all modes equally and do not consider the different traits of different modes (i.e., spatial and spectral/temporal modes). To address this problem, we suggest a new low-rank tensor representation based on the coupled nonlinear transform (called CoNoT) for a better low-rank approximation. Concretely, spatial and spectral/temporal transforms in the CoNoT, respectively, exploit the different traits of different modes and are coupled together to boost the implicit low-rank structure. Here, we use the convolutional neural network (CNN) as the CoNoT, which can be learned solely from an observed multidimensional image in an unsupervised manner. Based on this low-rank tensor representation, we build a new multidimensional image completion model. Moreover, we also propose an enhanced version (called Ms-CoNoT) to further exploit the spatial multiscale nature of real-world data. Extensive experiments on real-world data substantiate the superiority of the proposed models against many state-of-the-art methods both qualitatively and quantitatively.

Recently, tensor singular value decomposition (t-SVD) has emerged as a promising tool for hyperspectral image (HSI) processing. In the t-SVD, there are two key building blocks: (i) the low-rank enhanced transform and (ii) the accompanying low-rank characterization of transformed frontal slices. Previous t-SVD methods mainly focus on the developments of (i), while neglecting the other important aspect, i.e., the exact characterization of transformed frontal slices. In this letter, we exploit the potentiality in both building blocks by leveraging the Hierarchical nonlinear transform and the Hierarchical matrix factorization to establish a new Tensor Factorization (termed as H2TF). Compared to shallow counter partners, e.g., low-rank matrix factorization or its convex surrogates, H2TF can better capture complex structures of transformed frontal slices due to its hierarchical modeling abilities. We then suggest the H2TF-based HSI denoising model and develop an alternating direction method of multipliers-based algorithm to address the resultant model. Extensive experiments validate the superiority of our method over state-of-the-art HSI denoising methods.

Seismic data often undergoes severe noise due to environmental factors, which seriously affects subsequent applications. Traditional hand-crafted denoisers such as filters and regularizations utilize interpretable domain knowledge to design generalizable denoising techniques, while their representation capacities may be inferior to deep learning denoisers, which can learn complex and representative denoising mappings from abundant training pairs. However, due to the scarcity of high-quality training pairs, deep learning denoisers may sustain some generalization issues over various scenarios. In this work, we propose a self-supervised method that combines the capacities of deep denoiser and the generalization abilities of hand-crafted regularization for seismic data random noise attenuation. Specifically, we leverage the Self2Self (S2S) learning framework with a trace-wise masking strategy for seismic data denoising by solely using the observed noisy data. Parallelly, we suggest the weighted total variation (WTV) to further capture the horizontal local smooth structure of seismic data. Our method, dubbed as S2S-WTV, enjoys both high representation abilities brought from the self-supervised deep network and good generalization abilities of the hand-crafted WTV regularizer and the self-supervised nature. Therefore, our method can more effectively and stably remove the random noise and preserve the details and edges of the clean signal. To tackle the S2S-WTV optimization model, we introduce an alternating direction multiplier method (ADMM)-based algorithm. Extensive experiments on synthetic and field noisy seismic data demonstrate the effectiveness of our method as compared with state-of-the-art traditional and deep learning-based seismic data denoising methods.

Noise suppression for seismic data can meliorate the quality of many subsequent geophysical tasks. In this work, we propose a novel self-supervised learning method, the deep nonlocal regularizer (DNLR), for 3D seismic denoising. Our DNLR fully exploits the nonlocal self-similarity of seismic data under a self-supervised learning framework for noise attenuation. It can be flexibly combined with different hand-crafted regularizers, e.g., total variation, nuclear norm, and correlated total variation, by performing the regularizer on nonlocal self-similar patches, which more effectively characterizes the intrinsic structures underlying seismic data. Our DNLR can be easily plugged into existing self-supervised denoising methods, e.g., deep image prior and Self2Self, and consistently improve their performance. To make the optimization model tractable, an algorithm based on the alternating direction multiplier method is introduced to solve the DNLR-based seismic denoising problem. Extensive seismic denoising experiments on synthetic and field data validate the superior performances of our DNLR as compared with state-of-the-art model-based and deep learning seismic denoising methods. Code is available at https://github.com/XuZitai/DNLR.

Inverse problems in multi-dimensional imaging, e.g., completion, denoising, and compressive sensing, are challenging owing to the big volume of the data and the inherent illposedness. To tackle these issues, this work unsuper-visedly learns a hierarchical low-rank tensor factorization (HLRTF) by solely using an observed multi-dimensional image. Specifically, we embed a deep neural network (DNN) into the tensor singular value decompositionframe-work and develop the HLRTF, which captures the underlying low-rank structures of multi-dimensional images with compact representation abilities. This DNN herein serves as a nonlinear transform from a vector to another to help obtain a better low-rank representation. Our HLRTF infers the parameters of the DNN and the underlying low-rank structure of the original data from its observation via the gradient descent using a non-reference loss function in an unsupervised manner. To address the vanishing gradient in extreme scenarios, e.g., structural missing pixels, we introduce a parametric total variation regularization to constrain the DNN parameters and the tensor factor parameters with theoretical analysis. We apply our HLRTF for typical inverse problems in multi-dimensional imaging including completion, denoising, and snapshot spectral imaging, which demonstrates its generality and wide applicability. Extensive results illustrate the superiority of our method as compared with state-of-the-art methods.

Image deraining is an important task for subsequent multimedia applications in rainy weather. Traditional deep learning-based methods rely on the quantity and diversity of training data, which is hard to cover all complex real-world rain scenarios. In this work, we propose the first Unsupervised Controllable Network (UConNet) to flexibly tackle different rain scenarios by adaptively controlling the network at the inference stage. Specifically, our unsupervised network takes the physics-based regularizations as the unsupervised loss function. Then, we sensibly derive the relationship between trade-off parameters of the loss function and the weightings of feature maps. Based on this relationship, our learned UConNet can be flexibly customized for different rain scenarios by controlling the weightings of feature maps at the inference stage. Alternatively, these weightings can also be efficiently determined by a learned weightings recommendation network. Extensive experiments for image and video deraining show that our method achieves very promising effectiveness, efficiency, and generalization abilities as compared with state-of-the-art methods.

Recently, transform-based tensor nuclear norm (TNN) minimization methods have received increasing attention for recovering third-order tensors in multi-dimensional imaging problems. The main idea of these methods is to perform the linear transform along the third mode of third-order tensors and then minimize the nuclear norm of frontal slices of the transformed tensor. The main aim of this paper is to propose a nonlinear multilayer neural network to learn a nonlinear transform by solely using the observed tensor in a self-supervised manner. The proposed network makes use of the low-rank representation of the transformed tensor and data-fitting between the observed tensor and the reconstructed tensor to learn the nonlinear transform. Extensive experimental results on different data and different tasks including tensor completion, background subtraction, robust tensor completion, and snapshot compressive imaging demonstrate the superior performance of the proposed method over state-of-the-art methods.

Removing rain streaks in videos has recently received much attention. Existing hand-crafted priors-based methods suffer from limited representation abilities, and supervised deep learning methods need high-quality training data. This paper proposes a novel video rain streaks removal method by reconciling hand-crafted and self-supervised deep priors. The hand-crafted priors include the learned gradient prior, the sparse prior, and the temporal local smooth prior. Meanwhile, a deep convolutional neural network is employed to self-supervisedly capture the deep prior of the clean video without any training data. Our method organically integrates hand-crated priors and self-supervised deep priors to achieve both high generalization abilities and representation abilities. Thus, our method can faithfully remove directional rain streaks in real world videos. To address the resulting model, we introduce an alternating direction multiplier method algorithm. Extensive experiments validate the superiority of our method over state-of-the-art methods.