The manifold hypothesis says that natural high-dimensional data lie on or around a low-dimensional manifold. The recent success of statistical and learning-based methods in very high dimensions empirically supports this hypothesis, suggesting that typical worst-case analysis does not provide practical guarantees. A natural step for analysis is thus to assume the manifold hypothesis and derive bounds that are independent of any ambient dimensions that the data may be embedded in. Theoretical implications in this direction have recently been explored in terms of generalization of ReLU networks and convergence of Langevin methods. In this work, we consider optimal uniform approximations with functions of finite statistical complexity. While upper bounds on uniform approximation exist in the literature using ReLU neural networks, we consider the opposite: lower bounds to quantify the fundamental difficulty of approximation on manifolds. In particular, we demonstrate that the statistical complexity required to approximate a class of bounded Sobolev functions on a compact manifold is bounded from below, and moreover that this bound is dependent only on the intrinsic properties of the manifold, such as curvature, volume, and injectivity radius.

Abstract. We address the optimization problem in a data-driven variational reconstruction framework, where the regularizer is parametrized by an input-convex neural network. While gradient-based methods are commonly used to solve such problems, they struggle to effectively handle nonsmooth problems, which often leads to slow convergence. Moreover, the nested structure of the neural network complicates the application of standard nonsmooth optimization techniques, such as proximal algorithms. To overcome these challenges, we reformulate the problem and eliminate the network’s nested structure. By relating this reformulation to epigraphical projections of the activation functions, we transform the problem into a convex optimization problem that can be efficiently solved using a primal-dual algorithm. We also prove that this reformulation is equivalent to the original variational problem. Through experiments on several imaging tasks, we show that the proposed approach not only outperforms subgradient methods and even accelerated methods in the smooth setting but also facilitates the training of the regularizer itself.

The growing interconnectedness of industrial Internet of Things (IIoT) sensors and devices necessitates intelligent, robust, and lightweight security methods for transmitting private information. In order to overcome this difficulty in IIoT security, we propose PISAStego, a unique privacy-preserving instance-specific self-supervised convolutional autoencoder steganography. In our self-supervised PISAStego, a convolutional autoencoder architecture is used, where the encoder learns to distribute the secret information across visually less perceptible regions of the cover image. This concealment is processed by the reconstruction and smoothness constraints with the loss function in an instance-specific way, whereas the decoder effectively extracts the hidden data from the stego image with minimal loss. leaky rectified linear unit is incorporated to each convolutional layer in PISAStego to perform the nonlinearity to learn complex patterns, improve gradient flow, handle dead neurons, and preserve texture. Unlike conventional optimization-based or deep learning steganography models, the proposed model does not require any labeled data, large datasets, or longer training time. This makes the PISAStego a lightweight and faster algorithm. Our PISAStego algorithm has obtained an average 0.9959 normalized cross-correlation for hiding a$256 \times 256 \times 3$secret image inside a$256 \times 256 \times 3$cover image with 8 bits per pixel embedding capacity. Experimental results validate superior cost-effectiveness, lightweight with faster processing, and higher message hiding capacity compared to existing methods. Our model is lightweight, low in parameters, with higher embedding rate, which is ideal for providing robust authentication and faster covert communication to secure any IIoT system.

Despite impressive empirical performance in various imaging applications, iterative data-driven reconstruction (IDR) schemes such as plug-and-play algorithms and deep unrolling networks can have significant computational limitations, especially for large-scale imaging inverse problems. This is mostly because they need to involve the high-dimensional forward/adjoint operators that are expensive to compute in each iteration. In this work, we propose a new operator sketching framework tailored for designing efficient IDR schemes, which are currently state-of-the-art solutions for imaging inverse problems. Our framework performs dimensionality reduction in both image and measurement data domains, leading to efficient computations. Using this framework, we derive several accelerated IDR schemes, such as the plug-and-play multi-stage sketched gradient (PnP-MS2G) and sketching-based primal-dual (LSPD and Sk-LSPD) deep unrolling networks. Our experiments on X-ray CT image reconstruction demonstrate the remarkable effectiveness of the proposed sketched IDR methods.

We propose to learn a data-adaptive convex regularizer, which is parameterized using an input-convex neural network (ICNN), for variational image reconstruction. The regularizer parameters are learned adversarially by telling apart clean images from the artifact-ridden ones in a training dataset. Convexity of the regularizer is theoretically and practically important since (i) one can establish well-posedness guarantees for the corresponding variational reconstruction problem and (ii) devise provably convergent optimization algorithms for reconstruction. In particular, the resulting method is shown to be convergent in the sense of regularization and can be solved provably using a gradient-based solver. To demonstrate the performance of our approach for solving inverse problems, we consider deblurring natural images and reconstruction in X-ray computed tomography (CT) and show that the proposed convex regularizer is on par with and sometimes superior to state-of-the-art classical and data-driven techniques for inverse problems, especially with severely ill-posed forward operators (such as in limited-angle tomography).

Variational regularisation is the primary method for solving inverse problems, and recently there has been considerable work leveraging deeply learned regularisation for enhanced performance. However, few results exist addressing the convergence of such regularisation, particularly within the context of critical points as opposed to global minimisers. In this paper, we present a generalised formulation of convergent regularisation in terms of critical points, and show that this is achieved by a class of weakly convex regularisers. We prove convergence of the primal-dual hybrid gradient method for the associated variational problem, and, given a Kurdyka-Łojasiewicz condition, an $\mathcal{O}(\log{k}/k)$ ergodic convergence rate. Finally, applying this theory to learned regularisation, we prove universal approximation for input weakly convex neural networks (IWCNN), and show empirically that IWCNNs can lead to improved performance of learned adversarial regularisers for computed tomography (CT) reconstruction.

Abstract. Bayesian methods for solving inverse problems are a powerful alternative to classical methods since the Bayesian approach offers the ability to quantify the uncertainty in the solution. In recent years, data-driven techniques for solving inverse problems have also been remarkably successful, due to their superior representation ability. In this work, we incorporate data-based models into a class of Langevin-based sampling algorithms for Bayesian inference in imaging inverse problems. In particular, we introduce NF-ULA (normalizing flow-based unadjusted Langevin algorithm), which involves learning a normalizing flow (NF) as the image prior. We use NF to learn the prior because a tractable closed-form expression for the log prior enables the differentiation of it using autograd libraries. Our algorithm only requires a normalizing flow-based generative network, which can be pretrained independently of the considered inverse problem and the forward operator. We perform theoretical analysis by investigating the well-posedness and nonasymptotic convergence of the resulting NF-ULA algorithm. The efficacy of the proposed NF-ULA algorithm is demonstrated in various image restoration problems such as image deblurring, image inpainting, and limited-angle X-ray computed tomography reconstruction. NF-ULA is found to perform better than competing methods for severely ill-posed inverse problems.

Abstract. Plug-and-Play (PnP) methods are a class of efficient iterative methods that aim to combine data fidelity terms and deep denoisers using classical optimization algorithms, such as ISTA or ADMM, with applications in inverse problems and imaging. Provable PnP methods are a subclass of PnP methods with convergence guarantees, such as fixed point convergence or convergence to critical points of some energy function. Many existing provable PnP methods impose heavy restrictions on the denoiser or fidelity function, such as nonexpansiveness or strict convexity, respectively. In this work, we propose a novel algorithmic approach incorporating quasi-Newton steps into a provable PnP framework based on proximal denoisers, resulting in greatly accelerated convergence while retaining light assumptions on the denoiser. By characterizing the denoiser as the proximal operator of a weakly convex function, we show that the fixed points of the proposed quasi-Newton PnP algorithm are critical points of a weakly convex function. Numerical experiments on image deblurring and super-resolution demonstrate 2–8x faster convergence as compared to other provable PnP methods with similar reconstruction quality.

Grating interferometry CT (GI-CT) is a promising technology that could play an important role in future breast cancer imaging. Thanks to its sensitivity to refraction and small-angle scattering, GI-CT could augment the diagnostic content of conventional absorption-based CT. However, reconstructing GI-CT tomographies is a complex task because of ill problem conditioning and high noise amplitudes. It has previously been shown that combining data-driven regularization with iterative reconstruction is promising for tackling challenging inverse problems in medical imaging. In this work, we present an algorithm that allows seamless combination of data-driven regularization with quasi-Newton solvers, which can better deal with ill-conditioned problems compared to gradient descent-based optimization algorithms. Contrary to most available algorithms, our method applies regularization in the gradient domain rather than in the image domain. This comes with a crucial advantage when applied in conjunction with quasi-Newton solvers: the Hessian is approximated solely based on denoised data. We apply the proposed method, which we call GradReg, to both conventional breast CT and GI-CT and show that both significantly benefit from our approach in terms of dose efficiency. Moreover, our results suggest that thanks to its sharper gradients that carry more high spatial-frequency content, GI-CT can benefit more from GradReg compared to conventional breast CT. Crucially, GradReg can be applied to any image reconstruction task which relies on gradient-based updates.

Learning-to-optimize is an emerging framework that leverages training data to speed up the solution of certain optimization problems. One such approach is based on the classical mirror descent algorithm, where the mirror map is modelled using input-convex neural networks. In this work, we extend this functional parameterization approach by introducing momentum into the iterations, based on the classical accelerated mirror descent. Our approach combines short-time accelerated convergence with stable long-time behavior. We empirically demonstrate additional robustness with respect to multiple parameters on denoising and deconvolution experiments.

Recent advances in generative adversarial networks (GANs) have opened up the possibility of generating high-resolution photo-realistic images that were impossible to produce previously. The ability of GANs to sample from high-dimensional distributions has naturally motivated researchers to leverage their power for modeling the image prior in inverse problems. We extend this line of research by developing a Bayesian image reconstruction framework that utilizes the full potential of a pre-trained StyleGAN2 generator, which is the currently dominant GAN architecture, for constructing the prior distribution on the underlying image. Our proposed approach, which we refer to as learned Bayesian reconstruction with generative models (L-BRGM), entails joint optimization over the style-code and the input latent code, and enhances the expressive power of a pre-trained StyleGAN2 generator by allowing the style-codes to be different for different generator layers. Considering the inverse problems of image inpainting and super-resolution, we demonstrate that the proposed approach is competitive with, and sometimes superior to, state-of-the-art GAN-based image reconstruction methods.

We propose a new approach for learning end-to-end reconstruction operators based on unpaired training data for ill-posed inverse problems. The proposed method combines the classical variational framework with iterative unrolling and essentially seeks to minimize a weighted combination of the expected distortion in the measurement space and the Wasserstein-1 distance between the distributions of the reconstruction and the ground-truth. More specifically, the regularizer in the variational setting is parametrized by a deep neural network and learned simultaneously with the unrolled reconstruction operator. The variational problem is then initialized with the output of the reconstruction network and solved iteratively till convergence. Notably, it takes significantly fewer iterations to converge as compared to variational methods, thanks to the excellent initialization obtained via the unrolled operator. The resulting approach combines the computational efficiency of end-to-end unrolled reconstruction with the well-posedness and noise-stability guarantees of the variational setting. Moreover, we demonstrate with the example of image reconstruction in X-ray computed tomography (CT) that our approach outperforms state-of-the-art unsupervised methods and that it outperforms or is at least on par with state-of-the-art supervised data-driven reconstruction approaches.

We develop two techniques based on alternating minimization and alternating directions method of multipliers for phase retrieval (PR) by employing a variable-splitting approach in a maximum likelihood estimation framework. This leads to an additional equality constraint, which is incorporated in the optimization framework using a quadratic penalty. Both algorithms are iterative, wherein the updates are computed in closed-form. Experimental results show that: (i) the proposed techniques converge faster than the state-of-the-art PR algorithms; (ii) the complexity is comparable to the state of the art; and (iii) the performance does not depend critically on the choice of the penalty parameter. We also show how sparsity can be incorporated within the variable splitting framework and demonstrate concrete applications to image reconstruction in frequency-domain optical-coherence tomography.

We propose a novel speech denoising framework by minimizing the probability of error (PE), which measures the deviation probability of the estimate from its true value. To develop the minimum PE (MPE) criterion, one requires the knowledge of the noise probability density function (p.d.f.), which may not be available in a parametric form in speech denoising applications. Therefore, we adopt two approaches for modeling the noise p.d.f.: (i) Gaussian modeling based on adaptive variance estimation; and (ii) a Gaussian mixture model (GMM) in view of its approximation capabilities. We consider discrete cosine transform (DCT) domain shrinkage, where the optimum shrinkage parameter is obtained by minimizing an estimate of the PE. A performance assessment for real-world noise types shows that for input signal-to-noise ratios (SNR) greater than 5 dB, the proposed MPE-based point-wise shrinkage estimators outperform three benchmark techniques in terms of segmental SNR and short-time objective intelligibility (STOI) scores.

We consider the problem of signal reconstruction from quadratic measurements that are encoded as +1 or -1 depending on whether they exceed a predetermined positive threshold or not. Binary measurements are fast to acquire and inexpensive in terms of hardware. We formulate the problem of signal reconstruction using a consistency criterion, wherein one seeks to find a signal that is in agreement with the acquired measurements. To enforce consistency, we construct a convex cost using a one-sided quadratic penalty and minimize it using an iterative accelerated projected gradient-descent technique. The projected gradient-descent (PGD) scheme reduces the cost function in each iteration, whereas incorporating momentum into PGD, notwithstanding the lack of such a descent property, exhibits faster convergence than PGD empirically. We refer to the resulting algorithm as binary phase retrieval (BPR). Considering additive white noise contamination prior to quantization, we also derive the Cramér-Rao Bound (CRB) for the binary encoding model. Experimental results demonstrate that the BPR algorithm yields a signal-to-reconstruction error ratio (SRER) of approximately 25 dB in the absence of noise. In the presence of noise prior to quantization, the SRER is within 2 to 3 dB of the CRB.

We address the problem of learning a sparsifying synthesis dictionary over large datasets that occur in numerous signal and image processing applications, such as inpainting, super-resolution, etc. We develop a dictionary learning algorithm that exploits the similarity of the training examples to reduce the training time. Training datasets containing correlated examples typically occur in image processing applications, as the datasets contain the patches extracted from natural images as training vectors. Our algorithm employs a divide- and-conquer approach, where one leverages the correlation within the training examples to segment the dataset into clusters containing similar examples, and learn local dictionaries for each of them. This constitutes the divide step of the algorithm. In the conquer step, a global dictionary is trained using the atoms of the local dictionaries as the training examples. We analyze the run-time complexity and the representation error of the proposed divide-and-conquer dictionary learning algorithm, and compare the performance with the batch and online dictionary learning algorithms, both on synthesized dataset and natural images. The analysis reveals that the proposed algorithm has an asymptotic complexity that is linear and logarithmic in the number of training examples, corresponding to sequential and parallel implementations, respectively.

We address the problem of extending the bandwidth of speech signals, which is of importance to enhance the quality and intelligibility of the telephone speech. The low-pass filtering effect of the telephone communication channels eliminate the high-frequency components of the speech signal, and it is necessary to retrieve those to maintain the speech quality. We adopt a joint-dictionary training approach to recover the missing spectral information. By exploiting the sparsity of the spectrogram frames, the dictionaries for the wide-band (WB) and the corresponding narrow-band (NB) spectrogram frames are trained in a coupled manner in order to learn the mapping from NB to WB frames. We refer to this approach as the joint dictionary training for bandwidth extension (JDTBE). To ensure that the reconstructed bandwidth-extended speech is consistent with the measurement, we propose to apply a suitable affine transformation that depends on the properties of the telephone channel. We study the effect of the choice of sparsity on the quality of the reconstructed speech, for both male and female speakers. A comparison of the proposed JDTBE algorithm with a bandwidth extension technique based on stochastic modeling reveals the superiority of the JDTBE approach in terms of subjective listening test scores.

We address the problem of designing an optimal pointwise shrinkage estimator in the transform domain, based on the minimum probability of error (MPE) criterion. We assume an additive model for the noise corrupting the clean signal. The proposed formulation is general in the sense that it can handle various noise distributions. We consider various noise distributions (Gaussian, Student's-t, and Laplacian) and compare the denoising performance of the estimator obtained with the mean-squared error (MSE)-based estimators. The MSE optimization is carried out using an unbiased estimator of the MSE, namely Stein's Unbiased Risk Estimate (SURE). Experimental results show that the MPE estimator outperforms the SURE estimator in terms of SNR of the denoised output, for low (0-10 dB) and medium values (10-20 dB) of the input SNR.

We address the problem of signal reconstruction from the Fourier transform magnitude of a certain class of two-dimensional (2-D) signals that are characterized by first-order difference equations. We show that when such a signal has a Z-transform that includes the unit sphere in the region of convergence, it can be reconstructed uniquely from the Fourier magnitude. We employ the annihilating filter approach to find the parameters of the rational transfer function.

We address the problem of phase retrieval, which is frequently encountered in optical imaging. The measured quantity is the magnitude of the Fourier spectrum of a function (in optics, the function is also referred to as an object). The goal is to recover the object based on the magnitude measurements. In doing so, the standard assumptions are that the object is compactly supported and positive. In this paper, we consider objects that admit a sparse representation in some orthonormal basis. We develop a variant of the Fienup algorithm to incorporate the condition of sparsity and to successively estimate and refine the phase starting from the magnitude measurements. We show that the proposed iterative algorithm possesses Cauchy convergence properties. As far as the modality is concerned, we work with measurements obtained using a frequency-domain optical-coherence tomography experimental setup. The experimental results on real measured data show that the proposed technique exhibits good reconstruction performance even with fewer coefficients taken into account for reconstruction. It also suppresses the autocorrelation artifacts to a significant extent since it estimates the phase accurately.