Plug-and-play (PnP) methods are extensively used for solving imaging inverse problems by integrating physical measurement models with pre-trained deep denoisers as priors. Score-based diffusion models (SBMs) have recently emerged as a powerful framework for image generation by training deep denoisers to represent the score of the image prior. While both PnP and SBMs use deep denoisers, the score-based nature of PnP is unexplored in the literature due to its distinct origins rooted in proximal optimization. This paper introduces a novel view of PnP as a score-based method, a perspective that enables the re-use of powerful SBMs within classical PnP algorithms without retraining. We present a set of mathematical relationships for adapting popular SBMs as priors within PnP. We show that this approach enables a direct comparison between PnP and SBM-based reconstruction methods using the same neural network as the prior. Code is available at https://github.com/wustl-cig/scorepnp.

Ptychography uses overlapping X-ray diffraction measurements to perform nanometer-scale imaging of objects. Practical illumination sources for ptychography, such as synchrotrons, are not fully coherent, so they exhibit multiple modes. To address this, blind multi-mode ptychography algorithms have been developed that jointly estimate the image along with the multiple probe modes. However, commonly used algorithms, such as DM and SHARP, use a single global estimate of the probe state that does not allow for local spatial variation in the probe.In this paper, we compare a distributed probe state version of the BM-PMACE algorithm to a global probe state version of BM-PMACE as well as to DM and SHARP. Importantly, distributed state BM-PMACE maintains a location-specific probe state that captures spatially varying probe aberrations. Using measured data, we demonstrate that the distributed probe state improves both convergence speed and final image quality compared to the global state algorithms.

Computational wave imaging (CWI) extracts hidden structure and physical properties of a volume of material by analyzing wave signals that traverse that volume. Applications include seismic exploration of the Earth's subsurface, acoustic imaging and non-destructive testing in material science, and ultrasound computed tomography in medicine. Current approaches for solving CWI problems can be divided into two categories: those rooted in traditional physics, and those based on deep learning. Physics-based methods stand out for their ability to provide high-resolution and quantitatively accurate estimates of acoustic properties within the medium. However, they can be computationally intensive and are susceptible to ill-posedness and nonconvexity typical of CWI problems. Machine learning-based computational methods have recently emerged, offering a different perspective to address these challenges. Diverse scientific communities have independently pursued the integration of deep learning in CWI. This review discusses how contemporary scientific machine-learning (ML) techniques, and deep neural networks in particular, have been developed to enhance and integrate with traditional physics-based methods for solving CWI problems. We present a structured framework that consolidates existing research spanning multiple domains, including computational imaging, wave physics, and data science. This study concludes with important lessons learned from existing ML-based methods and identifies technical hurdles and emerging trends through a systematic analysis of the extensive literature on this topic.

Physical and budget constraints often result in irregular sampling, which complicates accurate subsurface imaging. Pre-processing approaches, such as missing trace or shot interpolation, are typically employed to enhance seismic data in such cases. Recently, deep learning has been used to address the trace interpolation problem at the expense of large amounts of training data to adequately represent typical seismic events. Nonetheless, most research in this area has focused on trace reconstruction, with little attention having been devoted to shot interpolation. Furthermore, existing methods assume regularly spaced receivers/sources failing in approximating seismic data from real (irregular) surveys. This work presents a novel shot gather interpolation approach which uses a continuous coordinate-based representation of the acquired seismic wavefield parameterized by a neural network. The proposed unsupervised approach, which we call coordinate-based seismic interpolation (CoBSI), enables the prediction of specific seismic characteristics in irregular land surveys without using external data during neural network training. Experimental results on real and synthetic 3D data validate the ability of the proposed method to estimate continuous smooth seismic events in the time-space and frequency-wavenumber domains, improving sparsity or low-rank-based interpolation methods.

Physical and budget constraints often result in inadequate sampling for accurate subsurface imaging. Preprocessing approaches, such as missing trace interpolation, are typically employed to enhance seismic data in such cases. The compressed sensing (CS) framework has been applied for modeling missing seismic data, which is estimated by sparsity-based computational algorithms. While existing work mainly focuses on recovering missing traces resulting from receiver subsampling, source subsampling has greater economical advantages, as sources are more expensive than receivers. Moreover, stronger image models different from sparsity have not been explored for source recovery. This work presents a consensus equilibrium (CE) approach to recover missing seismic shots, which enables to incorporate various regularization operators modeling different data priors. Simulation results from a real 3-D land seismic dataset demonstrate that the CE approach provides more accurate estimations of the linear and hyperbolic events in the recovered shots, compared with pure sparsity-based reconstructions.

Seismic full-waveform inversion (FWI) is a nonlinear computational imaging technique that can provide detailed estimates of subsurface geophysical properties. Solving the FWI problem can be challenging due to its ill-posedness and high computational cost. In this work, we develop a new hybrid computational approach to solve FWI that combines physics-based models with data-driven methodologies. In particular, we develop a data augmentation strategy that can not only improve the representativity of the training set but also incorporate important governing physics into the training process and, therefore, improve the inversion accuracy. To validate the performance, we apply our method to synthetic elastic seismic waveform data generated from a subsurface geologic model built on a carbon sequestration site at Kimberlina, California. We compare our physics-consistent data-driven inversion method to both purely physics-based and purely data-driven approaches and observe that our method yields higher accuracy and greater generalization ability.

Seismic full-waveform inversion (FWI), which uses iterative methods to estimate high-resolution subsurface models from seismograms, is a powerful imaging technique in exploration geophysics. In recent years, the computational cost of FWI has grown exponentially due to the increasing size and resolution of seismic data. Moreover, it is a nonconvex problem and can encounter local minima due to the limited accuracy of the initial velocity models or the absence of low frequencies in the measurements. To overcome these computational issues, we develop a multiscale data-driven FWI method based on fully convolutional networks (FCNs). In preparing the training data, we first develop a real-time style transform method to create a large set of synthetic subsurface velocity models from natural images. We then develop two convolutional neural networks with encoder–decoder structures to reconstruct the low- and high-frequency components of the subsurface velocity models, separately. To validate the performance of our data-driven inversion method and the effectiveness of the synthesized training set, we compare it with conventional physics-based waveform inversion approaches using both synthetic and field data. These numerical results demonstrate that, once our model is fully trained, it can significantly reduce the computation time and yield more accurate subsurface velocity models in comparison with conventional FWI.

4-D seismic imaging has been widely used in CO2sequestration projects to monitor the fluid flow in the volumetric subsurface region that is not sampled by wells. Ideally, real-time monitoring and near-future forecasting would provide site operators with great insights to understand the dynamics of the subsurface reservoir and assess any potential risks. However, due to obstacles such as high deployment cost, availability of acquisition equipment, exclusion zones around surface structures, only very sparse seismic imaging data can be obtained during monitoring. That leads to an unavoidable and growing knowledge gap over time. The operator needs to understand the fluid flow throughout the project lifetime and the seismic data are only available at a limited number of times. This is insufficient for understanding reservoir behavior. To overcome those challenges, we have developed spatio-temporal neural-network-based models that can produce high-fidelity interpolated or extrapolated images effectively and efficiently. Specifically, our models are built on an autoencoder, and incorporate the long short-term memory (LSTM) structure with a new loss function regularized by optical flow. We validate the performance of our models using real 4-D post-stack seismic imaging data acquired at the Sleipner CO2sequestration field. We employ two different strategies in evaluating our models. Numerically, we compare our models with different baseline approaches using classic pixel-based metrics. We also conduct a blind survey and collect a total of 20 responses from domain experts to evaluate the quality of data generated by our models. Via both numerical and expert evaluation, we conclude that our models can produce high-quality 2-D/3-D seismic imaging data at a reasonable cost, offering the possibility of real-time monitoring or even near-future forecasting of the CO2storage reservoir.

Seismic full-waveform inversion (FWI) techniques aim to find a high-resolution subsurface geophysical model provided with waveform data. Some recent effort in data-driven FWI has shown some encouraging results in obtaining 2-D velocity maps. However, due to high computational complexity and large memory consumption, the reconstruction of 3-D high-resolution velocity maps via deep networks is still a great challenge. In this article, we present InversionNet3D (InvNet3D), an efficient and scalable encoder–decoder network for 3-D FWI. The proposed method employs group convolution in the encoder to establish an effective hierarchy for learning information from multiple sources while cutting down unnecessary parameters and operations at the same time. The introduction of invertible layers further reduces the memory consumption of intermediate features during training and, thus, enables the development of deeper networks with more layers and higher capacity as required by different application scenarios. Experiments on the 3-D Kimberlina dataset demonstrate that InvNet3D achieves state-of-the-art reconstruction performance with lower computational cost and lower memory footprint compared to the baseline.

Deep unfolding networks are rapidly gaining attention for solving imaging inverse problems. However, the computational and memory complexity of existing deep unfolding networks scales with the size of the full measurement set, limiting their applicability to certain large-scale imaging inverse problems. We propose SCRED-Net as a novel methodology that introduces a stochastic approximation to the unfolded regularization by denoising (RED) algorithm. Our method uses only a subset of measurements within each cascade block, making it scalable to a large number of measurements for efficient end-to-end training. We present numerical results showing the effectiveness of SCRED-Net on intensity diffraction tomography (IDT) and sparse-view computed tomography (CT). Our results show that SCRED-Net matches the performance of a batch deep unfolding network at a fraction of training and operational complexity.

Energy resolved neutron imaging (ERNI) is an advanced neutron radiography technique capable of non-destructively extracting spatial isotopic information within a given material. Energy-dependent radiography image sequences can be created by utilizing neutron time-of-flight techniques. In combination with uniquely characteristic isotopic neutron cross-section spectra, isotopic areal densities can be determined on a per-pixel basis, thus resulting in a set of areal density images for each isotope present in the sample. By preforming ERNI measurements over several rotational views, an isotope decomposed D computed tomograpy is possible.We demonstrate a method involving a robust and automated background estimation based on a linear programming formulation. The extremely high noise due to low count measurements is overcome using a sparse coding approach. It allows for a significant computation time improvement, from weeks to a few hours compared to existing neutron evaluation tools, enabling at the present stage a semi-quantitative, user-friendly routine application.

The plug-and-play priors (PnP) and regularization by denoising (RED) methods have become widely used for solving inverse problems by leveraging pre-trained deep denoisers as image priors. While the empirical imaging performance and the theoretical convergence properties of these algorithms have been widely investigated, their recovery properties have not previously been theoretically analyzed. We address this gap by showing how to establish theoretical recovery guarantees for PnP/RED by assuming that the solution of these methods lies near the fixed-points of a deep neural network. We also present numerical results comparing the recovery performance of PnP/RED in compressive sensing against that of recent compressive sensing algorithms based on generative models. Our numerical results suggest that PnP with a pre-trained artifact removal network provides significantly better results compared to the existing state-of-the-art methods.

We present a new algorithm for estimating the Point Spread Function (PSF) in wide-field astronomical images with extreme source crowding. Robust and accurate PSF estimation in crowded astronomical images dramatically improves the fidelity of astrometric and photometric measurements extracted from wide-field sky monitoring imagery. Our radically new approach utilizes convolutional sparse representations to model the continuous functions involved in the image formation. This approach avoids the need to detect and precisely localize individual point sources that is shared by existing methods. In experiments involving simulated astronomical imagery, it significantly outperforms the recent alternative method with which it is compared.

Plug-and-play priors (PnP) is a methodology for regularized image reconstruction that specifies the prior through an image denoiser. While PnP algorithms are well understood for denoisers performing maximum a posteriori probability (MAP) estimation, they have not been analyzed for the minimum mean squared error (MMSE) denoisers. This letter addresses this gap by establishing the first theoretical convergence result for the iterative shrinkage/thresholding algorithm (ISTA) variant of PnP for MMSE denoisers. We show that the iterates produced by PnP-ISTA with an MMSE denoiser converge to a stationary point of some global cost function. We validate our analysis on sparse signal recovery in compressive sensing by comparing two types of denoisers, namely the exact MMSE denoiser and the approximate MMSE denoiser obtained by training a deep neural net.

The plug-and-play priors (PnP) framework has been recently shown to achieve state-of-the-art results in regularized image reconstruction by leveraging a sophisticated denoiser within an iterative algorithm. In this paper, we propose a new online PnP algorithm for Fourier ptychographic microscopy (FPM) based on the accelerated proximal gradient method (APGM). Specifically, the proposed algorithm uses only a subset of measurements, which makes it scalable to a large set of measurements. We validate the algorithm by showing that it can lead to significant performance gains on both simulated and experimental data.

There has been a growing interest in the use of data-driven regularizers to solve inverse problems associated with computational imaging systems. The convolutional sparse representation model has recently gained attention, driven by the development of fast algorithms for solving the dictionary learning and sparse coding problems for sufficiently large images and data sets. Nevertheless, this model has seen very limited application to tomographic reconstruction problems. In this paper, we present a model-based tomographic reconstruction algorithm using a learnt convolutional dictionary as a regularizer. The key contribution is the use of a data-dependent weighting scheme for the l1regularization to construct an effective denoising method that is integrated into the inversion using the Plug-and-Play reconstruction framework. Using simulated data sets we demonstrate that our approach can improve performance over traditional regularizers based on a Markov random field model and a patch-based sparse representation model for sparse and limited-view tomographic data sets.

Mixed norms that promote structured sparsity have numerous applications in signal processing and machine learning problems. In this work, we present a new algorithm, based on a Newton root search technique, for computing the projection onto the $\ell_{\infty,1}$ ball, which has found application in cognitive neuroscience and classification tasks. Numerical simulations show that our proposed method is between 8 and 10 times faster on average, and up to 20 times faster for very sparse solutions, than the previous state of the art. Tests on real functional magnetic resonance image data show that, for some data distributions, our algorithm can obtain speed improvements by a factor of between 10 and 100, depending on the implementation.

Mixed norms that promote structured sparsity have broad application in signal processing and machine learning problems. In this work we present a new algorithm for computing the projection onto the l∞,1 ball, which has found application in cognitive neuroscience and classification tasks. This algorithm is based on a Steffensen type root search technique, with a number of improvements over prior root search methods for the same problem. First, we theoretically derive an initial guess for the root search algorithm that helps to reduce the number of iterations to be performed. Second, we change the root search method, and through an analysis of the root search function, we construct a pruning strategy that significantly reduces the number of operations. Numerical simulations show that, compared to the state-of-the-art, our algorithm is between 4 and 5 times faster on average, and of up to 14 times faster for very sparse solutions.

Existing methods for constructing separable 2D dictionary filter banks approximate a set of K non-separable filters via a linear combination of R ≪ K separable filters. This approach involves the inefficiency of learning an initial set of non-separable filters, and places an upper bound on the quality of the separable filter banks. In this paper, we propose a method to directly learn a set of K separable dictionary filters from a given image training set by drawing ideas from convolutional dictionary learning (CDL) methods. We show that the separable filters obtained by our method match the performance of an equivalent number of non-separable filters. Furthermore, the computational performance of our learning method is shown to be substantially faster than a state-of-the-art non-separable CDL method for large numbers of filters or large training sets.

While convolutional sparse representations enjoy a number of useful properties, they have received limited attention for image reconstruction problems. The present paper compares the performance of block-based and convolutional sparse representations in the removal of Gaussian white noise. The usual formulation of the convolutional sparse coding problem is slightly inferior to the block-based representations in this problem, but the performance of the convolutional form can be boosted beyond that of the block-based form by the inclusion of suitable penalties on the gradients of the coefficient maps.

Convolutional sparse representations are a form of sparse representation with a structured, translation-invariant dictionary. Most convolutional dictionary learning algorithms to date operate in batch mode, requiring simultaneous access to all training images during the learning process, which results in very high memory usage and severely limits the training data size that can be used. Very recently, however, a number of authors have considered the design of online convolutional dictionary learning algorithms that offer far better scaling of memory and computational cost with training set size than batch methods. This paper extends our prior work, improving a number of aspects of our previous algorithm; proposing an entirely new one, with better performance, that supports the inclusion of a spatial mask for learning from incomplete data; and providing a rigorous theoretical analysis of these methods.

Convolutional sparse representations (CSR) of images are receiving increasing attention as an alternative to the usual independent patch-wise application of standard sparse representations. For CSR the dictionary is a filter bank of non-separable 2D filters, and the representation itself can be viewed as the synthesis dual of the analysis representation provided by a single level of a convolutional neural network (CNN). The current state-of-the-art convolutional sparse coding (CSC) algorithms achieve their computational efficiency by applying the convolutions in the frequency domain. It has been shown that any given 2D non-separable filter bank can be approximated as a linear combination of a relatively small number of separable filters. This approximation has been exploited for computationally efficient CNN implementations, but has thus far not been considered for convolutional sparse coding. In this paper we propose a computationally efficient algorithm, that apply the convolution in the spatial domain, to solve the CSC problem when the corresponding dictionary filters are separable. Our algorithm, based on the ISTA framework, use a two-term penalty function to attain competitive results when compared to the state-of-the-art methods in terms of computational performance, sparsity and reconstruction quality.

The current leading algorithms for both convolutional sparse coding and dictionary learning are based on variable splitting and Augmented Lagrangian methods. The dictionary learning algorithms alternate between sparse coding and dictionary subproblems, typically interleaving the updates for each of these two subproblems. Due to the variable splitting, in each subproblem one of these two variables must be chosen to be passed to the other subproblem. We perform a careful comparison of the algorithm convergence resulting from the different choices in conjunction with a number of different algorithms for the dictionary subproblem, showing that one of these choices consistently provides the best convergence.

Sparse representations are widely used in a broad variety of fields. A number of different methods have been proposed to solve the sparse coding problem, of which the alternating direction method of multipliers (ADMM) is one of the most popular. One of the disadvantages of this method, however, is the need to select an algorithm parameter, the penalty parameter, that has a significant effect on the rate of convergence of the algorithm. Although a number of heuristic methods have been proposed, as yet there is no general theory providing a good choice of this parameter for all problems. One obvious approach would be to try a number of different parameters at each iteration, proceeding further with the one that delivers the best reduction in functional value, but this would involve a substantial increase in computational cost. We show that, when solving the sparse coding problem for a dictionary corresponding to an operator with a fast transform, requiring iterative methods to solve the main linear system arising in the ADMM solution, it is possible to explore a large range of parameters at marginal additional cost, thus greatly improving the robustness of the method to the choice of penalty parameter.

While a number of different algorithms have recently been proposed for convolutional dictionary learning, this remains an expensive problem. The single biggest impediment to learning from large training sets is the memory requirements, which grow at least linearly with the size of the training set since all existing methods are batch algorithms. The work reported here addresses this limitation by extending online dictionary learning ideas to the convolutional context.

Denoising is often addressed via sparse coding with respect to an overcomplete dictionary. There are two main approaches when the dictionary is composed of translates of an orthonormal basis. The first, traditionally employed by techniques such as wavelet cycle spinning, separately seeks sparsity w.r.t. each translate of the orthonormal basis, solving multiple partial optimizations and obtaining a collection of sparse approximations of the noise-free image, which are aggregated together to obtain a final estimate. The second approach, recently employed by convolutional sparse representations, instead seeks sparsity over the entire dictionary via a global optimization. It is tempting to view the former approach as providing a suboptimal solution of the latter. In this letter, we analyze whether global sparsity is a desirable property, and under what conditions the global optimization provides a better solution to the denoising problem. In particular, our experimental analysis shows that the two approaches attain comparable performance in case of natural images and global optimization outperforms the simpler aggregation of partial estimates only when the image admits an extremely sparse representation. We explain this phenomenon by separately studying the bias and variance of these solutions, and by noting that the variance of the global solution increases very rapidly as the original signal becomes less and less sparse.

This letter extends our prior work on context-dependent piano transcription to estimate the length of the notes in addition to their pitch and onset. This approach employs convolutional sparse coding along with lateral inhibition constraints to approximate a musical signal as the sum of piano note waveforms (dictionary elements) convolved with their temporal activations. The waveforms are pre-recorded for the specific piano to be transcribed in the specific environment. A dictionary containing multiple waveforms per pitch is generated by truncating a long waveform for each pitch to different lengths. During transcription, the dictionary elements are fixed and their temporal activations are estimated and postprocessed to obtain the pitch, onset, and note length estimation. A sparsity penalty promotes globally sparse activations of the dictionary elements, and a lateral inhibition term penalizes concurrent activations of different waveforms corresponding to the same pitch within a temporal neighborhood, to achieve note length estimation. Experiments on the MIDI aligned piano sounds dataset show that the proposed approach significantly outperforms a state-of-the-art music transcription method trained in the same context-dependent setting in transcription accuracy.

In the past two decades, nonlinear image reconstruction methods have led to substantial improvements in the capabilities of numerous imaging systems. Such methods are traditionally formulated as optimization problems that are solved iteratively by simultaneously enforcing data consistency and incorporating prior models. Recently, the Plug-and-Play Priors (PPP) framework suggested that by using more sophisticated denoisers, not necessarily corresponding to an optimization objective, it is possible to improve the quality of reconstructed images. In this letter, we show that the PPP approach is applicable beyond linear inverse problems. In particular, we develop the fast iterative shrinkage/thresholding algorithm variant of PPP for model-based nonlinear inverse scattering. The key advantage of the proposed formulation over the original ADMM-based one is that it does not need to perform an inversion on the forward model. We show that the proposed method produces high quality images using both simulated and experimentally measured data.

Data fusion is the operation of integrating data from different modalities to construct a single consistent representation. This paper proposes variations of coupled dictionary learning through an additional factorization. One variation of this model is applicable to the pansharpening data fusion problem. Real world pansharpening data was applied to train and test our proposed formulation. The results demonstrate that the data fusion model can successfully be applied to the pan-sharpening problem.

The automatic detection of anomalies, defined as patterns that are not encountered in representative set of normal images, is an important problem in industrial control and biomedical applications. We have shown that this problem can be successfully addressed by the sparse representation of individual image patches using a dictionary learned from a large set of patches extracted from normal images. Anomalous patches are detected as those for which the sparse representation on this dictionary exceeds sparsity or error tolerances. Unfortunately, this solution is not suitable for many real-world visual inspection-systems since it is not scale invariant: since the dictionary is learned at a single scale, patches in normal images acquired at a different magnification level might be detected as anomalous. We present an anomaly-detection algorithm that learns a dictionary that is invariant to a range of scale changes, and overcomes this limitation by use of an appropriate sparse coding stage. The algorithm was successfully tested in an industrial application by analyzing a dataset of Scanning Electron Microscope (SEM) images, which typically exhibit different magnification levels.

Convolutional sparse representations differ from the standard form in representing the signal to be decomposed as the sum of a set of convolutions with dictionary filters instead of a linear combination of dictionary vectors. The advantage of the convolutional form is that it provides a single-valued representation optimised over an entire signal. The substantial computational cost of the convolutional sparse coding and dictionary learning problems has recently been shown to be greatly reduced by solving in the frequency domain, but the periodic boundary conditions imposed by this approach have the potential to create boundary artifacts. The present paper compares different approaches to avoiding these effects in both sparse coding and dictionary learning.

This paper presents a novel approach to automatic transcription of piano music in a context-dependent setting. This approach employs convolutional sparse coding to approximate the music waveform as the summation of piano note waveforms (dictionary elements) convolved with their temporal activations (onset transcription). The piano note waveforms are pre-recorded for the specific piano to be transcribed in the specific environment. During transcription, the note waveforms are fixed and their temporal activations are estimated and post-processed to obtain the pitch and onset transcription. This approach works in the time domain, models temporal evolution of piano notes, and estimates pitches and onsets simultaneously in the same framework. Experiments show that it significantly outperforms a state-of-the-art music transcription method trained in the same context-dependent setting, in both transcription accuracy and time precision, in various scenarios including synthetic, anechoic, noisy, and reverberant environments.

We present a comprehensive performance study of a new time-domain approach for estimating the components of an observed monaural audio mixture. Unlike existing time-frequency approaches that use the product of a set of spectral templates and their corresponding activation patterns to approximate the spectrogram of the mixture, the proposed approach uses the sum of a set of convolutions of estimated activations with prelearned dictionary filters to approximate the audio mixture directly in the time domain. The approximation problem can be solved by an efficient convolutional sparse coding algorithm. The effectiveness of this approach for source separation of musical audio has been demonstrated in our prior work, but under rather restricted and controlled conditions, requiring the musical score of the mixture being informed a priori and little mismatch between the dictionary filters and the source signals. In this paper, we report an evaluation that considers wider, and more practical, experimental settings. This includes the use of an audio-based multipitch estimation algorithm to replace the musical score, and an external dataset of audio single notes to construct the dictionary filters. Our result shows that the proposed approach remains effective with a larger dictionary, and compares favorably with the state-of-the-art nonnegative matrix factorization approach. However, in the absence of the score and in the case of a small dictionary, our approach may not be better.

When applying sparse representation techniques to images, the standard approach is to independently compute the representations for a set of overlapping image patches. This method performs very well in a variety of applications, but results in a representation that is multi-valued and not optimized with respect to the entire image. An alternative representation structure is provided by a convolutional sparse representation, in which a sparse representation of an entire image is computed by replacing the linear combination of a set of dictionary vectors by the sum of a set of convolutions with dictionary filters. The resulting representation is both single-valued and jointly optimized over the entire image. While this form of a sparse representation has been applied to a variety of problems in signal and image processing and computer vision, the computational expense of the corresponding optimization problems has restricted application to relatively small signals and images. This paper presents new, efficient algorithms that substantially improve on the performance of other recent methods, contributing to the development of this type of representation as a practical tool for a wider range of problems.

Monaural source separation is a challenging problem that has many important applications in music information retrieval. In this paper, we focus on the score-informed variant of this problem. While non-negative matrix factorization and some other approaches have been shown effective, few existing approaches have properly taken the phase information into account. There are unnatural sound in the separation result, as the phase of each source signal is considered equivalent to the phase of the mixed signal. To remedy this, we propose to perform source separation directly in the time domain using a convolutional sparse coding (CSC) approach. Evaluation on the Bach10 dataset shows that, when the instrument, pitch and onset/offset time are informed, the source to distortion ratio of the separation result reaches 8.59 dB, which is 2.02 dB higher than a state-of-the-art system called Soundprism.

While Principal Component Pursuit (PCP) is currently considered to be the state of the art method for video background modeling, it suffers from a number of limitations, including a high computational cost, a batch operating mode, and sensitivity to camera jitter. In this paper we propose a novel fully incremental PCP algorithm for video background modeling that is robust to translational and rotational jitter. It processes one frame at a time, obtaining similar results to standard batch PCP algorithms, while being able to deal with translational and rotational jitter. It also has extremely low memory footprint, and a computational complexity that allows almost real-time processing.

We address the problem of detecting anomalies in images, specifically that of detecting regions characterized by structures that do not conform those of normal images. In the proposed approach we exploit convolutional sparse models to learn a dictionary of filters from a training set of normal images. These filters capture the structure of normal images and are leveraged to quantitatively assess whether regions of a test image are normal or anomalous. Each test image is at first encoded with respect to the learned dictionary, yielding sparse coefficient maps, and then analyzed by computing indicator vectors that assess the conformance of local image regions with the learned filters. Anomalies are then detected by identifying outliers in these indicators. Our experiments demonstrate that a convolutional sparse model provides better anomaly-detection performance than an equivalent method based on standard patch-based sparsity. Most importantly, our results highlight that monitoring the local group sparsity, namely the spread of nonzero coefficients across different maps, is essential for detecting anomalous regions.

We propose a novel approach to performing change-detection based on sparse representations and dictionary learning. We operate on observations that are finite support signals, which in stationary conditions lie within a union of low dimensional subspaces. We model changes as perturbations of these subspaces and provide an online and sequential monitoring solution to detect them. This approach allows extension of the change-detection framework to operate on streams of observations that are signals, rather than scalar or multi-variate measurements, and is shown to be effective for both synthetic data and on bursts acquired by rockfall monitoring systems.

Several approaches used for inpainting of images take advantage of sparse representations. Some of these seek to learn a dictionary that will adapt the sparse representation to the available data. A further refinement is to adapt the learning process to the task itself. In this paper, we formulate a task-driven approach to inpainting as an optimization problem, and derive an algorithm for solving it. We demonstrate via numerical experiments that a purely task-driven approach gives superior results to other dictionary-learning approaches.

When applying sparse representation techniques to images, the standard approach is to independently compute the representations for a set of overlapping image patches. This method performs very well in a variety of applications, but the independent sparse coding of each patch results in a representation that is not optimal for the image as a whole. A recent development is convolutional sparse coding, in which a sparse representation for an entire image is computed by replacing the linear combination of a set of dictionary vectors by the sum of a set of convolutions with dictionary filters. A disadvantage of this formulation is its computational expense, but the development of efficient algorithms has received some attention in the literature, with the current leading method exploiting a Fourier domain approach. The present paper introduces a new way of solving the problem in the Fourier domain, leading to substantially reduced computational cost.

Video background modeling is an important task in many video processing applications. Most existing algorithms assume a Gaussian noise model, but digital videos are, in practice, prone to be degraded by impulse noise, due to transmission errors in wireless or high data-rate wired channels. Principal Component Pursuit (PCP), which also assumes a Gaussian noise model, is currently considered the state of the art for video background modeling. We propose a new PCP-based algorithm that fully integrates the impulse noise model and has computational performance comparable with that of current PCP implementations.

Iteratively Reweighted Least Squares (IRLS) is a well-established method of optimizing ℓpnorm problems such as Total Variation (TV) regularization. Within this general framework, there are several possible ways of constructing the weights and the form of the linear system that is iteratively solved as part of the algorithm. Many of these choices are equally reasonable from a theoretical perspective, and there has, thus far, been no systematic comparison between them. In this paper we provide such a comparison between the main choices in IRLS algorithms for ℓ1- and ℓ2-TV denoising, finding that there is a significant variation in the computational cost and reconstruction quality of the different variants.

In this proposal we present a Matlab-only implementation of a simple, novel, and fully incremental Principal Component Pursuit (PCP) algorithm for Video Background Modeling. Our implementation can process full HD color 1920 × 1088 videos at a rate of 0.61 seconds per frame running on a standard laptop (Intel i7-2670QM quad-core, 6GB RAM, 2.2 GHz). Unlike other incremental or online PCP-like algorithms, such as ReProCS, GRASTA or pROST, the initialization stage of our implementation is extremely fast, has modest memory requirements (6.5 seconds and less than 0.5 Gb for a full HD video), and is also able to quickly adapt to changes in the background. Moreover our implementation can also process live-feed videos, which in our proposed demonstration will be acquired via a wireless camera, resulting in an interactive demonstration where the the moving objects to be segmented are the audience.

Subspace models for image data sets, constructed by computing sparse representations of each image with respect to other images in the set, have been found to perform very well in a variety of applications, including clustering and classification problems. One of the limitations of these methods, however, is that the subspace representation is unable to directly model the effects of non-linear transformations such as translation, rotation, and dilation that frequently occur in practice. In this paper it is shown that the properties of convolutional sparse representations can be exploited to make these methods translation invariant, thereby simplifying or eliminating the alignment pre-processing task. The potential of the proposed approach is demonstrated in two diverse applications: image clustering and video background modeling.

We present an efficient algorithm for computing sparse representations whose nonzero coefficients can be divided into groups, few of which are nonzero. In addition to this group sparsity, we further impose that the nonzero groups themselves be sparse. We use a nonconvex optimization approach for this purpose, and use an efficient ADMM algorithm to solve the nonconvex problem. The efficiency comes from using a novel shrinkage operator, one that minimizes nonconvex penalty functions for enforcing sparsity and group sparsity simultaneously. Our numerical experiments show that combining sparsity and group sparsity improves signal reconstruction accuracy compared with either property alone. We also find that using nonconvex optimization significantly improves results in comparison with convex optimization.

We propose a simple alternating minimization algorithm for solving a minor variation on the original Principal Component Pursuit (PCP) functional. In computational experiments in the video background modeling problem, the proposed algorithm is able to deliver a consistent sparse approximation even after the first outer loop, (taking approximately 12 seconds for a 640 × 480 × 400 color test video) which is approximately an order of magnitude faster than Inexact ALM to construct a sparse component of the same quality.

Several Total Variation (TV) regularization methods have recently been proposed to address denoising under mixed Gaussian and impulse noise. While achieving high-quality denoising results, these new methods are based on complicated cost functionals that are difficult to optimize, which negatively affects their computational performance. In this paper we propose a simple cost functional consisting of a TV regularization term and ℓ2and ℓ1data fidelity terms, for Gaussian and impulse noise respectively, with local regularization parameters selected by an impulse noise detector. The computational performance of the proposed algorithm greatly exceeds that of the state of the art algorithms within the TV framework, and its reconstruction quality performance is competitive for high noise levels, for both grayscale and vector-valued images.

A robust version of Principal Component Analysis (PCA) can be constructed via a decomposition of a data matrix into low rank and sparse components, the former representing a low-dimensional linear model of the data, and the latter representing sparse deviations from the low-dimensional subspace. This decomposition has been shown to be highly effective, but the underlying model is not appropriate when the data are not modeled well by a single low-dimensional subspace. We construct a new decomposition corresponding to a more general underlying model consisting of a union of low-dimensional subspaces, and demonstrate the performance on a video background removal problem.

Alternating minimization algorithms with a shrinkage step, derived within the Split Bregman (SB) or Alternating Direction Method of Multipliers (ADMM) frameworks, have become very popular for ℓ1-regularized problems, including Total Variation and Basis Pursuit Denoising. It appears to be generally assumed that they deliver much better computational performance than older methods such as Iteratively Reweighted Least Squares (IRLS). We show, however, that IRLS type methods are computationally competitive with SB/ADMM methods for a variety of problems, and in some cases outperform them.

We describe an approach for improving the robustness to misregistration of pixel-wise anomalous change detection (ACD) algorithms. The aim of ACD is to distinguish actual anomalous changes from the irrelevant incidental differences that occur throughout the scene. For such change detection to be effective, it is important that corresponding pixels in the two images of interest correspond to the same location in the scene. Indeed, one of the most confounding sources of incidental differences is the inevitable imprecision in the coregistration of the two images. We address this with small local adjustments to the coregistration which leads to a modified misregistration-insensitive measure of anomalousness. Several variants are considered, and the resulting performance improvements are evaluated using both real and simulated changes, and real and simulated misregistration.

Exemplar-based methods, in which actual image blocks are used to fill in missing content, have achieved state of the art performance in image inpainting. The majority of these adopt a progressive approach, filling in the missing region inwards from the boundary. The final result is highly dependent on fill order, and while significant progress has been made on the choice of this order, the greedy nature of such a process leads to artifacts in some cases. The alternative exemplar-based approach proposed here is defined via joint optimization of a single functional, simultaneously assigning an estimated value to the entire inpainting region. The results are found to be highly competitive with other recent inpainting methods.

We present a new algorithm for bound-constrained total-variation (TV) regularization that in comparison with its predecessors is simple, fast, and flexible. We use a splitting approach to decouple TV minimization from enforcing the constraints. Consequently, existing TV solvers can be employed with minimal alteration. This also makes the approach straightforward to generalize to any situation where TV can be applied. We consider deblurring of images with Gaussian or salt-and-pepper noise, as well as Abel inversion of radiographs with Poisson noise.

We derive a class of algorithms for detecting anomalous changes in hyperspectral image pairs by modeling the data with elliptically contoured (EC) distributions. These algorithms are generalizations of well-known detectors that are obtained when the EC function is Gaussian. The performance of these EC-based anomalous change detectors is assessed on real data using both real and simulated changes. In these experiments, the EC-based detectors substantially outperform their Gaussian counterparts.

We introduce a new exemplar-based inpainting algorithm that represents the region to be inpainted as a sparse linear combination of example blocks, extracted from the image being inpainted or an external training image set. This method is conceptually simple, being computed by minimization of a simple functional, and avoids the complexity of correctly ordering the filling in of missing regions of other exemplar-based methods. Initial performance comparisons on small inpainting regions indicate that this method provides similar or better performance than other recent methods.

There has recently been considerable interest in applying Total Variation regularization with an ℓ1data fidelity term to the denoising of images subject to salt and pepper noise, but the extension of this formulation to more general problems, such as deconvolution, has received little attention. We consider this problem, comparing the performance of ℓ1-TV deconvolution, computed via our Iteratively Reweighted Norm algorithm, with an alternative variational approach based on Mumford-Shah regularization. The ℓ1-TV deconvolution method is found to have a significant advantage in reconstruction quality, with comparable computational cost.

We propose a simple but flexible method for solving the generalized vector-valued TV (VTV) functional, which includes both the ¿2-VTV and ¿1-VTV regularizations as special cases, to address the problems of deconvolution and denoising of vector-valued (e.g. color) images with Gaussian or salt-and-pepper noise. This algorithm is the vectorial extension of the Iteratively Reweighted Norm (IRN) algorithm [1] originally developed for scalar (grayscale) images. This method offers competitive computational performance for denoising and deconvolving vector-valued images corrupted with Gaussian (¿2-VTV case) and salt-and-pepper noise (¿1-VTV case).

Replacing the l(2) data fidelity term of the standard Total Variation (TV) functional with an l(1) data fidelity term has been found to offer a number of theoretical and practical benefits. Efficient algorithms for minimizing this l(1)-TV functional have only recently begun to be developed, the fastest of which exploit graph representations, and are restricted to the denoising problem. We describe an alternative approach that minimizes a generalized TV functional, including both l(2)-TV and l(1)-TV as special cases, and is capable of solving more general inverse problems than denoising (e.g., deconvolution). This algorithm is competitive with the graph-based methods in the denoising case, and is the fastest algorithm of which we are aware for general inverse problems involving a nontrivial forward linear operator.

Total variation (TV) regularization has become a popular method for a wide variety of image restoration problems, including denoising and deconvolution. A number of authors have recently noted the advantages of replacing the standard lscr2data fidelity term with an lscr1norm. We propose a simple but very flexible method for solving a generalized TV functional that includes both the lscr2-TV and lscr1-TV problems as special cases. This method offers competitive computational performance for lscr2-TV and is comparable to or faster than any other lscr1-TV algorithms of which we are aware.

A typical subsurface environment is heterogeneous, consists of multiple materials (geologic facies), and is often insufficiently characterized by data. The ability to delineate geologic facies and to estimate their properties from sparse data is essential for modeling physical and biochemical processes occurring in the subsurface. We demonstrate that the support vector machine is a viable and efficient tool for lithofacies delineation, and we compare it with a geostatistical approach. To illustrate our approach, and to demonstrate its advantages, we construct a synthetic porous medium consisting of two heterogeneous materials and then estimate boundaries between these materials from a few selected data points. Our analysis shows that the error in facies delineation by means of support vector machines decreases logarithmically with increasing sampling density. We also introduce and analyze the use of regression support vector machines to estimate the parameter values between points where the parameter is sampled.

We present a technique for combining prior knowledge about transformations that should be ignored with a covariance matrix estimated from training data to make an improved Mahalanobis distance classifier. Modern classification problems often involve objects represented by high-dimensional vectors or images (for example, sampled speech or human faces). The complex statistical structure of these representations is often difficult to infer from the relatively limited training data sets that are available in practice. Thus, we wish to efficiently utilize any available a priori information, such as transformations or the representations with respect to which the associated objects are known to retain the same classification (for example, spatial shifts of an image of a handwritten digit do not alter the identity of the digit). These transformations, which are often relatively simple in the space of the underlying objects, are usually nonlinear in the space of the object representation, making their inclusion within the framework of a standard statistical classifier difficult. Motivated by prior work of Simard et al. (1998; 2000), we have constructed a new classifier which combines statistical information from training data and linear approximations to known invariance transformations. When tested on a face recognition task, performance was found to exceed by a significant margin that of the best algorithm in a reference software distribution.

Fractal image compression is a technique based on the representation of an image by a contractive transform, on the space of images, for which the fixed point is close to the original image. This broad principle encompasses a very wide variety of coding schemes, many of which have been explored in the rapidly growing body of published research. While certain theoretical aspects of this representation are well established, relatively little attention has been given to the construction of a coherent underlying image model that would justify its use. Most purely fractal-based schemes are not competitive with the current state of the art, but hybrid schemes incorporating fractal compression and alternative techniques have achieved considerably greater success. This review represents a survey of the most significant advances, both practical and theoretical, since the publication of Jacquin's (1990) original fractal coding scheme.