Single-photon lidar (SPL) can achieve high-accuracy, lowlight ranging; however, velocity estimation typically requires regression over multiple distance measurements. Here, we introduce Doppler SPL, which enables joint instantaneous velocity and range estimation. First, we derive a measurement model for SPL, showing that a target moving at a constant velocity introduces a Doppler shift into the sequence of photon detection times. We then introduce estimators for range and velocity based on Fourier analysis of the detection time sequence. Simulations show improved accuracy of our method over baseline approaches, and we further validate our approach on experimental SPL data for a moving target.

Conventional wisdom suggests that single-photon lidar (SPL) should operate in low-light conditions to minimize dead-time effects. Many methods have been developed to mitigate these effects in synchronous SPL systems. However, solutions for free-running SPL remain limited despite the advantage of reduced histogram distortion from dead times. To improve the accuracy of free-running SPL, we propose a computationally efficient joint maximum likelihood estimator of the signal flux, the background flux, and the depth using only histograms, along with a complementary regularization framework that incorporates a learned point cloud score model as a prior. Simulations and experiments demonstrate that free-running SPL yields lower estimation errors than its synchronous counterpart under identical conditions, with our regularization further improving accuracy.

We consider the problem of simultaneously learning a convex penalty function and its proximity operator for image reconstruction from incomplete measurements. Our goal is to apply Accelerated Proximal Gradient Method (APGM) using a learned proximity operator in place of the true proximity operator of the learned penalty function. Starting from a Gaussian image denoiser, we learn an associated penalty function and its proximity operator. The learned penalty function offers provable reconstruction guarantees, whereas access to its proximity operator presents the opportunity to achieve APGM convergence rates, which are faster than those of subgradient descent approaches.

Recent developments in wave-based sensor technologies, such as ground penetrating radar (GPR), provide new opportunities for accurate imaging of underground scenes. Given measurements of the scattered electromagnetic wavefield, the goal is to estimate the spatial distribution of the permittivity of the underground scenes. However, such problems are highly ill-posed, difficult to formulate, and computationally expensive. In this paper, we propose a physics-inspired machine learning-based method to learn the wave-matter interaction under the GPR setting. The learned forward model is combined with a learned signal prior to recover the permittivity distribution of the unknown underground scenes. We test our approach on a dataset of 400 permittivity maps with a three-layer background, which is challenging to solve using existing methods. We demonstrate via numerical simulation that our method achieves a 50% improvement in mean squared error over benchmark machine learning-based solvers for reconstructing layered underground scenes.

We introduce a scalable multi-modal approach to learn dense, i.e., pixel-level, correspondences and occlusion maps, between images in a video sequence. The problems of finding dense correspondences and occlusion maps are fundamental in computer vision. In this work we jointly train a deep network to tackle both, with a shared feature extraction stage. We use depth and color images with ground truth optical flow and occlusion maps to train the network end-to-end. From the multi-modal input, the network learns to estimate occlusion maps, optical flows, and a correspondence embedding providing a meaningful latent feature space. We evaluate the performance on a dataset of images derived from synthetic characters, and perform a thorough ablation study to demonstrate that the proposed components of our architecture combine to achieve the lowest correspondence error. The scalability of our proposed method comes from the ability to incorporate additional modalities, e.g., infrared images.

Recent advances in solving imaging inverse problems have witnessed the combination of deep learning models with classical image models for better signal representation. One such approach, DeepRED, combines the deep image prior (DIP) with the regularization by denoising (RED) framework to boost the performance of image deblurring and super resolution tasks. In this paper, we formulate DeepRED as a consensus equilibrium problem and set up a fixed-point algorithm for solving the equilibrium equations. We also derive sufficient conditions that the DIP generative prior should satisfy to ensure that the corresponding fixed-point operator is non-expansive. We then demonstrate that the fixed-point algorithm that solves the CE equations results in improved image reconstruction quality in a deblurring setting compared to state-of-the-art methods.

In several applications, including imaging of deformable objects while in motion, simultaneous localization and mapping, and unlabeled sensing, we encounter the problem of recovering a signal that is measured subject to unknown permutations. In this paper we take a fresh look at this problem through the lens of optimal transport (OT). In particular, we recognize that in most practical applications the unknown permutations are not arbitrary but some are more likely to occur than others. We exploit this by introducing a regularization function that promotes the more likely permutations in the solution. We show that, even though the general problem is not convex, an appropriate relaxation of the resulting regularized problem allows us to exploit the well-developed machinery of OT and develop a tractable algorithm.

We study the problem of reconstructing an object from phaseless measurements in the context of inverse multiple scattering. Our formulation explicitly decouples the variables that represent the unknown object image and the unknown phase, respectively, in the forward model. This enables us to simultaneously optimize over both unknowns with appropriate regularization for each. The resulting optimization problem is nonconvex due to the nonlinear propagation model for multiple scattering and the nonconvex regularization of the phase variables. Nevertheless, we demonstrate experimentally that we can solve the optimization problem using a variation of the fast iterative shrinkage-thresholding algorithm (FISTA)-a convex algorithm, popular for its speed and simplicity-that converges well in our experiments. Numerical results with both simulated and experimentally measured data show that the proposed method outperforms the state-of-the-art phaseless inverse scattering method.

We address the problem of sharpening low spatial-resolution multi-spectral (MS) images with their associated misaligned high spatial-resolution panchromatic (PAN) image, based on priors on the spatial blur kernel and on the cross-channel relationship. In particular, we formulate the blind pan-sharpening problem within a multi-convex optimization framework using total generalized variation for the blur kernel and local Laplacian prior for the cross-channel relationship. The problem is solved by the alternating direction method of multipliers (ADMM), which alternately updates the blur kernel and sharpens intermediate MS images. Numerical experiments demonstrate that our approach is more robust to large misalignment errors and yields better super resolved MS images compared to state-of-the-art optimization-based and deep-learning-based algorithms.

Time-correlated single photon counting (TCSPC) is a powerful technique for lidar depth imaging, allowing for accurate range measurements from very low light levels. However, single-photon detectors used in TCSPC have a dead time after each photon detection, which blocks registration of subsequent photons arriving within that dead time, causing a distortion of the detection time distribution. The most common approach to avoiding dead time distortion is to optically reduce the photon arrival rate such that with high probability no photons arrive during the dead time. However, this prevents the high photon flux acquisition necessary for real-time applications such as autonomous navigation. In this paper, we propose a dead time compensation method that enables fast data acquisition with dead time-limited detectors. Specifically, we model dead time-affected detection times as a Markov chain, present a simple method for approximating the stationary distribution, and estimate depths using a log-matched filter matched to that distribution. Our method applies to multimodal imaging systems where a standard camera is used in conjunction with lidar to provide information about scene reflectivity. Simulation results for real 3D scenes show that our method reduces the root mean squared error by several orders of magnitude for the same acquisition time.

The ability to image scenery outside a camera's line-of-sight would be useful in a variety of applications, including autonomous vehicle collision avoidance, or for first responders to anticipate danger around a corner. When a wall obstructs the camera, light cast onto the floor from behind the wall may be used to recover angular variation of light intensity reflected by the hidden scene, forming a 1D scene projection. Recent work has demonstrated that temporal variation in a video, or sequence of floor images, may be used to image moving components of the hidden scene. However, in many applications, it would be useful to be able to image stationary components as well. This earlier approach was also designed for, and tested on, floors that have approximately uniform albedo, while many real floors have spatially varying albedo patterns such as checkered tiles and patterned carpets. In this work, we propose a method to reconstruct a 1D projection of all components in a hidden scene from a single photograph of the floor without assuming uniform floor albedo. Specifically, we derive a forward model that describes the measured photograph as a nonlinear combination of the unknown floor albedo and the light from behind the wall. The inverse problem, which is the joint estimation of floor albedo and a 1D reconstruction of the hidden scene, is then solved via optimization, where we introduce regularizers that help separate light variations in the measured photograph due to floor pattern and hidden scene, respectively. We demonstrate the effectiveness of our formulation and algorithm using synthetic and experimentally measured data.

Approximate message passing (AMP) is a class of low-complexity, scalable algorithms for solving high-dimensional linear regression tasks where one wishes to recover an unknown signal from noisy, linear measurements. AMP is an iterative algorithm that performs estimation by updating an estimate of the unknown signal at each iteration and the performance of AMP (quantified, for example, by the mean squared error of its estimates) depends on the choice of a “denoiser” function that is used to produce these signal estimates at each iteration. An attractive feature of AMP is that its performance can be tracked by a scalar recursion referred to as state evolution. Previous theoretical analysis of the accuracy of the state evolution predictions has been limited to the use of only separable denoisers or block-separable denoisers, a class of denoisers that underperform when sophisticated dependencies exist between signal entries. Since signals with entrywise dependencies are common in image/video-processing applications, in this work we study the high-dimensional linear regression task when the dependence structure of the input signal is modeled by a Markov random field prior distribution. We provide a rigorous analysis of the performance of AMP, demonstrating the accuracy of the state evolution predictions, when a class of non-separable sliding-window denoisers is applied. Moreover, we provide numerical examples where AMP with sliding-window denoisers can successfully capture local dependencies in images.

The problem of reconstructing an object from the measurements of the light it scatters is common in numerous imaging applications. While the most popular formulations of the problem are based on linearizing the object-light relationship, there is an increased interest in considering nonlinear formulations that can account for multiple light scattering. In this paper, we propose an image reconstruction method, called CISOR, for nonlinear diffractive imaging, based on our new variant of fast iterative shrinkage/thresholding algorithm (FISTA) and total variation (TV) regularization. We prove that CISOR reliably converges for our nonconvex optimization problem, and systematically compare our method with other state-of-the-art methods on simulated as well as experimentally measured data.

Solving a large-scale regularized linear inverse problem using multiple processors is important in various real-world applications due to the limitations of individual processors and constraints on data sharing policies. This paper focuses on the setting where the matrix is partitioned column-wise. We extend the algorithmic framework and the theoretical analysis of approximate message passing (AMP), an iterative algorithm for solving linear inverse problems, whose asymptotic dynamics are characterized by state evolution (SE). In particular, we show that column-wise multiprocessor AMP (C-MP-AMP) obeys an SE under the same assumptions when the SE for AMP holds. The SE results imply that (i) the SE of C-MP-AMP converges to a state that is no worse than that of AMP and (ii) the asymptotic dynamics of C-MP-AMP and AMP can be identical. Moreover, for a setting that is not covered by SE, numerical results show that damping can improve the convergence performance of C-MP-AMP.

We consider the problem of fusing multiple cloud-contaminated aerial images of a 3D scene to generate a cloud-free image, where the images are captured from multiple unknown view angles. In order to fuse these images, we propose an end-to-end framework incorporating epipolar geometry and low-rank matrix completion. In particular, we first warp the multi-angular images to single-angle ones based on the estimated fundamental matrices that relate the multi-angular images according to their projective relations to the 3D scene. Then we formulate the fusion process of the warpped images as a low-rank matrix completion problem where each column of the matrix corresponds to a vectorized image with missing entries corresponding to cloud or occluded areas. Results using DigitalGlobe high spatial resolution images demonstrate that our algorithm outperforms existing approaches.

Approximate message passing (AMP) is a class of efficient algorithms for solving high-dimensional linear regression tasks where one wishes to recover an unknown signal βο from noisy, linear measurements y = Aβ0+ w. When applying a separable denoiser at each iteration, the performance of AMP (for example, the mean squared error of its estimates) can be accurately tracked by a simple, scalar iteration referred to as state evolution. Although separable denoisers are sufficient if the unknown signal has independent and identically distributed entries, in many real-world applications, like image or audio signal reconstruction, the unknown signal contains dependencies between entries. In these cases, a coordinate-wise independence structure is not a good approximation to the true prior of the unknown signal. In this paper we assume the unknown signal has dependent entries, and using a class of non-separable sliding-window denoisers, we prove that a new form of state evolution still accurately predicts AMP performance. This is an early step in understanding the role of non-separable denoisers within AMP, and will lead to a characterization of more general denoisers in problems including compressive image reconstruction.

We study the excess mean square error (EMSE) above the minimum mean square error (MMSE) in large linear systems where the posterior mean estimator (PME) is evaluated with a postulated prior that differs from the true prior of the input signal. We focus on large linear systems where the measurements are acquired via an independent and identically distributed random matrix, and are corrupted by additive white Gaussian noise (AWGN). The relationship between the EMSE in large linear systems and EMSE in scalar channels is derived, and closed form approximations are provided. Our analysis is based on the decoupling principle, which links scalar channels to large linear system analyses. Numerical examples demonstrate that our closed form approximations are accurate.