High quality and high speed videography using Non-Line-of-Sight (NLOS) imaging benefit autonomous navigation, collision prevention, and post-disaster search and rescue tasks. Current solutions have to balance between the frame rate and image quality. High frame rates, for example, can be achieved by reducing either per-point scanning time or scanning density, but at the cost of lowering the information density at individual frames. Fast scanning process further reduces the signal-to-noise ratio and different scanning systems exhibit different distortion characteristics. In this work, we design and employ a new Transient Transformer architecture called TransiT to achieve real-time NLOS recovery under fast scans. TransiT directly compresses the temporal dimension of input transients to extract features, reducing computation costs and meeting high frame rate requirements. It further adopts a feature fusion mechanism as well as employs a spatial-temporal Transformer to help capture features of NLOS transient videos. Moreover, TransiT applies transfer learning to bridge the gap between synthetic and real-measured data. In real experiments, TransiT manages to reconstruct from sparse transients of $16 \times 16$ measured at an exposure time of 0.4 ms per point to NLOS videos at a $64 \times 64$ resolution at 10 frames per second. We will make our code and dataset available to the community.

We introduce HOLI-1-to-3, a novel technique for holistic 3D shape recovery from a single-viewpoint input, by effectively combining line-of-sight (LOS) and non-line-of-sight (NLOS) imaging. We leverage advancements in ultrafast time-of-flight (ToF) sensors and learning-based 3D shape inference techniques, such as diffusion models. HOLI-1-to-3 employs a new neural plenoptic representation, which unifies radiance fields (for LOS RGB images) and transient fields (for NLOS transients). HOLI-1-to-3 is optimized through a two-stage pipeline involving diffusion priors and transients prior. Our technique allows for accurate and continuous reconstruction of both visible and invisible parts of objects from a single view. Comprehensive experiments on both simulated and real-world datasets demonstrate the effectiveness of HOLI-1-to-3in resolving ambiguities in invisible parts of objects and significantly improving overall generation quality. The datasets used in our experiments will be made available to the research community to facilitate further achievements in holistic 3D shape recovery.

Reconstructing an object behind a scattering medium needs to tackle different light scatterings inside the medium and in the free space. Major approaches, e.g., diffuse optical tomography and non-line-of-sight imaging, address either of the light scatterings. Confocal diffuse tomography (CDT) considers the media acting as a diffuse kernel on the free-space scattering and recovers the object by deconvolving the inside-medium scattering. Inspired by CDT, we present a Neural De-Scatterer to solve this challenging problem. We exploit neural implicit fields to represent the the free-space scattering and use a multilayer perceptron (MLP) to learn the density and albedo of the hidden object. Furthermore, we tailor a bi-directional training strategy to optimize the MLP with forward and backward losses and employ hash encoding for memory and computation efficiency. The Neural De-Scatterer enables us to reconstruct objects at arbitrary resolution. Comprehensive experiments with synthetic and real measurements demonstrate that our Neural De-Scatterer outperforms state-of-the-art methods. Our data and code are publicly available.

Recovering information from non-line-of-sight (NLOS) imaging is a computationally-intensive inverse problem. Most physics-based NLOS imaging methods address the complexity of this problem by assuming three-bounce reflections and no self-occlusion. However, these assumptions may break down for objects with large depth variations, preventing physics-based algorithms from accurately reconstructing the details and high-frequency information. On the other hand, while learning-based methods can avoid these assumptions, they may struggle to reconstruct details without specific designs due to the spectral bias of neural networks. To overcome these issues, we propose a novel approach that enhances physics-based NLOS imaging methods by introducing a learnable inverse kernel in the Fourier domain and using an attention mechanism to improve the neural network to learn high-frequency information. Our method is evaluated on publicly available and new synthetic datasets, demonstrating its commendable performance compared to prior physics-based and learning-based methods, especially for objects with large depth variations. Moreover, our approach generalizes well to real data and can be applied to tasks such as classification and depth reconstruction. We will make our code and dataset publicly available: https://sci2020.github.io.