Recently, 3D reconstruction from multiview stereo (MVS) has advanced significantly with the introduction of neural implicit representation methods, which estimate voxel densities or signed distance fields (SDFs) to describe the 3D structure of a scene. Although such neural-based methods typically require a large number of captured images to estimate dense volumetric information during training, developing systems that can recover the 3D shape of moving objects using only a small number of stationary cameras remains highly demanding and challenging. To address the issue of sparse views, various active lighting techniques have been proposed. However, the problem remains inherently difficult, particularly when attempting to capture the complete shape of an object with a wide baseline. In this paper, we propose a novel approach that combines active lighting with photometric stereo (PS) using neural representations. Additionally, we introduce a multiplexed illumination technique that captures the entire shape of an object in a single shot. Although this results in a low signal-to-noise ratio (SNR), our method also addresses this issue. The advantages of our technique are demonstrated through real-world experiments, showcasing its ability to capture a 4D scene.

Since high-fidelity reference images are difficult to obtain in real underwater scenes, most deep models trained by synthetic paired data cannot match real-world data exactly. In this paper, we propose an unsupervised training framework for underwater image enhancement (UIE) by leveraging an iterative training strategy and quantification of specific neural units. Specifically, to eliminate the heavy color cast and distortion in the underwater images, we decompose the unsupervised image enhancement as two targeted sub-tasks, namely colorization and color compensation. First, a diffusion model is introduced for colorization to correct the green and blue color casts. Then, to intensify the learning ability of balanced color information, we introduce an extra network branch and propose a quantification mechanism for color compensation. The extra branch encodes style information from normal images into the generative model, while the quantification mechanism identifies and adjusts neural units relevant to warm colors, improving the model’s ability to learn balanced color feature representations for robust generation. In the end, through iterative training, color cast and distortion are progressively reduced, leading to a gradual improvement in the quality of the generated images. Experimental results on various widely used underwater datasets demonstrate that our approach achieves excellent performance, even when compared to recent supervised methods.

Multi-line laser ID assignment is crucial for underwater 3D reconstruction but fails when lines fragment. We reformulate this as a graph-based sequence labeling task and propose a novel two-stage hierarchical framework using Spatio-Temporal Graph Attention Networks (ST-GAT). Our method first reasons over a spatio-temporal graph of laser endpoints and intersections to handle local fragmentation, then elevates this to a global segment-level optimization with trajectory-constrained Viterbi decoding to ensure temporal consistency. This GNN-based approach eliminates the reliance on complete epipolar geometry. Experiments on real underwater datasets demonstrate superior reconstruction completeness and temporal stability, especially in challenging environments where traditional methods fail.

Time-of-flight (ToF) cameras, widely used in commercial applications such as augmented reality and autonomous driving, measure depth by analyzing the flight time of emitted laser signals. Indirect ToF (I-ToF) cameras are particularly popular due to their high resolution, high frame rate, and affordability. However, they struggle with depth estimation in scenes containing translucent objects, as their fundamental assumption – single direct reflectance – breaks down due to complex light interactions. In this paper, we address depth estimation with translucent objects by leveraging coding curve (CC) distortions, which have recently been shown to mitigate multi-path interference (MPI). The CC is formulated as a function derived from multiple values of various types of temporal encoding by the photodetector at each pixel. Specifically, inspired by recent MPI solutions using CC, we sample both values and depth errors under translucent objects at fixed intervals to train a model that predicts foreground and background depth errors from the CC, ensuring accurate reconstruction of translucent scenes through error correction. Our approach is validated through real-world experiments, demonstrating its effectiveness in improving depth estimation in scenes with translucent objects.

Among various active 3D measurement techniques, Structured Light (SL) is one of the most popular methods for its robustness and high accuracy. The ordinary SL system consists of a camera and a projector, and by projecting a pre-defined pattern, we can obtain pixel-to-pixel correspondences between the camera and the projector for triangulation. However, if we lack knowledge of the projected pattern for some reason, e.g., the projected pattern is not as expected due to lens distortion, inaccurate calibration, undesired optical phenomena like inter-reflection, and so on, the accuracy of conventional SL is severely degraded. As a remedy, we propose unsupervised structured light (USSL), which does not explicitly use prior knowledge of the pattern. Inspired by the fact that humans can recognize the scene structure illuminated by an unknown light source (e.g. rotating mirror ball), and some prior works have succeeded in novel-view-synthesis under unknown illumination conditions, we implement USSL on Neural Signed Distance Fields (Neural SDF) pipeline with implicit reflection module powered by a neural network. Additionally, since every SL method causes occlusion (shadow) by pattern projection, we must consider it for accurate shape reconstruction. To this end, we integrate shadow volume rendering into the proposed pipeline. Experiments with synthetic and real datasets are conducted to confirm the feasibility of the proposed method.

3D-shape reconstruction in extreme environments, such as low illumination or scattering condition, has been an open problem and intensively researched. Active stereo is one of potential solution for such environments for its robustness and high accuracy. However, active stereo systems usually consist of specialized system configurations with complicated algorithms, which narrow their application. In this paper, we propose Neural Signed Distance Field for active stereo systems to enable implicit correspondence search and triangulation in generalized Structured Light. With our technique, textureless or equivalent surfaces by low light condition are successfully reconstructed even with a small number of captured images. Experiments were conducted to confirm that the proposed method could achieve state-of-the-art reconstruction quality under such severe condition. We also demonstrated that the proposed method worked in an underwater scenario.

In recent years, the increasing demand for underwater 3D measurement for various applications has brought about challenges such as low accuracy in 3D shape acquisition and difficulties in localizing sensor positions. This paper introduces a robust calibration method for underwater 3D sensors, comprising line lasers and cameras, utilizing a physically accurate model. Specifically, our proposed line laser calibration method estimates laser plane parameters using two types of planar constraints, avoiding the need for the costly process of backward-projection of refraction for optimization. For camera calibration, we advocate a two-step approach incorporating a simple yet effective deep-learning-based marker detection algorithm to estimate parameters of refraction, representing a physically correct lens model. Through experiments, we validate the superior performance of our methods over previous approximation-based approaches, as demonstrated in simulations and actual experiments conducted in a swimming pool.

The indirect time-of-flight camera measures depth based on the phase shift between modulated laser and its reflected light. However, when there are multi-paths due to interreflections, depth estimation is significantly affected as it assumes that only a single reflected light is observed. In this paper, we propose a method for mitigating multi-path interference utilizing coding curves derived from measurements of multiple sensor taps, which represent the contribution of global light component, i.e., multi-path. First, to obtain the coding curve, we offset the laser light to the sensor and measure the sensor tap values for different delays. We propose a data-driven method for mitigating MPI, utilizing a dataset where distorted coding curves are paired with MPI intensity. By using the measured coding curve as a query to obtain correction values from the dataset, we can mitigate the effect of MPI and obtain the correct depth. Unlike previous methods, our approach can solve the general MPI problem while imposing fewer restrictions on the type of reflections, the number of light paths, and the modulation frequency. We validate the effectiveness of our method through both simulation and real experiments.

The demand for 3D shape measurement of underwater scene is increasing in various applications. Especially, simultaneous localization and mapping (SLAM) technique utilizing remotely operated vehicle (ROV) attached with 3D sensors has been intensively researched. This paper focuses on solving pose optimization problem for underwater robots with camera/multiple-line-lasers setup, especially for the scene with some textures (color information). To this end, a two-stage pose optimization technique is proposed. In the first stage, due to the sparse nature of the reconstructed shape in the light-sectioning method consisting of several 3D curves, we bundle 10 to 20 consecutive frames to form a block shape, refining significant errors in the initial sensor poses using a novel bundle adjustment algorithm. In the second stage, remaining pose errors are corrected by a block-based matching algorithm utilizing iterative closest point (ICP) algorithm with color information. Through experiments in underwater environment with a real system, it was validated that the proposed method demonstrates superior performance compared to past underwater SLAM techniques.

This paper addresses the problem of reconstructing scenes behind optical diffusers, which is common in applications such as imaging through frosted glass. We propose a new approach that exploits specular reflection to capture sharp light distributions with a point light source, which can be used to detect reflections in low signal-to-noise scenarios. In this paper, we propose a rasterizer-based differentiable renderer to solve this problem by minimizing the difference between the captured and rendered images. Because our method can simultaneously optimize multiple observations for different light source positions, it is confirmed that ambiguities of the scene are efficiently eliminated by increasing the number of observations. Experiments show that the proposed method can reconstruct a scene with several mirror-like objects behind the diffuser in both simulated and real environments.

In this paper, we present an approach to image enhancement with diffusion model in underwater scenes. Our method adapts conditional denoising diffusion probabilistic models to generate the corresponding enhanced images by using the underwater images and the Gaussian noise as the inputs. Additionally, in order to improve the efficiency of the reverse process in the diffusion model, we adopt two different ways. We firstly propose a lightweight transformer-based denoising network, which can effectively promote the time of network forward per iteration. On the other hand, we introduce a skip sampling strategy to reduce the number of iterations. Besides, based on the skip sampling strategy, we propose two different non-uniform sampling methods for the sequence of the time step, namely piecewise sampling and searching with the evolutionary algorithm. Both of them are effective and can further improve performance by using the same steps against the previous uniform sampling. In the end, we conduct a relative evaluation of the widely used underwater enhancement datasets between the recent state-of-the-art methods and the proposed approach. The experimental results prove that our approach can achieve both competitive performance and high efficiency. Our code is available at https://github.com/piggy2009/DM_underwater.

Photometric stereo (PS) is a major technique to recover surface normal for each pixel. However, since it assumes Lambertian surface and directional light to estimate the value, a large number of images are usually required to avoid the effects of outliers and noise. In this paper, we propose a technique to reduce the number of images by using distributed light sources, where the patterns are optimized by a deep neural network (DNN). In addition, to efficiently realize the distributed light, we use an optical diffuser with a video projector, where the diffuser is illuminated by the projector from behind, the illuminated area on the diffuser works as if an arbitrary-shaped area light. To estimate the surface normal using the distributed light source, we propose a near-light photometric stereo (NLPS) using DNN. Since optimization of the pattern of distributed light is achieved by a differentiable renderer, it is connected with NLPS network, achieving end-to-end learning. The experiments are conducted to show the successful estimation of the surface normal by our method from a small number of images.

There are various demands for underwater 3D reconstruction, however, since most active stereo 3D reconstruction methods focus on the air environment, it is difficult to directly apply them to underwater due to the several critical reasons, such as refraction, water flow and severe attenuation. Typically, calibration-markers or laser-lines are strongly blurred and saturated by attenuation, which makes difficult to recover shape in the water. Another problem is that it is difficult to keep cameras, projectors and objects static in the water because of strong water flow, which prevents accurate calibration. In this paper, we propose a method to solve those problems by novel algorithm using deep neural network (DNN), epipolar constraint and specially designed devices. We also built a real system and tested it in the water, e.g., pool and sea. Experimental results confirmed the effectiveness of the proposed method. We also demonstrated real 3D scan in the sea.

High-precision three-dimensional scanning systems have been intensively researched and developed. Recently, for acquisition of large scale scene with high density, simultaneous localisation and mapping (SLAM) technique is preferred because of its simplicity; a single sensor that is moved around freely during 3D scanning. However, to integrate multiple scans, captured data as well as position of each sensor must be highly accurate, making these systems difficult to use in environments not accessible by humans, such as underwater, internal body, or outer space. In this paper, we propose a new, flexible system with multiple line lasers that reconstructs dense and accurate 3D scenes. The advantages of our proposed system are (1) no need of synchronization nor precalibration between lasers and a camera, and (2) the system can reconstruct 3D scenes in extreme conditions, such as underwater. We propose a new self-calibration method leveraging coplanarity and Epipolar constraints is proposed. We also propose a new bundle adjustment (BA) technique that is tailored to the system for a dense integration of multiple line laser scans. Experimental evaluation in both air and underwater environments confirms the advantages of the proposed method.

We propose a technique to estimate a high-resolution depth image from a sparse depth image captured by depth camera and a high-resolution shading image obtained by a RGB camera using deep neural network (DNN). In our technique, the network model is pretrained by synthetic images which are generated by rendering high-frequency shapes created by arithmetic model, such as sinusoidal wave of wide variation of parameters. Although the preparation of an appropriate synthetic dataset is critical for such tasks, it is not trivial to find a compact and optimal distribution of shape parameters. In this paper, we propose an auto augmentation technique to optimize hyperparameters for shapes achieving minimum number for training DNN. The proposed augmentation network directly optimizes the hyperparameters of a 3D scene including parameters of procedural shapes and their positions by gradient descent algorithm via a differentiable rendering technique. Unlike previous data augmentation techniques which only have basic image processing methods, such as affine and color transformations, the proposed method can generate optimal training dataset by changing the 3D shape and its position by using a differentiable renderer. In our experiments, we confirmed that our method improved the accuracy of high-resolution depth estimation as well as efficiency of training the network.

Importance of structured-light based one-shot scanning technique is increasing because of its simple system configuration and ability of capturing moving objects. One severe limitation of the technique is that it can capture only sparse shape, but not high frequency shapes, because certain area of projection pattern is required to encode spatial information. In this paper, we propose a technique to recover high-frequency shapes by using shading information, which is captured by one-shot RGB-D sensor based on structured light with single camera. Since color image comprises shading information of object surface, high-frequency shapes can be recovered by shape from shading techniques. Although multiple images with different lighting positions are required for shape from shading techniques, we propose a learning based approach to recover shape from a single image. In addition, to overcome the problem of preparing sufficient amount of data for training, we propose a new data augmentation method for high-frequency shapes using synthetic data and domain adaptation. Experimental results are shown to confirm the effectiveness of the proposed method.

Among various 3D capturing systems, since the system with line lasers based on the light sectioning method is simple and accurate, it has widely attracted many developers and used for many purposes. In addition, there is no need to synchronize the camera and the laser and also the configuration of the camera and the lasers is flexible, and thus, the system can be used for extreme conditions, such as underwater. There are two open problems for the system. The first problem is a low density of the 3D shape obtained from a single image, i.e., just several curves. The second problem is the accuracy of line detection in the wild. In this paper, we propose a self-calibration method using visual odometry (VO) to bundle a large number of frames to increase the density to solve the first problem. We also propose a robust line detection algorithm using CNN to solve the second problem. Comparative experiments prove the effectiveness of our proposed method. In addition, the system was tested in the extreme condition for demonstration.

The decomposition of light transport into direct and global components, diffuse and specular interreflections, and subsurface scattering allows for new visualizations of light in everyday scenes. In particular, indirect light contains a myriad of information about the complex appearance of materials useful for computer vision and inverse rendering applications. In this paper, we present a new imaging technique that captures and analyzes components of indirect light via light transport using a synchronized projector-camera system. The rectified system illuminates the scene with epipolar planes corresponding to projector rows, and we vary two key parameters to capture plane-to-ray light transport between projector row and camera pixel: (1) the offset between projector row and camera row in the rolling shutter (implemented as synchronization delay), and (2) the exposure of the camera row. We describe how this synchronized rolling shutter performs illumination multiplexing, and develop a nonlinear optimization algorithm to demultiplex the resulting 3D light transport operator. Using our system, we are able to capture live short and long-range non-epipolar indirect light transport, disambiguate subsurface scattering, diffuse and specular interreflections, and distinguish materials according to their subsurface scattering properties. In particular, we show the utility of indirect imaging for capturing and analyzing the hidden structure of veins in human skin.

Separation of light transport into direct and indirect paths has enabled new visualizations of light in everyday scenes. However, indirect light itself contains a variety of components from subsurface scattering to diffuse and specular interreflections, all of which contribute to complex visual appearance. In this paper, we present a new imaging technique that captures and analyzes these components of indirect light via light transport between epipolar planes of illumination and rays of received light. This plane-to-ray light transport is captured using a rectified projector-camera system where we vary the offset between projector and camera rows (implemented as synchronization delay) as well as the exposure of each camera row. The resulting delay-exposure stack of images can capture live short and long-range indirect light transport, disambiguate subsurface scattering, diffuse and specular interreflections, and distinguish materials according to their subsurface scattering properties.

Light interacts with a scene in various ways. For scene understanding, a light transport is useful because it describes a relationship between the incident light ray and the result of the interaction. Our goal is to acquire the 4D light transport between the projector and the camera, focusing on direct and short-range transport that include the effect of the diffuse reflections, subsurface scattering, and inter-reflections. The acquisition of the light transport is challenging since the acquisition of the full 4D light transport requires a large number of measurement. We propose an efficient method to acquire short range light transport, which is dominant in the general scene, using synchronized projector-camera system. We show the transport profile of various materials, including uniform or heterogeneous subsurface scattering.