3D reconstruction has been widely studied and applied in various fields. Multi-view stereo (MVS) methods can recover dense geometry from multiple views, but often fail for texture-less objects due to unreliable feature matching. Active stereo with structured light (SL) addresses this limitation, however, when using multiple cameras and projectors for entire shape acquisition, overlapped SL patterns interfere with one another, leading to decoding failures. We propose a novel MVS framework based on neural signed distance field (Neural SDF) with multiple projectors that employs Direct Sequence Spread Spectrum (DSSS) to separate multiplexed patterns. This approach enables robust and accurate 3D shape reconstruction through Neural SDF optimization with a photometric loss that accounts for both the positions and the patterns of the projectors. We built a real scanning device that surrounds an object and captures its entire shape at once. Experiments on several static and a dynamics are conducted to validate the effectiveness of the proposed method.

Active stereo systems based on static pattern projection, a.k.a. oneshot scan, have been widely used for measuring dynamic scenes. Many patterns used for oneshot active stereo have grid-structures and grid-wise codes. For such systems, the grid structure is first detected, and graph matching methods are applied to estimate correspondences. However, such graph matching is often vulnerable to graph connection errors caused by grid structure analysis based on image features. Also, dense reconstruction for such systems is an open problem, where pixel-wise correspondence estimation from sparse image features is required. We propose a learning-based method to capture grid structure information and pixel-wise positional information simultaneously. We also propose to represent the grid structure by graphs with augmented connections other than 4-neighbor connections and applying them to a graph convolutional network (GCN). The proposed method can analyze large variety of grid patterns, has auto-calibration capability, can reconstruct dense shapes for fast moving objects.

When dynamic objects are captured by a camera, motion blur inevitably occurs. Such a blur is sometimes considered as just a noise, however, it sometimes gives an important effect to add dynamism in the scene for photographs or videos. Unlike the similar effects, such as defocus blur, which is now easily controlled even by smartphones, motion blur is still uncontrollable and makes undesired effects on photographs. In this paper, an unified framework to add motion blur on per-object basis is proposed. In the method, multiple frames are captured without motion blur and they are accumulated to create motion blur on target objects. To capture images without motion blur, shutter speed must be short, however, it makes captured images dark, and thus, a sensor gain should be increased to compensate it. Since a sensor gain causes a severe noise on image, we propose a color compensation algorithm based on non-linear filtering technique for solution. Another contribution is that our technique can be used to make HDR images for fast moving objects by using multi-exposure images. In the experiments, effectiveness of the method is confirmed by ablation study using several data sets.

Simultaneous registration and shape fusion using 3D scanners have been proposed for conducting wide-area and dense 3D shape reconstruction. However, because the 3D scanners for such a system must be robust and should provide feedback in real time, only a few devices are available, thereby limiting the application of the technique. In this study, we propose a new wide-area scanning algorithm that only requires an off-the-shelf projector and a camera. In our technique, the devices are not necessarily fixed to each other and the relative positions of the devices as well as the scene shapes can be precisely estimated by bundle adjustment (BA) in case of structured light. To efficiently perform shape registration, a robust and dense shape reconstruction is required, which is currently considered to be an open problem for structured light systems. In this study, we suggest a novel network-based feature detection algorithm as well as shape fusion algorithm for the solution.

Dense 3D shape acquisition of swimming human or live fish is an important research topic for sports, biological science and so on. For this purpose, active stereo sensor is usually used in the air, however it cannot be applied to the underwater environment because of refraction, strong light attenuation and severe interference of bubbles. Passive stereo is a simple solution for capturing dynamic scenes at underwater environment, however the shape with textureless surfaces or irregular reflections cannot be recovered. Recently, the stereo camera pair with a pattern projector for adding artificial textures on the objects is proposed. However, to use the system for underwater environment, several problems should be compensated, i.e., disturbance by fluctuation and bubbles. Simple solution is to use convolutional neural network for stereo to cancel the effects of bubbles and/or water fluctuation. Since it is not easy to train CNN with small size of database with large variation, we develop a special bubble generation device to efficiently create real bubble database of multiple size and density. In addition, we propose a transfer learning technique for multi-scale CNN to effectively remove bubbles and projected-patterns on the object. Further, we develop a real system and actually captured live swimming human, which has not been done before. Experiments are conducted to show the effectiveness of our method compared with the state of the art techniques.

Demands on capturing dynamic scenes of underwater environments are rapidly growing. Passive stereo is applicable to capture dynamic scenes, however the shape with textureless surfaces or irregular reflections cannot be recovered by the technique. In our system, we add a pattern projector to the stereo camera pair so that artificial textures are augmented on the objects. To use the system at underwater environments, several problems should be compensated, i.e., refraction, disturbance by fluctuation and bubbles. Further, since surface of the objects are interfered by the bubbles, projected patterns, etc., those noises and patterns should be removed from captured images to recover original texture. To solve these problems, we propose three approaches; a depth-dependent calibration, Convolutional Neural Network(CNN)-stereo method and CNN-based texture recovery method. A depth-dependent calibration I sour analysis to find the acceptable depth range for approximation by center projection to find the certain target depth for calibration. In terms of CNN stereo, unlike common CNN based stereo methods which do not consider strong disturbances like refraction or bubbles, we designed a novel CNN architecture for stereo matching using multi-scale information, which is intended to be robust against such disturbances. Finally, we propose a multi-scale method for bubble and a projected-pattern removal method using CNNs to recover original textures. Experimental results are shown to prove the effectiveness of our method compared with the state of the art techniques. Furthermore, reconstruction of a live swimming fish is demonstrated to confirm the feasibility of our techniques.

Shape reconstruction techniques using structured light have been widely researched and developed due to their robustness, high precision, and density. Because the techniques are based on decoding a pattern to find correspondences, it implicitly requires that the projected patterns be clearly captured by an image sensor, i.e., to avoid defocus and motion blur of the projected pattern. Although intensive researches have been conducted for solving defocus blur, few researches for motion blur and only solution is to capture with extremely fast shutter speed. In this paper, unlike the previous approaches, we actively utilize motion blur, which we refer to as a light flow, to estimate depth. Analysis reveals that minimum two light flows, which are retrieved from two projected patterns on the object, are required for depth estimation. To retrieve two light flows at the same time, two sets of parallel line patterns are illuminated from two video projectors and the size of motion blur of each line is precisely measured. By analyzing the light flows, i.e. lengths of the blurs, scene depth information is estimated. In the experiments, 3D shapes of fast moving objects, which are inevitably captured with motion blur, are successfully reconstructed by our technique.

One of the solutions of depth imaging of moving scene is to project a static pattern on the object and use just a single image for reconstruction. However, if the motion of the object is too fast with respect to the exposure time of the image sensor, patterns on the captured image are blurred and reconstruction fails. In this paper, we impose multiple projection patterns into each single captured image to realize temporal super resolution of the depth image sequences. With our method, multiple patterns are projected onto the object with higher fps than possible with a camera. In this case, the observed pattern varies depending on the depth and motion of the object, so we can extract temporal information of the scene from each single image. The decoding process is realized using a learning-based approach where no geometric calibration is needed. Experiments confirm the effectiveness of our method where sequential shapes are reconstructed from a single image. Both quantitative evaluations and comparisons with recent techniques were also conducted.

3D reconstruction methods based on active stereo technique have been widely used for many practical systems. Many of these systems are configured with a single camera and a single projector. Since such systems can only capture one side of the target object, several attempts have been conducted to enlarge the captured area, especially multi-projector systems attract many researchers. For multi-projector based systems, overlap between multiple pattern projections is a serious problem. Even if different color channels are used for each projector, complete separation is not possible because of color crosstalks. Another open problem is decoding errors of the projected patterns, which causes a failure on extracting positional information of the projected pattern form the captured image. Among several reasons for such errors, color crosstalks are crucial because their features are similar to the main signal and difficult to be decomposed. In this paper, we solve these problems by utilizing machine learning techniques where a convolutional neural network is trained to extract low dimensional pattern features for each projector. In addition, it is trained to suppress the color crosstalks from different projectors. Using this new technique, we succeeded in reconstructing 3D shapes from images where multiple patterns are overlapped.

Underwater 3D shape scanning technique becomes popular because of several rising research topics, such as map making of submarine topography for autonomous underwater vehicle (UAV), shape measurement of live fish, motion capture of swimming human, etc. Structured light systems (SLS) based active 3D scanning systems are widely used in the air and also promising to apply underwater environment. When SLS is used in the air, the stereo correspondences can be efficiently retrieved by epipolar constraint. However, in the underwater environment, the camera and projector are usually set in special housings and refraction occurs at the interfaces between water/glass and glass/air, resulting in invalid conditions for epipolar constraint which severely deteriorates the correspondence search process. In this paper, we propose an efficient technique to calibrate the underwater SLS systems as well as robust 3D shape acquisition technique. In order to avoid the calculation complexity, we approximate the system with central projection model. Although such an approximation produces an inevitable errors in the system, such errors are diminished by a combination of grid based SLS technique and a bundle adjustment algorithm. We tested our method with a real underwater SLS, consisting ofcustom-made laser pattern projector and underwater housings, showing the validity ofour method.

One-shot active stereo using structured light is a practical solution for dynamic scene acquisition. Basically, those methods are based on encoding positional information of the pixel into the single projected pattern. A disadvantage of such methods is decreases of the spatial resolution caused by requiring a certain area of the pattern to encode the positional information. Among those methods, grid-based patterns are promising at the point of accuracy and robustness, since triangulation for 3D reconstruction is conducted with light-sectioning method and a line detection is usually a stable image processing. However, no shapes are recovered between the grid lines, and thus, the whole reconstructed shape tends to be sparse. To deal with the problem, integrating multiple shapes that are sequentially captured using registration algorithm such as ICP is one solution. In previous work, we show that naive ICP works poorly for grid-like structured point clouds, and proposed a specialized ICP algorithm for aligning a set of grid-like structured 3D shapes. In this paper, we extend this approach and propose a process for entire shape modeling by capturing objects from all the directions using turn table, and integrating into a single shape using our improved ICP. To achieve this, setting good initial 3D shapes is important. For solution, we interpolation grid shapes to create smooth surface so that common ICP works. Comprehensive experiments are conducted to show the strength of our method compared to common ICP.

Active one-shot scanning techniques have been widely used for various applications. Stereo-based active one-shot scanning embeds a positional information regarding the image plane of a projector onto a projected pattern to retrieve correspondences entirely from a captured image. Many combinations of patterns and decoding algorithms for active one-shot scanning have been proposed. If the capturing environment lacks the assumed conditions, such as the absence of strong external lights, then reconstruction using those methods is degraded, because the pattern decoding fails. In this paper, we propose a general reconstruction algorithm that can be used for any kind of patterns without strict assumptions. The technique is based on an efficient feature extraction function that can drastically reduce redundant information from the raw pixel values of patches of captured images. Shapes are reconstructed by efficiently finding correspondences between a captured image and the pattern using low-dimensional feature vectors. Such a function is created automatically by a convolutional neural network using a large database of pattern images that are efficiently synthesized by using GPU with wide variation of depth and surface orientation. Experimental results show that our technique can be used for several existing patterns without any ad hoc algorithm or information regarding the scene or the sensor.

The central projection model commonly used to model cameras as well as projectors, results in similar advantages and disadvantages in both types of system. Considering the case of active stereo systems using a projector and camera setup, a central projection model creates several problems, among them, narrow depth range and necessity of wide baseline are crucial. In the paper, we solve the problems by introducing a light field projector, which can project a depth-dependent pattern. The light field projector is realized by attaching a coded aperture with a high frequency mask in front of the lens of the video projector, which also projects a high frequency pattern. Because the light field projector cannot be approximated by a thin lens model and a precise calibration method is not established yet, an image-based approach is proposed to apply a stereo technique to the system. Although image-based techniques usually require a large database and often imply heavy computational costs, we propose a hierarchical approach and a feature-based search for solution. In the experiments, it is confirmed that our method can accurately recover the dense shape of curved and textured objects for a wide range of depths from a single captured image.

Dense 3D reconstruction of fast moving objects could contribute to various applications such as body structure analysis, accident avoidance, and so on. In this paper, we propose a technique based on a one-shot scanning method, which reconstructs 3D shapes for each frame of a high frame-rate video capturing the scenes projected by a static pattern. To avoid instability of image processing, we restrict the number of colors used in the pattern to less than two. The proposed technique comprises (1) an efficient algorithm to eliminate ambiguity of projected parallel-line patterns by using intersection points, (2) a batch reconstruction algorithm of multiple frames by using spatio-temporal constraints, and (3) an efficient detection method of color-encoded grid pattern based on de Bruijn sequence. In the experiments, the line detection algorithm worked effectively and the dense reconstruction algorithm produces accurate and robust results. We also show the improved results by using temporal constraints. Finally, the dense reconstructions of fast moving objects in a high frame-rate video are presented.

In this paper, we propose an active 3D reconstruction method with two cameras and one projector (2C1P) system for capturing moving objects. The system reconstructs the shapes from a single frame of each camera by finding the correspondence between the cameras and the projector Based on projecting wave grid pattern. The projected pattern gives the constraint of correspondence between the two cameras in addition to between a projector and a camera. The proposed method finds correspondence by energy minimization on graphs constructed by detecting a grid pattern in camera images. Since the graphs of two cameras are connected as a single graph by using the constraint between cameras, the proposed method simultaneously finds the correspondences for two cameras, which contributes to the robustness of correspondence search. By merging range images created by the correspondence of each camera, we reduce the occluded area compared to the case of one camera. Finally, the proposed method optimizes the shape as three-view stereo to improve the accuracy of shape measurements. In the experiment, we show the effectiveness of using two cameras by making comparison with the case of one camera.

In this study, the authors propose a new technique to achieve one‐shot scan using single colour and static pattern projector; such a method is ideal for acquisition of moving objects. Since projector–camera systems generally have uncertainties on retrieving correspondences between the captured image and the projected pattern, many solutions have been proposed. Especially for one‐shot scan, which means that only a single image is required for shape reconstruction, positional information of a pixel of the projected pattern should be encoded by spatial and/or colour information. Although colour information is frequently used for encoding, it is severely affected by texture and material of the object and leads unstable reconstruction. In this study, the authors propose a technique to solve the problem by using geometrically unique pattern only with black and white colour that further considers the shape distortion by surface orientation of the shape. The authors technique successfully acquires high‐precision one‐shot scan with an actual system.

3D scanning of moving objects has many applications, for example, marker-less motion capture, analysis on fluid dynamics, object explosion and so on. One of the approach to acquire accurate shape is a projector-camera system, especially the methods that reconstructs a shape by using a single image with static pattern is suitable for capturing fast moving object. In this paper, we propose a method that uses a grid pattern consisting of sets of parallel lines. The pattern is spatially encoded by a periodic color pattern. While informations are sparse in the camera image, the proposed method extracts the dense (pixel-wise) phase informations from the sparse pattern. As the result, continuous regions in the camera images can be extracted by analyzing the phase. Since there remain one DOF for each region, we propose the linear solution to eliminate the DOF by using geometric informations of the devices, i.e. epipolar constraint. In addition, solution space is finite because projected pattern consists of parallel lines with same intervals, the linear equation can be efficiently solved by integer least square method. In this paper, the formulations for both single and multiple projectors are presented. We evaluated the accuracy of correspondences and showed the comparison with respect to the number of projectors by simulation. Finally, the dense 3D reconstruction of moving objects are presented in the experiments.

Recently it is relatively easy to produce digital point sampled 3D geometric models. In sight of the increasing capability of 3D scanning systems to produce models with millions of points, compression efficiency is of paramount importance. In this paper, we propose a novel competition-based predictive method for single-rate compression of 3D models represented as point cloud. In particular we aim at 3D scanning methods based on grid pattern. The proposed method takes advantage of the pattern characteristic made of vertical and horizontal lines, by assuming that the object surface is sampled in curve of points. We then designed and implemented a predictive coder driven by this curve-based point representation. Novel prediction techniques are specifically designed for a curve-based cloud of points, and been competing between them to achieve high quality 3D reconstruction. Experimental results demonstrate the effectiveness of the proposed method.

Dense 3D reconstruction of extremely fast moving objects could contribute to various applications such as body structure analysis and accident avoidance and so on. The actual cases for scanning we assume are, for example, acquiring sequential shape at the moment when an object explodes, or observing fast rotating turbine's blades. In this paper, we propose such a technique based on a one-shot scanning method that reconstructs 3D shape from a single image where dense and simple pattern are projected onto an object. To realize dense 3D reconstruction from a single image, there are several issues to be solved; e.g. instability derived from using multiple colors, and difficulty on detecting dense pattern because of influence of object color and texture compression. This paper describes the solutions of the issues by combining two methods, that is (1) an efficient line detection technique based on de Bruijn sequence and belief propagation, and (2) an extension of shape from intersections of lines method. As a result, a scanning system that can capture an object in fast motion has been actually developed by using a high-speed camera. In the experiments, the proposed method successfully captured the sequence of dense shapes of an exploding balloon, and a breaking ceramic dish at 300–1000 fps.

An active 3D scanning method that can capture a fast motion is strongly required in wide areas. Since most commercial products using a laser projector basically reconstruct the shape with a point or a line for each scan, a fast motion cannot be captured in principle. In addition, an extended method using a structured light can drastically reduce the number of projecting patterns, however, they still require several patterns and a fast motion cannot be captured. One solution for the purpose is to use a single pattern (one shot scan). Although, one shot scanning methods have been intensively studied, they often have stability problems and their result tend to have low resolution. In this paper, we develop a new system which achieves dense and robust 3D measurement from a single grid pattern. A 3D reconstruction can be achieved by identifying grid patterns using coplanarity constraints. We also propose a coarse-to-fine method to increase the density of the shape with a single pattern. 1

3D acquisition techniques to measure dynamic scenes and deformable objects with little texture are extensively researched for applications like the motion capturing of human facial expression. To allow such measurement, several techniques using structured light have been proposed. These techniques can be largely categorized into two types. The first involves techniques to temporally encode positional information of a projector’s pixels using multiple projected patterns, and the second involves techniques to spatially encode positional information into areas or color spaces. Although the former allows dense reconstruction with a sufficient number of patterns, it has difficulty in scanning objects in rapid motion. The latter technique uses only a single pattern, so this problem can be resolved, however, it often uses complex patterns or color intensities, which are weak to noise, shape distortions, or textures. Thus, it remains an open problem to achieve dense and stable 3D acquisition in real cases. In this paper, we propose a technique to achieve dense shape reconstruction that requires only a single-frame image of a grid pattern. The proposed technique also has the advantage of being robust in terms of image processing.

Methods for scanning dynamic scenes are important in many applications and many systems using structured light have been proposed. Many of these systems use either multiple patterns projected rapidly or a single pattern. Although the former allows dense reconstruction with a sufficient number of patterns, it has difficulty in capturing objects in rapid motion. The latter technique uses only a single pattern and have no such difficulties, however, they often have stability problems and their result tend to have low resolution. In this paper, we develop a system to achieve dense and accurate 3D measurement from only a single image. The proposed system also has the advantage of being robust in terms of image processing.

So far, there have been many techniques and applications developed for handling 3D contents efficiently on the Web. However, there is currently no application that successfully makes 3D contents pervasive on the Web. One of the main reasons for this can be considered be the fact that there are only a few pages on the Web where a 3D object is sufficiently annotated with meta-information; meta-information is fundamental for the Web. In this paper, (1) a semi-automatic annotation module which supports users in creating 3D contents with rich meta-information by collecting and filtering meta-data on the Web using 3D information; and (2) a view-dependent information representation module which shows information corresponding to the viewing direction of 3D objects are presented. In those modules, view-dependent information representation module is most important in our system, because such system is highly suitable to show meta-information to users, however, not proposed yet. We implemented the proposed system and conducted experiments which show that it is possible to realize semi-automatic annotation of meta-data on 3D objects and efficiently present view-dependent meta-information to users.

Coplanarity is a relationship of a set of points that exist on a single plane. Coplanarities can be easily observed in a scene with planer surfaces, and these types of coplanarities have been widely used for 3D reconstructions based on geometrical constraints. Other types of coplanarities that can be observed from images are those observed as cross sections of planes and scenes; for example, points lit by a line laser, or boundary points of a shadow of a straight edge. Although these types of coplanarities have been implicitly used in variations of light sectioning methods, they have not been used in an unified manner with the former types. In this paper, we describe a new 3D reconstruction method based on coplanarities and other geometrical constraints. In particular, we make use of the above two types of coplanarities in an unified manner. This enables us to reconstruct 3D scenes scanned using line lasers or shadows of straight edges observed by a partially-calibrated single camera utilizing geometrical relationships between the planes in the scenes and the planes of line lasers or the planes of shadow boundaries. 1

In this paper, we propose an uncalibrated, multi-image 3D reconstruction technique, using coded structured light. Normally, a conventional coded structured light system consists of a camera and a projector and needs precalibration before scanning. Since the camera and the projector have to be fixed after calibration, reconstruction of a wide area of the scene or reducing occlusions are difficult and sometimes impossible. In the proposed method, precalibration can be successfully omitted by applying the uncalibrated stereo technique, thereby multiple scanning while moving the camera or the projector is possible. As the result, users can freely move either the cameras or projectors to scan a wide range of objects.

In this paper, we propose a novel synthesis technique for BTFs. A BTF (bidirectional texture function) is a 6D function which can represent appearances of a texture under arbitrary view and lighting conditions. Until now, several approaches of BTF synthesis have been researched. For ordinary textures, patch based methods are promising techniques for texture synthesis. However, it has not been effectively tried to BTFs yet. This is mainly because data size of BTFs is so large and it is not easy to apply the techniques to BTFs. Further, efficient rendering of BTFs is still under research. In this paper, we extend a patch based synthesis technique for BTFs by utilizing compact representation of BTFs. In addition, we propose a novel technique to render the synthesized BTFs efficiently. With our proposed method, we can successfully and effectively render objects with complicated surfaces under arbitrary sizes.

This paper presents a new contour tracking method based on active contour model. In this method, we introduce a new image space that integrates the idea of multiscale (pyramid) image and spatio-temporal solid. We applied an active contour model shaped like a tube. This model can deal with relatively large shifts of target objects, and can be applicable to movies that has arbitrary length.