Recently, 3D reconstruction from multiview stereo (MVS) has advanced significantly with the introduction of neural implicit representation methods, which estimate voxel densities or signed distance fields (SDFs) to describe the 3D structure of a scene. Although such neural-based methods typically require a large number of captured images to estimate dense volumetric information during training, developing systems that can recover the 3D shape of moving objects using only a small number of stationary cameras remains highly demanding and challenging. To address the issue of sparse views, various active lighting techniques have been proposed. However, the problem remains inherently difficult, particularly when attempting to capture the complete shape of an object with a wide baseline. In this paper, we propose a novel approach that combines active lighting with photometric stereo (PS) using neural representations. Additionally, we introduce a multiplexed illumination technique that captures the entire shape of an object in a single shot. Although this results in a low signal-to-noise ratio (SNR), our method also addresses this issue. The advantages of our technique are demonstrated through real-world experiments, showcasing its ability to capture a 4D scene.

3D reconstruction has been widely studied and applied in various fields. Multi-view stereo (MVS) methods can recover dense geometry from multiple views, but often fail for texture-less objects due to unreliable feature matching. Active stereo with structured light (SL) addresses this limitation, however, when using multiple cameras and projectors for entire shape acquisition, overlapped SL patterns interfere with one another, leading to decoding failures. We propose a novel MVS framework based on neural signed distance field (Neural SDF) with multiple projectors that employs Direct Sequence Spread Spectrum (DSSS) to separate multiplexed patterns. This approach enables robust and accurate 3D shape reconstruction through Neural SDF optimization with a photometric loss that accounts for both the positions and the patterns of the projectors. We built a real scanning device that surrounds an object and captures its entire shape at once. Experiments on several static and a dynamics are conducted to validate the effectiveness of the proposed method.

In this paper, we introduce MeshMamba, a neural network model for learning 3D articulated mesh models by employing the recently proposed Mamba State Space Models (Mamba-SSMs). MeshMamba is efficient and scalable in handling a large number of input tokens, enabling the generation and reconstruction of body mesh models with more than 10,000 vertices, capturing clothing and hand geometries. The key to effectively learning MeshMamba is the serialization technique of mesh vertices into orderings that are easily processed by Mamba. This is achieved by sorting the vertices based on body part annotations or the 3D vertex locations of a template mesh, such that the ordering respects the structure of articulated shapes. Based on MeshMamba, we design 1) MambaDiff3D, a denoising diffusion model for generating 3D articulated meshes and 2) Mamba-HMR, a 3D human mesh recovery model that reconstructs a human body shape and pose from a single image. Experimental results showed that MambaDiff3D can generate dense 3D human meshes in clothes, with grasping hands, etc., and outperforms previous approaches in the 3D human shape generation task. Additionally, Mamba-HMR extends the capabilities of previous non-parametric human mesh recovery approaches, which were limited to handling body-only poses using around 500 vertex tokens, to the whole-body setting with face and hands, while achieving competitive performance in (near) real-time.

Time-of-flight (ToF) cameras, widely used in commercial applications such as augmented reality and autonomous driving, measure depth by analyzing the flight time of emitted laser signals. Indirect ToF (I-ToF) cameras are particularly popular due to their high resolution, high frame rate, and affordability. However, they struggle with depth estimation in scenes containing translucent objects, as their fundamental assumption – single direct reflectance – breaks down due to complex light interactions. In this paper, we address depth estimation with translucent objects by leveraging coding curve (CC) distortions, which have recently been shown to mitigate multi-path interference (MPI). The CC is formulated as a function derived from multiple values of various types of temporal encoding by the photodetector at each pixel. Specifically, inspired by recent MPI solutions using CC, we sample both values and depth errors under translucent objects at fixed intervals to train a model that predicts foreground and background depth errors from the CC, ensuring accurate reconstruction of translucent scenes through error correction. Our approach is validated through real-world experiments, demonstrating its effectiveness in improving depth estimation in scenes with translucent objects.

The indirect time-of-flight camera measures depth based on the phase shift between modulated laser and its reflected light. However, when there are multi-paths due to interreflections, depth estimation is significantly affected as it assumes that only a single reflected light is observed. In this paper, we propose a method for mitigating multi-path interference utilizing coding curves derived from measurements of multiple sensor taps, which represent the contribution of global light component, i.e., multi-path. First, to obtain the coding curve, we offset the laser light to the sensor and measure the sensor tap values for different delays. We propose a data-driven method for mitigating MPI, utilizing a dataset where distorted coding curves are paired with MPI intensity. By using the measured coding curve as a query to obtain correction values from the dataset, we can mitigate the effect of MPI and obtain the correct depth. Unlike previous methods, our approach can solve the general MPI problem while imposing fewer restrictions on the type of reflections, the number of light paths, and the modulation frequency. We validate the effectiveness of our method through both simulation and real experiments.

Event cameras are recent sensors that measure intensity changes in each pixel asynchronously. It is being used due to lower latency and higher temporal resolution compared to traditional frame-based camera. We propose a method of 3D model-based object tracking directly from events captured by event camera. To enable reliable and accurate tracking of objects, we use a new event representation and predict brightness increment images with motion interpolation. Results of object tracking show the new methods significantly improves tracking duration and robustness, both for perspective and fisheye cameras. Our implementation succeeds in tracking objects when the camera speed is reaching 2 m/s.

Recognizing daily human activities is essential for domestic robots to assist humans effectively in indoor environments. These activities typically involve sequences of interactions between humans and objects across different locations and times within a household. Identifying these events and understanding their temporal and spatial relationships is crucial for accurately modeling human behavior patterns. However, most current methods and datasets for human activity recognition focus on identifying singular events at specific moments and locations, neglecting the complexity of activities that span multiple times and places. To address this gap, we collected data on human activity patterns over a single day through crowdsourcing. Based on this, we introduce a novel synthetic video question-answering dataset. Our proposed dataset includes videos of daily activities accompanied by question-answer pairs that require models to reason about sequences of activities in both time and space. We evaluated state-of-the-art methods against our dataset, highlighting their limitations in handling the intricate spatio-temporal dynamics of human activity sequences. To improve upon these methods, we propose a two-stage model. The proposed model initially decodes the detailed content of individual videos using a transformer-based approach, then employs LLMs for advanced spatio-temporal reasoning across multiple videos. We hope our research provides valuable benchmarks and insights, paving the way for advancements in the recognition of daily human activity patterns.

Optical sensors, such as the Far Infrared (FIR) sensor, have demonstrated advantages over traditional imaging. For example, 3D reconstruction in the FIR field captures the heat distribution of a scene that is invisible to RGB, aiding various applications like gas leak detection. However, less texture information and challenges in acquiring FIR frames hinder the reconstruction process. Given that implicit neural representations (INRs) can integrate geometric information across different sensors, we propose Implicit Neural Fusion (INF) of RGB and FIR for 3D reconstruction of invisible scenes in the FIR field. Our method first obtains a neural density field of objects from RGB frames. Then, with the trained object density field, a separate neural density field of gases is optimized using limited view inputs of FIR frames. Our method not only demonstrates outstanding reconstruction quality in the FIR field through extensive experiments but also can isolate the geometric information of the invisible, offering a new dimension of scene understanding.

This paper presents a novel method for detecting and estimating contact forces only from human motions using machine learning techniques. Knowing the location of the contacts with the environment and the magnitude of the exerted force is critical for dynamic human motion analysis. However, their annotation is usually made manually from captured motion data especially in case of multiple contacts even if the data includes force measurement. Moreover, most existing human motion datasets do not include contact force. To overcome these bottlenecks, we introduce a network that leverages vector-quantized variational autoencoder (VQ-VAE) and self-attention that learns a small set of discrete feature values representing various contact states. These feature values, called contact codes, allow human motions to be converted to contact states and resulting forces. By applying an optimization for contact estimation with a reduced set of manual annotations, the existence of contacts can be automatically determined, which is essential information for dynamic analysis. We validated the effectiveness and potential usefulness of the proposed method with a human walking gait dataset, by converting the human motions into contact sequences and forces and applying the estimated contacts to dynamic motion analysis.

Active stereo systems based on static pattern projection, a.k.a. oneshot scan, have been widely used for measuring dynamic scenes. Many patterns used for oneshot active stereo have grid-structures and grid-wise codes. For such systems, the grid structure is first detected, and graph matching methods are applied to estimate correspondences. However, such graph matching is often vulnerable to graph connection errors caused by grid structure analysis based on image features. Also, dense reconstruction for such systems is an open problem, where pixel-wise correspondence estimation from sparse image features is required. We propose a learning-based method to capture grid structure information and pixel-wise positional information simultaneously. We also propose to represent the grid structure by graphs with augmented connections other than 4-neighbor connections and applying them to a graph convolutional network (GCN). The proposed method can analyze large variety of grid patterns, has auto-calibration capability, can reconstruct dense shapes for fast moving objects.

The paper proposes a method for automatic multi-view dataset construction based on formula-driven supervised learning (FDSL). Although data collection and human annotation of 3D objects are labor-intensive, we automatically generate their training data and labels in the proposed multi-view dataset. To create a large-scale multi-view dataset, we employ fractal geometry, which is considered the background information of many objects in the real world. We project in a circle from the rendered 3D fractal models to construct the Multi-view Fractal DataBase (MV-FractalDB), which is then used to make a pre-trained CNN model. According to the experimental results, the MV-FractalDB pre-trained model surpasses the accuracies with self-supervised methods (e.g., SimCLR and MoCo) and is close to supervised methods (e.g., ImageNet) in terms of performance rates on multi-view image datasets. We demonstrate the potential of FDSL for multi-view image recognition.

Recent advances in deep learning have shown high success in obtaining the 6-DoF pose of rigid objects. However, most works rely on a pre-existing dataset and do not tackle the data gathering part. The time-consuming and tedious tasks required to build datasets are, to a large extent, what is keeping these techniques from being more widely used in practical applications. We present a whole pipeline from data gathering to pose recognition and an example application of robot grasping. For our data gathering method we require as minimum user intervention as possible and, even without using depth information or 3D models, by using a novel RGB-only Neural Network design we are able to obtain results very close to the state of the art. We call this method Affordable Pose Estimation (APE).

Many algorithms of computer vision use light sources to illuminate objects to actively create situation appropriate to extract their characteristics. For example, the shape and reflectance are measured by a projector-camera system, and some human-machine or VR systems use projectors and displays for interaction. As existing active lighting systems usually assume no severe external lights to observe projected lights clearly, it is one of the limitations of active illumination. In this paper, we propose a method of energy-efficient active illumination in an environment with severe external lights. The proposed method extracts the light signals of illuminants by removing external light using spread spectrum modulation. Because an image sequence is needed to observe modulated signals, the proposed method extends signal processing to realize signal detection projected onto moving objects by combining spread spectrum modulation and spatio-temporal filtering. In the experiments, we apply the proposed method to a structured-light system under sunlight, to photometric stereo with external lights, and to insensible image embedding.

Shape reconstruction techniques using structured light have been widely researched and developed due to their robustness, high precision, and density. Because the techniques are based on decoding a pattern to find correspondences, it implicitly requires that the projected patterns be clearly captured by an image sensor, i.e., to avoid defocus and motion blur of the projected pattern. Although intensive researches have been conducted for solving defocus blur, few researches for motion blur and only solution is to capture with extremely fast shutter speed. In this paper, unlike the previous approaches, we actively utilize motion blur, which we refer to as a light flow, to estimate depth. Analysis reveals that minimum two light flows, which are retrieved from two projected patterns on the object, are required for depth estimation. To retrieve two light flows at the same time, two sets of parallel line patterns are illuminated from two video projectors and the size of motion blur of each line is precisely measured. By analyzing the light flows, i.e. lengths of the blurs, scene depth information is estimated. In the experiments, 3D shapes of fast moving objects, which are inevitably captured with motion blur, are successfully reconstructed by our technique.

3D reconstruction methods based on active stereo technique have been widely used for many practical systems. Many of these systems are configured with a single camera and a single projector. Since such systems can only capture one side of the target object, several attempts have been conducted to enlarge the captured area, especially multi-projector systems attract many researchers. For multi-projector based systems, overlap between multiple pattern projections is a serious problem. Even if different color channels are used for each projector, complete separation is not possible because of color crosstalks. Another open problem is decoding errors of the projected patterns, which causes a failure on extracting positional information of the projected pattern form the captured image. Among several reasons for such errors, color crosstalks are crucial because their features are similar to the main signal and difficult to be decomposed. In this paper, we solve these problems by utilizing machine learning techniques where a convolutional neural network is trained to extract low dimensional pattern features for each projector. In addition, it is trained to suppress the color crosstalks from different projectors. Using this new technique, we succeeded in reconstructing 3D shapes from images where multiple patterns are overlapped.

Underwater 3D shape scanning technique becomes popular because of several rising research topics, such as map making of submarine topography for autonomous underwater vehicle (UAV), shape measurement of live fish, motion capture of swimming human, etc. Structured light systems (SLS) based active 3D scanning systems are widely used in the air and also promising to apply underwater environment. When SLS is used in the air, the stereo correspondences can be efficiently retrieved by epipolar constraint. However, in the underwater environment, the camera and projector are usually set in special housings and refraction occurs at the interfaces between water/glass and glass/air, resulting in invalid conditions for epipolar constraint which severely deteriorates the correspondence search process. In this paper, we propose an efficient technique to calibrate the underwater SLS systems as well as robust 3D shape acquisition technique. In order to avoid the calculation complexity, we approximate the system with central projection model. Although such an approximation produces an inevitable errors in the system, such errors are diminished by a combination of grid based SLS technique and a bundle adjustment algorithm. We tested our method with a real underwater SLS, consisting ofcustom-made laser pattern projector and underwater housings, showing the validity ofour method.

Active one-shot scanning techniques have been widely used for various applications. Stereo-based active one-shot scanning embeds a positional information regarding the image plane of a projector onto a projected pattern to retrieve correspondences entirely from a captured image. Many combinations of patterns and decoding algorithms for active one-shot scanning have been proposed. If the capturing environment lacks the assumed conditions, such as the absence of strong external lights, then reconstruction using those methods is degraded, because the pattern decoding fails. In this paper, we propose a general reconstruction algorithm that can be used for any kind of patterns without strict assumptions. The technique is based on an efficient feature extraction function that can drastically reduce redundant information from the raw pixel values of patches of captured images. Shapes are reconstructed by efficiently finding correspondences between a captured image and the pattern using low-dimensional feature vectors. Such a function is created automatically by a convolutional neural network using a large database of pattern images that are efficiently synthesized by using GPU with wide variation of depth and surface orientation. Experimental results show that our technique can be used for several existing patterns without any ad hoc algorithm or information regarding the scene or the sensor.

This paper describes the indoor snowfall simulation chamber and its application for evaluating visibility performance of vision-based sensors affected by environmental conditions. First, vision-based safety sensors are introduced and outdoor environmental requirements for these sensors are discussed. And also, the relationship between the visibility and the performance of vision based sensors is discussed and the current problems for evaluating visibility performance are described. Next, the indoor snowfall simulation method using expanded polystyrene beads (EPB) is explained and the spectral transmission measurement equipment for evaluating the visibility reduction is presented. Finally, the visibility reduction caused by simulated and artificial snowfall are measured for verifying the proposed indoor snowfall simulation chamber. Preliminary experiments are also carried out in both snowfall systems in order to evaluate various kinds of vision-based safety sensors for personal care robots.

For mobile robots, 3D acquisition is required to model the environment. Particularly for humanoid robots, a modeled environment is necessary to plan the walking control. This environment can include both static objects, such as a ground surface with obstacles, and dynamic objects, such as a person moving around the robot. This paper proposes a system for a robot to obtain a sufficiently accurate shape of the environment for walking on a ground surface with obstacles and a method to detect dynamic objects in the modeled environment, which is necessary for the robot to react to sudden changes in the scene. The 3D acquisition is achieved by a projector-camera system mounted on the robot head that uses a structured-light method to reconstruct the shapes of moving objects from a single frame. The acquired shapes are aligned and merged into a common coordinate system using the simultaneous localization and mapping method. Dynamic objects are detected as shapes that are inconsistent with the previous frames. Experiments were performed to evaluate the accuracy of the 3D acquisition and the robustness with regard to detecting dynamic objects when serving as the vision system of a humanoid robot.

We present a nonrigid shape matching technique for establishing correspondences of incomplete 3D surfaces that exhibit intrinsic reflectional symmetry. The key for solving the symmetry ambiguity problem is to use a point-wise local mesh descriptor that has orientation and is thus sensitive to local reflectional symmetry, e.g. discriminating the left hand and the right hand. We devise a way to compute the descriptor orientation by taking the gradients of a scalar field called the average diffusion distance (ADD). Because ADD is smoothly defined on a surface, invariant under isometry/scale and robust to topological errors, the robustness of the descriptor to non-rigid deformations is improved. In addition, we propose a graph matching algorithm called iterative spectral relaxation which combines spectral embedding and spectral graph matching. This formulation allows us to define pairwise constraints in a scale-invariant manner from k-nearest neighbor local pairs such that non-isometric deformations can be robustly handled. Experimental results show that our method can match challenging surfaces with global intrinsic symmetry, data incompleteness and non-isometric deformations.

Dense 3D reconstruction of fast moving objects could contribute to various applications such as body structure analysis, accident avoidance, and so on. In this paper, we propose a technique based on a one-shot scanning method, which reconstructs 3D shapes for each frame of a high frame-rate video capturing the scenes projected by a static pattern. To avoid instability of image processing, we restrict the number of colors used in the pattern to less than two. The proposed technique comprises (1) an efficient algorithm to eliminate ambiguity of projected parallel-line patterns by using intersection points, (2) a batch reconstruction algorithm of multiple frames by using spatio-temporal constraints, and (3) an efficient detection method of color-encoded grid pattern based on de Bruijn sequence. In the experiments, the line detection algorithm worked effectively and the dense reconstruction algorithm produces accurate and robust results. We also show the improved results by using temporal constraints. Finally, the dense reconstructions of fast moving objects in a high frame-rate video are presented.

In this paper, we propose an active 3D reconstruction method with two cameras and one projector (2C1P) system for capturing moving objects. The system reconstructs the shapes from a single frame of each camera by finding the correspondence between the cameras and the projector Based on projecting wave grid pattern. The projected pattern gives the constraint of correspondence between the two cameras in addition to between a projector and a camera. The proposed method finds correspondence by energy minimization on graphs constructed by detecting a grid pattern in camera images. Since the graphs of two cameras are connected as a single graph by using the constraint between cameras, the proposed method simultaneously finds the correspondences for two cameras, which contributes to the robustness of correspondence search. By merging range images created by the correspondence of each camera, we reduce the occluded area compared to the case of one camera. Finally, the proposed method optimizes the shape as three-view stereo to improve the accuracy of shape measurements. In the experiment, we show the effectiveness of using two cameras by making comparison with the case of one camera.

In this study, the authors propose a new technique to achieve one‐shot scan using single colour and static pattern projector; such a method is ideal for acquisition of moving objects. Since projector–camera systems generally have uncertainties on retrieving correspondences between the captured image and the projected pattern, many solutions have been proposed. Especially for one‐shot scan, which means that only a single image is required for shape reconstruction, positional information of a pixel of the projected pattern should be encoded by spatial and/or colour information. Although colour information is frequently used for encoding, it is severely affected by texture and material of the object and leads unstable reconstruction. In this study, the authors propose a technique to solve the problem by using geometrically unique pattern only with black and white colour that further considers the shape distortion by surface orientation of the shape. The authors technique successfully acquires high‐precision one‐shot scan with an actual system.

In this paper, we propose a method of inertial sensor-based gait authentication by inter-period phase registration of an owner's gallery. In spite of the importance for gait authentication of constructing a gallery of phase-registered gait patterns, previous implementations just relied on simple methods of period detection based on heuristic knowledge such as local peaks/valleys or local auto-correlation of the gait signals. Consequently, we propose to improve a gait gallery by incorporating a phase registration technique which globally optimizes inter-period phase consistency in an energy minimization framework. However, the previous phase registration technique suffers from a phase distortion problem due to ambiguities in the combination of a periodic signal function and a phase evolution function. We present a linear phase evolution prior to constructing an undistorted gait signal for better matching performance. Experiments using real gait signals from 32 subjects show that the proposed methods outperform the latest methods in the field.

3D scanning of moving objects has many applications, for example, marker-less motion capture, analysis on fluid dynamics, object explosion and so on. One of the approach to acquire accurate shape is a projector-camera system, especially the methods that reconstructs a shape by using a single image with static pattern is suitable for capturing fast moving object. In this paper, we propose a method that uses a grid pattern consisting of sets of parallel lines. The pattern is spatially encoded by a periodic color pattern. While informations are sparse in the camera image, the proposed method extracts the dense (pixel-wise) phase informations from the sparse pattern. As the result, continuous regions in the camera images can be extracted by analyzing the phase. Since there remain one DOF for each region, we propose the linear solution to eliminate the DOF by using geometric informations of the devices, i.e. epipolar constraint. In addition, solution space is finite because projected pattern consists of parallel lines with same intervals, the linear equation can be efficiently solved by integer least square method. In this paper, the formulations for both single and multiple projectors are presented. We evaluated the accuracy of correspondences and showed the comparison with respect to the number of projectors by simulation. Finally, the dense 3D reconstruction of moving objects are presented in the experiments.

Recently it is relatively easy to produce digital point sampled 3D geometric models. In sight of the increasing capability of 3D scanning systems to produce models with millions of points, compression efficiency is of paramount importance. In this paper, we propose a novel competition-based predictive method for single-rate compression of 3D models represented as point cloud. In particular we aim at 3D scanning methods based on grid pattern. The proposed method takes advantage of the pattern characteristic made of vertical and horizontal lines, by assuming that the object surface is sampled in curve of points. We then designed and implemented a predictive coder driven by this curve-based point representation. Novel prediction techniques are specifically designed for a curve-based cloud of points, and been competing between them to achieve high quality 3D reconstruction. Experimental results demonstrate the effectiveness of the proposed method.

Dense 3D reconstruction of extremely fast moving objects could contribute to various applications such as body structure analysis and accident avoidance and so on. The actual cases for scanning we assume are, for example, acquiring sequential shape at the moment when an object explodes, or observing fast rotating turbine's blades. In this paper, we propose such a technique based on a one-shot scanning method that reconstructs 3D shape from a single image where dense and simple pattern are projected onto an object. To realize dense 3D reconstruction from a single image, there are several issues to be solved; e.g. instability derived from using multiple colors, and difficulty on detecting dense pattern because of influence of object color and texture compression. This paper describes the solutions of the issues by combining two methods, that is (1) an efficient line detection technique based on de Bruijn sequence and belief propagation, and (2) an extension of shape from intersections of lines method. As a result, a scanning system that can capture an object in fast motion has been actually developed by using a high-speed camera. In the experiments, the proposed method successfully captured the sequence of dense shapes of an exploding balloon, and a breaking ceramic dish at 300–1000 fps.

Although RANSAC is the most widely used robust estimator in computer vision, it has certain limitations making it ineffective in some situations, such as the motion estimation problem, in which uncertainty on the image features changes according to the capturing conditions. The greatest problem is that the threshold used by RANSAC to detect inliers cannot be changed adaptively; instead it is fixed by the user. An adaptive scale algorithm must therefore be applied in such cases. In this paper, we propose a new adaptive scale robust estimator that adaptively finds the best solution with the best scale to fit the inliers, without the need for predefined information. Our new adaptive scale estimator matches the residual probability density from an estimate and the standard Gaussian probability density function to find the best inlier scale. Our algorithm is evaluated in several motion estimation experiments under varying conditions and the results are compared with several of the latest adaptive-scale robust estimators.

3D acquisition techniques to measure dynamic scenes and deformable objects with little texture are extensively researched for applications like the motion capturing of human facial expression. To allow such measurement, several techniques using structured light have been proposed. These techniques can be largely categorized into two types. The first involves techniques to temporally encode positional information of a projector’s pixels using multiple projected patterns, and the second involves techniques to spatially encode positional information into areas or color spaces. Although the former allows dense reconstruction with a sufficient number of patterns, it has difficulty in scanning objects in rapid motion. The latter technique uses only a single pattern, so this problem can be resolved, however, it often uses complex patterns or color intensities, which are weak to noise, shape distortions, or textures. Thus, it remains an open problem to achieve dense and stable 3D acquisition in real cases. In this paper, we propose a technique to achieve dense shape reconstruction that requires only a single-frame image of a grid pattern. The proposed technique also has the advantage of being robust in terms of image processing.

Methods for scanning dynamic scenes are important in many applications and many systems using structured light have been proposed. Many of these systems use either multiple patterns projected rapidly or a single pattern. Although the former allows dense reconstruction with a sufficient number of patterns, it has difficulty in capturing objects in rapid motion. The latter technique uses only a single pattern and have no such difficulties, however, they often have stability problems and their result tend to have low resolution. In this paper, we develop a system to achieve dense and accurate 3D measurement from only a single image. The proposed system also has the advantage of being robust in terms of image processing.

In this paper, we bring out a new density-based clustering initialization algorithm which is invariant to the scale factor. Instead of using the scale factor while the cluster initialization, in this research, we determine the number and position of clusters according to the changes of cluster density with the division and agglomeration processes. During the division process, the initial cluster seeds are produced by a self-propagate method according to the density changes. The number of clusters is determined by agglomerating pair of RNN (reciprocal nearest neighbor) cluster seeds, when the density of newly merged cluster is increased. When no more cluster seeds can be merged any more, the remained number of cluster seeds is regarded as the real cluster number. Through various experiments, the effectiveness of the proposed algorithm has been proved.

This study describes a method of estimating the intrinsic parameters of a perspective camera. In previous calibration methods for perspective cameras, the intrinsic and extrinsic parameters are estimated simultaneously during calibration. Thus, the intrinsic parameters depend on the estimation of the extrinsic parameters, which is inconsistent with the fact that intrinsic parameters are independent of extrinsic ones. Moreover, in a situation where the extrinsic parameters are not used, only the intrinsic parameters need to be estimated. In this case, an intrinsic parameter, such as focal length, is not sufficiently robust to combat the image processing noise, that is absorbed by both parameter types, during calibration. We therefore propose a new method that allows the estimation of intrinsic parameters without estimating the extrinsic parameters. In order to calibrate the intrinsic parameters, the proposed method observes parallel light pairs that are projected on different points. This is accomplished by applying the constraint that the relative angle of two parallel rays is constant irrespective of where the rays are projected. This method focuses only on intrinsic parameters and the calibrations are sufficiently robust as demonstrated in this study. Moreover, our method can visualize the error of the calibrated result and the degeneracy of the input data.

We propose a new egomotion estimation algorithm for a compound omnidirectional camera. Image features are detected by a conventional feature detector and then quickly classified into near and far features by checking infinity on the omnidirectional image of the compound omnidirectional sensor. Egomotion estimation is performed in two steps: first, rotation is recovered using far features; then translation is estimated from near features using the estimated rotation. RANSAC is used for estimations of both rotation and translation. Experiments in various environments show that our approach is robust and provides good accuracy in real-time for large motions.

When we use range finders to observe the shape of an object, many occluded areas may occur. These become holes and gaps in the model and make it undesirable for various applications. We propose a novel method to fill holes and gaps to complete this incomplete model. As an intermediate representation, we use a Signed Distance Field (SDF), which stores Euclidean signed distances from a voxel to the nearest point of the mesh model. By using an SDF, we can obtain interpolating surfaces for holes and gaps. The proposed method generates an interpolating surface that becomes smoothly continuous with real surfaces by minimizing the area of the interpolating surface. Since the isosurface of an SDF can be identified as being a real or interpolating surface from the magnitude of signed distances, our method computes the area of an interpolating surface in the neighborhood of a voxel both before and after flipping the sign of the signed distance of the voxel. If the area is reduced by flipping the sign, our method changes the sign for the voxel. Therefore, we minimize the area of the interpolating surface by iterating this computation until convergence. Unlike methods based on Partial Differential Equations (PDE), our method does not require any boundary condition, and the initial state that we use is automatically obtained by computing the distance to the closest point of the real surface. Moreover, because our method can be applied to an SDF of adaptive resolution, our method efficiently interpolates large holes and gaps of high curvature. We tested the proposed method with both synthesized and real objects and evaluated the interpolating surfaces.

High dynamic range images (HDRIs) are needed for capturing scenes that include drastic lighting changes. This paper presents a method to improve the dynamic range of a camera by using a reflective liquid crystal. The system consists of a camera and a reflective liquid crystal placed in front of the camera. By controlling the attenuation rate of the liquid crystal, the scene radiance for each pixel is adaptively controlled. After the control, the original scene radiance is derived from the attenuation rate of the liquid crystal and the radiance obtained by the camera. A prototype system has been developed and tested for a scene that includes drastic lighting changes. The radiance of each pixel was independently controlled and the HDRIs were obtained by calculating the original scene radiance from these results.

This paper describes a method of mirror localization to calibrate a catadioptric imaging system. Even though the calibration of a catadioptric system includes the estimation of various parameters, in this paper we focus on the localization of the mirror. Since some previously proposed methods assume a single view point system, they have strong restrictions on the position and shape of the mirror. We propose a method that uses parallel lights to simplify the geometry of projection for estimating the position of the mirror, thereby not restricting the position or shape of the mirror. Further, we omit the translation process between the camera and calibration objects from the parameters to be estimated by observing some parallel lights from a different direction. We obtain the constraints on the projection and compute the error between the model of the mirror and the measurements. The position of the mirror is estimated by minimizing the error. We also test our method by simulation and real experiments, and finally we evaluate the accuracy of our method.

Camera ego-motion consists of translation and rotation, in which rotation can be described simply by distant features. We present a robust rotation estimation using distant features given by our compound omnidirectional sensor. Features are detected by a conventional feature detector, and then distant features are identified by checking the infinity on the omnidirectional image of the compound sensor. The rotation matrix is estimated between consecutive video frames using RANSAC with only distant features. Experiments with various environments show that our approach is robust and also gives reasonable accuracy in real-time.

You can create catadioptric omnidirectional stereovision using several mirrors with a single camera. These systems have interesting advantages, for instance in the case of mobile robot navigation and environment reconstruction. Our paper aims at estimating the” quality” of such stereovision system. What happens when the number of mirrors increases? Is it better to increase the base-line or to increase the number of mirrors? We propose some criteria and a methodology to compare different significant categories (seven): three already existing systems and four new designs that we propose. We also study and propose a global comparison between the best configurations.

This paper describes a new method to automatically calibrate lens distortion of wide-angle lenses. We project structured-light patterns using a flat display to generate a map between the display and the image coordinate systems. This approach has two advantages. First, it is easier to take correspondences of image and marker (display) coordinates around the edge of a camera image than using a usual marker, e.g. a checker board. Second, since we can easily construct a dense map, a simple linear interpolation is enough to create an undistorted image. Our method is not restricted by the distortion parameters because it directly generates the map. We have evaluated the accuracy of our method and the error becomes smaller than results by parameter fitting.

In this paper, we tackle the problem of geometric and photometric modeling of large intricately shaped objects. Typical target objects we consider are cultural heritage objects. When constructing models of such objects, we are faced with several important issues that have not been addressed in the past-issues that mainly arise due to the large amount of data that has to be handled. We propose two novel approaches to efficiently handle such large amounts of data: A highly adaptive algorithm for merging range images and an adaptive nearest-neighbor search to be used with the algorithm. We construct an integrated mesh model of the target object in adaptive resolution, taking into account the geometric and/or photometric attributes associated with the range images. We use surface curvature for the geometric attributes and (laser) reflectance values for the photometric attributes. This adaptive merging framework leads to a significant reduction in the necessary amount of computational resources. Furthermore, the resulting adaptive mesh models can be of great use for applications such as texture mapping, as we will briefly demonstrate. Additionally, we propose an additional test for the k-d tree nearest-neighbor search algorithm. Our approach successfully omits back-tracking, which is controlled adaptively depending on the distance to the nearest neighbor. Since the main consumption of computational cost lies in the nearest-neighbor search, the proposed algorithm leads to a significant speed-up of the whole merging process. In this paper, we present the theories and algorithms of our approaches with pseudo code and apply them to several real objects, including large-scale cultural assets.

This paper describes a novel system for detecting objects close to our sensor. For real time detection and portability, we have developed a small sensor with compound spherical mirrors. Since an object is projected onto each mirror, our method computes the range by a catadioptric stereo method. Our method creates a lookup table of corresponding points for an infinite range. If an object is close enough to the sensor, the projected points of the object are different from these corresponding points. Thus, our method can detect near objects by taking the differences in intensity of the corresponding points between the images in the mirrors. We show the experimental setup of our sensor and the result for detecting near objects.

We propose a method which refines the range measurement of range finders by computing correspondences of vertices of multiple range images acquired from various viewpoints. Our method assumes that a range image acquired by a laser rangefinder has anisotropic error distribution which is parallel to the ray direction. Thus, we find the corresponding points of range images along with the ray direction. We iteratively converge range images to minimize the distance of corresponding points. We demonstrate the effectiveness of our method by presenting the experimental results of artificial and real range data. Also, we show that our method refines a 3D shape more accurately as opposed to that achieved by using the Gaussian filter.

Integration of multiple range images is important to make use of 3D data acquired from stereo systems, laser range finders, etc. We propose a new range image integration method based on volumetric representation. Unlike other volume-based integration methods, we adaptively subdivide voxels depending on the curvature of the surface to be reconstructed, providing efficient representation of the underlying geometry and efficient use of computational resources. In our range image merging framework, additional attributes, e.g., color, laser reflectance power, etc., can be taken into account as well as 3D geometric information. This ability allows us to generate 3D models preserving sharp edges around texture boundaries, thereby providing a good basis for efficient rendering and texture mapping. The overall framework is designed to be robust against noise, taking consensus carefully in both geometry and color, which could be suitable for 3D model reconstruction from noisy stereo images. In this paper, we describe the system, and present several results of applying our framework to real data. We also present some other future applications based on our framework.

This paper describes a volumetric view-merging algorithm that generates a consensus surface of an object from its range images. Our original method merges a set of range images into a volumetric implicit-surface representation, which is converted to a surface mesh by using a variant of the marching-cubes algorithm. We propose a method that increases the computation and memory efficiency for computing signed distances and the method of parallel computing on a PC cluster Since our method permits a reduction in the data amount allocated in memory, the closest point is searched efficiently; this allows us to increase the number of parallel traversals and to reduce the computation time. In this paper, we describe the following two algorithms which are complementary in terms of the efficiency of CPU and memory usage: distributed allocation of range data and parallel traversal of partial octrees. By adjusting them according to the system specifications, we can build the model efficiently by a PC cluster We have implemented this system and evaluated its performance.

This paper describes a vision system which recognizes 3D objects in real-time by modeling the shapes of objects and matching the generated models. We develop the following methods for practically solving important problems of integration such as the estimation of sensor accuracy as well as real-time processing: 1) we reduce the computation of signed-distance, which is necessary to apply the marching cubes algorithm, and select the optimal resolution of models to be generated using an octree, thereby enabling us to generate hierarchical mesh models in real-time; 2) we apply spin-image matching by selecting the resolution of generated models and the coarse to-fine algorithm, consequently, we are able to efficiently match multiple objects of different sizes.