In this study, we discuss the imaging of depth and velocity using heterodyne-mode time-of-flight (ToF) cameras. In particular, Doppler ToF (D-ToF) imaging utilizes heterodyne modulation to measure the velocity from the Doppler frequency shift, which uniquely facilitates the instantaneous radial velocity estimation. However, theoretical discussion on D-ToF is limited to orthogonal frequency and sinusoidal waveform modulation. This study extends the formulation of the D-ToF imaging, and proposes an arbitrary-frequency, arbitrary-waveform framework considering a phase-compensated, symmetrical two-dimensional correlation map. With the proposed framework, the optimal heterodyne frequency for frequency decoding is found. A differential frequency sampling and decoding method is then proposed, which computes the frequency and phase from as few as four simultaneously captured images. With an experiment platform we built, it is confirmed that the minimum velocity sensing error is half that of the orthogonal frequency method, and the sensible phase range is approximately 2.5 times larger. The conclusions in this study allow the ToF velocity imaging to be applied at the optimal sample frequencies for a wide range of ToF sensors. This pushes one step further to the practical use of ToF velocity imaging.

High-speed 3D shape sensing is an essential technology for three-dimensional recognition and manipulation in dynamic scenes. However, conventional high-speed sensing methods mainly focus on the image capturing speed and the actual processing time to obtain a point cloud is not optimized in the measurement scheme and sensing pattern configuration. On the other hand, measurement latency is critical to respond in real-time and physically handle the dynamic scenes. This paper introduces a physically stereo-rectified projector-camera system for high-speed and low-latency 3D sensing with fast sequential memory access in the decoding process. Moreover, we configure a structured light pattern named “Parallel-bus pattern” with a De Bruijn torus and clock lines to maximize information density and robustly decode the pattern as data transfer in a parallel bus interface. We measured dynamically moving and deforming objects with the proposed system and evaluated the measurement performance. As a result, the developed 3D sensing system with the parallel-bus pattern achieved 27K-points measurement at higher than 1000 fps with 0.336 ms latency and 0.838 mm accuracy on average.

We propose a projector-based method that provides an illusion of geometry change, similar to that caused by a change in physical properties such as elasticity, in response to inertia caused by physical motion. The proposed method is named “ElaMorph projection.” Although several projection mapping methods capable of deforming targets have been previously proposed, these methods require the preparation of animations in advance. Moreover, these methods are unable to deform the shape in real-time according to actual movements of the object. To address these issues, we perform real-time geometry deformation and rendering based on the spatial motion of the object. To render the projection image, we extend the conventional method of deformation for 2D pictures or static 3D objects to include dynamic 3D objects. This study involves projection onto a dynamic 3D object; however, the projection quality decreases if a part of the rendered image extends beyond the projection target. To address this issue, the proposed algorithm ensures that the vertices after deformation always remain within the projection target. In addition, we develop a robust algorithm to generate projection images under dynamic illuminative conditions, through real-time estimation of the environmental lighting required for rendering. Moreover, using an elasticity map that can be easily constructed using a UV map, our method enables users to specify the vertices to be deformed, using an elasticity map. We present projections under several different sets of elasticity maps, environmental lighting, and elasticities. Finally, we evaluate the latency and throughput of our system.

Based on the concept of spatial augmented reality (SAR), a number of applications have been developed by extending the type of target objects to include dynamically moving objects; however, the systems themselves still remain fixed. In this paper, we propose a new paradigm for dynamic SAR implemented with a mobile device and a practical architecture using a 3D-stacked vision chip and a high-speed optical gaze controller. We evaluated the prototype system that we developed, called “Portable Lumipen”, and confirmed that image capturing at 1000 fps, rapid processing and a 3 ms response time for projection were achieved on a mobile device. We also propose potential applications using this portable dynamic SAR system and demonstrated miniaturization of a high-speed visual feedback system.

The visual appearance of an object can be disguised by projecting virtual shading as if overwriting the material. However, conventional projection-mapping methods depend on markers on a target or a model of the target shape, which limits the types of targets and the visual quality. In this paper, we focus on the fact that the shading of a virtual material in a virtual scene is mainly characterized by surface normals of the target, and we attempt to realize markerless and modelless projection mapping for material representation. In order to deal with various targets, including static, dynamic, rigid, soft, and fluid objects, without any interference with visible light, we measure surface normals in the infrared region in real time and project material shading with a novel high-speed texturing algorithm in screen space. Our system achieved 500-fps high-speed projection mapping of a uniform material and a tileable-textured material with millisecond-order latency, and it realized dynamic and flexible material representation for unknown objects. We also demonstrated advanced applications and showed the expressive shading performance of our technique.

Reality is the most realistic representation. We introduce a material display called ZoeMatrope that can reproduce a variety of materials with high resolution, dynamic range and light field reproducibility by using compositing and animation principles used in a zoetrope and a thaumatrope. With ZoeMatrope, the quality of the material is equivalent to that of real objects and the range of expressible materials is diversified by overlaying a set of base materials in a linear combination. ZoeMatrope is also able to express spatially-varying materials, and even augmented materials such as materials with an alpha channel. In this paper, we propose a method for selecting the optimal material set and determining the weights of the linear combination to reproduce a wide range of target materials properly. We also demonstrate the effectiveness of this approach with the developed system and show the results for various materials.

We developed an optical high-speed image rotation controller and realized a high-speed roll camera that is able to cancel the rotational motion blur of a rotating target. This system is composed of a hollow motor, a Dove prism, and a high-speed camera and controls optical image rotation according to the target rotation by using high-speed image processing. This so-called optical lever formed of the Dove prism worked effectively also for a high-speed rotating target, and our prototype system shows that rotational motion blur was suppressed to 0.125 [°] at 1420 [r/min].

We propose a novel three-dimensional motion sensing method using lasers. Recently, object motion information is being used in various applications, and the types of targets that can be sensed continue to diversify. Nevertheless, conventional motion sensing systems have low universality because they require some devices to be mounted on the target, such as accelerometers and gyro sensors, or because they are based on cameras, which limits the types of targets that can be detected. Our method solves this problem and enables noncontact, high-speed, deterministic measurement of the velocity of a moving target without any prior knowledge about the target shape and texture, and can be applied to any unconstrained, unspecified target. These distinctive features are achieved by using a system consisting of a laser range finder, a laser Doppler velocimeter, and a beam controller, in addition to a robust 3D motion calculation method. The motion of the target is recovered from fragmentary physical information, such as the distance and speed of the target at the laser irradiation points. From the acquired laser information, our method can provide a numerically stable solution based on the generalized weighted Tikhonov regularization. Using this technique and a prototype system that we developed, we also demonstrated a number of applications, including motion capture, video game control, and 3D shape integration with everyday objects.

In this paper, we propose an algebraic solution for rapid SVBRDF measurement. The algebraic approach requires only a few reflectance samples to obtain the parameters described by the physically based Cook - Torrance model. This solution, however, also involves constraints concerning light and the normal direction in the acquisition process. To meet these constraints, we developed a system that changes the illumination according to the target 3D shape at high speed. As a result, the proposed method provides BRDF parameters at each texel without optimization and over-sampling. We demonstrated rapid measurement with real objects that do not have uniform reflectance and confirmed the validity of this approach by comparison with conventional methods.