The ultrasonic sensors are widely used for vehicles to extract obstacles in the scene. They can obtain the distance to the object directly at a low cost even in harsh environments. However, since the information from a single ultrasonic sensor is very limited, it has not been used for recovering the detailed 3D structure and semantic labels of the scene, unlike in-vehicle cameras or LiDARs. Therefore, we in this paper propose a method for recovering the dense 3D structure and semantic labels of the scene from a moving ultrasonic sensor mounted on a vehicle. Our method uses the raw profiles of the ultrasonic sensor signals and learns the relationship between the raw ultrasonic signals and the 3D scene using multi-task learning. As a result, our method can recover the dense 3D structure and semantic labels of the scene similar to what we would recover with cameras and LiDARs just from a single moving ultrasonic sensor. The efficiency of the proposed method is tested using real sensor data as well as synthetic sensor data.

In this paper, we propose a method of visualizing 3D motion with zero latency. This method achieves motion visualization by projecting special high-frequency light patterns on moving objects without using any feedback mechanisms. For this objective, we focus on the time integration of light rays in the sensing system of observers. It is known that the visual system of human observers integrates light rays in a certain period. Similarly, the image sensor in a camera integrates light rays during the exposure time. Thus, our method embeds multiple images into a time-varying light field, such that the observer of the time-varying light field observes completely different images according to the dynamic motion of the scene. Based on this concept, we propose a method of generating special high-frequency patterns of projector lights. After projection onto target objects with projectors, the image observed on the target changes automatically depending on the motion of the objects and without any scene sensing and data analysis. In other words, we achieve motion visualization without the time delay incurred during sensing and computing.

In this paper, we propose a novel method for measuring the temporal modulation of lights by using off-the-shelf cameras. In particular, we show that the invisible flicker patterns of various lights such as fluorescent lights can be measured by a simple combination of an off-the-shelf camera and any moving object with specular reflection. Unlike the existing methods, we do not need high speed cameras nor specially designed coded exposure cameras. Based on the extracted flicker patterns of environment lights, we also propose an efficient method for deblurring motion blurs in images. The proposed method enables us to deblur images with better frequency characteristics, which are induced by the flicker patterns of environment lights. The real image experiments show the efficiency of the proposed method.

In this paper, we propose a new camera model for reconstructing 3D objects under light ray distortion caused by refractive medias. The proposed method can reconstruct 3D scene, even if light rays projected into the cameras are refracted by the refractive media, such as glasses and raindrops. For this objective, we represent light ray projection of multiple cameras by using a pair of planes shared by the multiple cameras in the scene. By using this model, intrinsic and extrinsic camera parameters as well as the refractive properties of the refractive media can be represented efficiently. By using the newly defined camera model, we propose a method for recovering 3D points and camera parameters with refractive properties simultaneously. The experimental results show the efficiency of the proposed camera model and reconstruction method.

It is known that time-to-contact toward objects can be estimated just from changes in the object size in camera images, and we do not need any additional information, such as camera parameters and motions. However, the existing methods for measuring the time-to-contact are based on geometric image features, such as corners and edge lines, and thus they cannot be used when there are no geometric features in images. In this paper, we propose a new method for computing the time-to-contact from photometric information in images. When a light source moves in the 3D scene, an observed intensity changes according to the motion of the light source. In this paper, we analyze the change in photometric information in images, and show that the time-to-contact can be estimated just from the changes in intensity in images. Our method does not need any additional information, such as radiance of light source, reflectance of object and orientation of object surface. The proposed method can be used in various applications, such as vehicle driver assistance.

Stereo reconstruction generally requires image correspondence such as point and line correspondences in multiple images. Cameras need to be synchronized to obtain corresponding points of time-varying shapes. However, the image information obtained from synchronized multiple cameras is redundant, and has limitations with resolution. In this paper, we show that by using a set of unsynchronized cameras instead of synchronized one, we can obtain much more information on 3D motions, and can reconstruct higher frequency 3D motions than that with the standard synchronized cameras. As a result, we can attain super resolution 3D reconstruction from unsynchronized cameras. In the standard reconstruction with multiple cameras, the cameras are synchronized, and observe the same set of M sequential points in 3D space as shown in Fig. 1 (a). Thus, the maximum frequency fS of 3D motion, which can be recovered from camera images, is fS < 1 2 M. If we observe the same 3D motion by using K unsynchronized cameras, we observe K×M different points in the 3D space as shown in Fig. 1 (b). As a result, all the 2KM observations from K cameras are independent of one another, unlike those from the synchronized cameras. Thus, we have a possibility of reconstructing 3D points up to 2 3 KM. Therefore, the maximum frequency fU of recoverable 3D motion is fU < 1 3 KM. Thus, we find that the unsynchronized cameras have a possibility of reconstructing 3 K times higher frequency 3D motion than the synchronized cameras as follows:

In this paper, we propose a method for detecting abnormal driving of vehicles, such as meandering, transverse motion and acceleration/deceleration. In particular, we extract abnormal vehicle motions in the sense of group behavior by using multilinear relationship in space-time images. The multilinear relationship in space-time images holds when multiple cameras move with translational motions in different direction with different speed. Therefore abnormal drivings, which are not translational motion with uniform velocity, do not meet the requirement, and the multilinear relationship in space-time images does not hold. We focus on this property, and define the degree of abnormality, which is used for detecting abnormal drivings. The efficiency of the proposed method is shown by real image experiments.

In this paper, we propose a method for calibrating relative position and orientation of multiple unsynchronized cameras from general moving points. If the sampling times of multiple cameras are different from each other, there is no corresponding point in the images of moving points and we cannot calibrate these cameras. In this paper we analyze geometric relationships of multiple cameras in the frequency space, and show that they enable us to calibrate multiple unsynchronized cameras accurately even if exact corresponding points do not exist in images. The proposed method does not require any interpolation of moving points in images.

In this paper, we analyze the multiple view geometry for a camera and mirrors, and propose a method for computing the geometry of the camera and mirrors accurately from fewer corresponding points than the existing methods. The geometry between a camera and mirrors can be described as the multiple view geometry for a real camera and virtual cameras. We show that very strong constraints on geometries can be obtained in addition to the ordinary multilinear constraints. By using these constraints, we can estimate multiple view geometry more accurately from fewer corresponding points than usual. The experimental results show the efficiency of the proposed method.

In this paper, we derive a novel multilinear relationship for close light sources and cameras. In this multilinear relationship, image intensities and image point coordinates can be handled in a single framework. We first derive a linear representation of image intensity taken under a general close light source. We next analyze multiple view geometry among close light sources and cameras, and derive novel multilinear constraints among image intensity and image coordinates. In particular, we study the detail of the multilinear relationship among 7 lights and a camera. Finally, we show some experimental results, and show that the new multilinear relationship can be used for linearly generating images illuminated by arbitrary close light sources.

In this paper, we propose a method for projecting images onto non-planar screens by using projector-camera systems eliminating distortion in projected images. In this system, point-to-point correspondences in a projector image and a camera image should be extracted. For finding correspondences, the epipolar geometry between a projector and a camera is used. By using dynamic programming method on epipolar lines, correspondences between projector image and camera image are obtained. Furthermore, in order to achieve faster and more robust matching, the non-planar screen is approximately represented by a B-spline surface. The small number of parameters for the B-spline surface are estimated from corresponding pixels on epipolar lines rapidly. Experimental results show the proposed method works well for projecting images onto non-planar screens.

Recently, Shadow Graph has been proposed for recovering 3D shapes from shadows projected on curved surfaces. Unfortunately, this method requires a large computational cost. In this paper, we introduce 1D Shadow Graph which can be used for recovering 3D shapes with much smaller computational costs. We also extend our method, so that we can estimate both 3D shapes and light source positions simultaneously under a condition where 3D shapes and light sources are unknown.

Eigenspaces are often utilized in computer vision for various applications. However, careful manipulations are required for sensitive applications in photometric analyses and human interface applications. While several formulations have already been utilized in each application, a unified framework of eigenspace manipulations should be created for further progress of computer vision. This paper provides a concept of homogeneous eigenspaces for this purpose. After brief discussion of eigenspace construction, this paper focuses on how an optimal projection of an input image should be done. Three approaches are presented for the projection and normalization in order to minimize the effects of noise. Both the theoretical and experimental comparisons show that the normalization after projection can suppress the effects of noise more effectively than the other methods. The normalization-after-projection formulation can be reformulated by the homogeneous eigenspace. In the framework of homogeneous eigenspace, a projection onto the normalized eigenspace is accomplished without using an explicit normalization. Furthermore, an optimal partial projection to the normalized eigenspace is also reduced to linear mapping in this framework. Two applications are also discussed for illumination analysis and for photometric classification. These problems are efficiently solved by partial projections onto homogeneous eigenspaces.

In this paper, we propose a new calibration method for projector-camera systems. The projector-camera systems have been studied extensively as one of new information presenting systems. For calibrating the systems, we in this paper use virtual mutual projections between a camera and a projector. We first show that by using a shadow of the camera, we can generate mutual projection between a camera and a projector virtually. By using the mutual projection, the projector camera system can be calibrated more stably than the existing calibration methods. Some experimental results show that the mutual projections work quite well for calibrating projector-camera systems.