Neural radiance fields (NeRFs) have been proposed as methods of novel view synthesis and have been used to address various problems because of its versatility. NeRF can represent colors and densities in 3D space using neural rendering assuming a straight light path. However, a medium with a different refractive index in the scene, such as a transparent medium, causes light refraction and breaks the assumption of the straight path of light. Therefore, the NeRFs cannot be learned consistently across multi-view images. To solve this problem, this study proposes a method to learn consistent radiance fields across multiple viewpoints by introducing the light refraction effect as an offset from the straight line originating from the camera center. The experimental results quantitatively and qualitatively verified that our method can interpolate viewpoints better than the conventional NeRF method when considering the refraction of transparent objects.

This work explores the human visual exploratory activity (VEA) in a microgravity environment compared to one-G. Parabolic flights are the only way to experience microgravity without astronaut training, and the duration of each microgravity segment is less than 20 seconds. Under such special conditions, the test subject visually searches a virtual representation of the International Space Station located in his Field of Regard (FOR). The task was repeated in two different postural positions. Interestingly, the test subject reported a significant reduction of microgravity-related motion sickness while experiencing the VR simulation, in comparison to his previous parabolic flights without VR.

In recent years, research efforts have enabled using the internet in disaster areas, and information technology (IT) is expected to improve our comprehension and evaluation of disasters. In disaster areas, crowdsourcing can be employed to secure human resources and controlled-task distribution. In crowdsourcing, many workers with publicly defined tasks (also called microtasks) are used. A simplified and more efficient microtask framework is required for disaster areas due to the lack of human resources and facilities. In this paper, we describe a method that incorporates information collection from a disaster area into microtasks with machine processing support.

This paper proposes a calibration method for a large-scale floor projection system at a school gym. In our system, the projectors are installed on the ceiling of a school gym, and the contents are projected onto the floor. The projection results suffer from both projective distortion and lens distortion. It is hard to use chessboard-based calibration methods or self-calibration methods in our system due to the restrictive deploy environment and practical limitations. Our method is basing on the projector-camera system and "straight lines have to be straight". The experiments show that our method fulfills the on-site requirements of the floor projection system at school gym for learning aid usages.

We propose a calibration method for deploying a large-scale projection-based floor display system. In our system, multiple projectors are installed on the ceiling of a large indoor space like a gymnasium to achieve a large projection area on the floor. The projection results suffer from both perspective distortion and lens distortion. In this paper, we use projector-camera systems, in which a camera is mounted on each projector, with the “straight lines have to be straight” methodology, to calibrate our projection system. Different from conventional approaches, our method does not use any calibration board and makes no requirement on the overlapping among the projections and the cameras' fields of view.

Thanks to the rapid progress of the Internet and mobile devices, information related to disaster areas can be collected through the Internet. To grasp the degree of damage in a disaster situation, the use of crowdsourcing for coordinating the individual efforts (micro tasks) of an enormous number of users (workers) on the Internet has been drawing attention as a means of quickly solving problems. However, the information gathered from the Internet is huge and diverse, so it is difficult to formulate as a crowdsourcing task. This paper proposes a conversion platform for the images of a disaster site photographed by various users as information about the site, integrating the images into a single map using 3D image processing, and providing the map to crowdsourcing as a micro task.

This paper proposes an interactive display system that displays both of the exterior and interior construction of a building diorama by using a projection-based Mixed-Reality (MR) technique, which is useful for understanding the complex construction and the spatial relationships between outside and inside. The users can hold and move the diorama model using their hands/body motion, so that they can observe the model from their favorite viewpoint. Our system obtains both of the user's information (the viewpoint and the gesture) and the diorama model's information (the pose) in 3D space by using two RGB-D cameras. The CG image corresponding to the user's viewpoint, gesture and the pose of the diorama is rendered by Dual Rendering algorithm in real time. As the result, the generated CG image is projected onto the diorama to realize MR display. We confirm the effectiveness of our proposed method by developing a pilot system.

We present a new matching algorithm considering a priori (the prior probability) based on Bayes' theorem. Performance of point cloud registration between target and source clouds is effectively improved by introducing maximum a posteriori (MAP) estimation. The standard Iterative Closest Point (ICP) algorithm for the registration sometimes falls into misalignment due to measurement errors, narrow sensing field of view, or the movement of objects during measurement. Our approach resolves such problems by considering both the likelihood of the measurement and the prior probability of the initial guess for registration in the objective function. We have implemented a new 6DOF Iterative Closest Point matching using MAP estimation, and evaluated the method in real environments comparing with conventional registration methods. The experimental results have shown that our proposed method has wide convergence region and matches point clouds accurately preventing the misalignment problem.

We propose a new computer vision approach to locate a walking pedestrian by a camera image of first-person vision in practical situation. We assume reference points have been registered with other first-person vision images. We utilize SURF and define seven matching criteria that derive from the property of first-person vision so that it rejects false matching. We have implemented a preliminary system that can respond to a query within 1/2 seconds for a path of approximately 1 km long around Tokyo downtown area where pedestrians and vehicles are always in images.

Our research aims to generate a player's view video stream by using a 3D free-viewpoint video technique. Since player trajectories are necessary to generate the video, we propose a real-time player trajectory estimation method by utilizing the shadow regions from soccer scenes. This paper describes our trial to realize real-time processing. We divide the process into capture and server computers. In addition, we reduced the processing cost with pipeline parallelization and optimization. We apply our proposed method to an actual soccer match held in a stadium and show its effectiveness.

This paper proposes a method to estimate the trajectories of soccer players by using two cameras set in a large-scale outdoor space such as a soccer stadium, which is normally not advantageous for image processing. We estimate a player¿s position as the intersection of the primary axes of two body regions, corresponding to the two cameras, and a shadow region on the surface of the field. By projecting the foreground silhouette regions extracted from both cameras¿ images onto the soccer field, each player¿s body region and its shadow region are identified. Furthermore, our method utilizes color information of the players' uniforms to improve the accuracy of object tracking. We have applied our proposed method to a real soccer game held in a soccer stadium and demonstrated its effectiveness.

We propose a new on-line modeling and rendering method for visualizing players in sports. This method targets the intricate geometry of players in sports such as wrestling. Our system can maintain modeling and rendering quality on producing free-viewpoint 3D video of the players by utilizing the virtual viewpoint information of the viewer. Viewers can move a virtual camera freely over the scene while the system manages the processing cost by changing the size of the voxels and microfacets used for spatial modeling of the players. The system first estimates the rough 3D shape of the players in voxel format, and then assigns a microfacet to each voxel. The texture of the microfacet is obtained from the video cameras surrounding the players. Our preliminary system was evaluated on the wrestling data taken in the Yoyogi National Gymnasium, Tokyo, Japan. We also conducted subjective evaluation of the free-viewpoint video thus produced and found a good balance of rendering quality and high frame-rate performance.

This paper proposes a novel remote face-to-face mixed reality (MR) system that enables two people in distant places to share MR space. Challenging issues to realize such an MR system include capturing, sending, and rendering each user's appearance in real time. We developed a method to represent user's upper body and hands on the table as a single deformed-billboard. An MR Othello game is implemented as a test bed of the remote face-to-face MR system. Users can play the tabletop game as if their opponent were sitting across from the table, despite being physically separated. By detecting and sending the status of each real game board to the other site, both users feel that they are sharing tabletop objects.

We present an active lighting method that can control brightness of multiple objects in a scene to any values on video images. Our approach does not modify pixel values such as in retouching but changes intensities of multiple lights in the scene. Our algorithm needs little computation to calculate appropriate intensities of the lights because we formulate the brightness of the objects so that the essential parameters in our formulation can be measured in advance. We have implemented a prototype system that can control brightness of faces of two people in a room to intended values with eight controllable lights as they are walking.

We present a new video browsing method for multiple videos that are taken at large-scale space for live 3D events such as soccer games. By our method, multiple viewers over a computer network can browse a live 3D event from any viewpoint and each viewer can move his/her viewpoint freely. Our algorithm consists of five steps. Our system first captures videos from multiple cameras, then extracts texture segments from the videos, selects appropriate segments according to a viewpoint which is given by user dynamically, transmits them to users, and lays out the segments in virtual space so that each viewer can see the segments in a virtual environment as if the viewers were in the event. Our 3D video display system requires 10 Mbps at most to browse a soccer game. We conducted experiments at two real soccer stadiums and succeeded in realizing live realistic visualization with free viewpoint at about 26 fps.

This paper presents a new outdoor mixed-reality system designed for people who carry a camera-attached small handy device in an outdoor scene where a number of surveillance cameras are embedded. We propose a new functionality in outdoor mixed reality that the handy device can display live status of invisible areas hidden by some structures such as buildings, walls, etc. The function is implemented on a camera-attached, small handy subnotebook PC (HPC). The videos of the invisible areas are taken by surveillance cameras and they are precisely overlapped on the video of HPC camera, hence a user can notice objects in the invisible areas and see directly what the objects do. We utilize surveillance cameras for two purposes. (1) They obtain videos of invisible areas. The videos are trimmed and warped so as to impose them into the video of the HPC camera. (2) They are also used for updating textures of calibration markers in order to handle possible texture changes in real outdoor world. We have implemented a preliminary system with four surveillance cameras and proved that our system can visualize invisible areas in real time.

An advanced multimedia lecture recording system is presented in this paper. The purpose of our system is to capture multimodal information that can be received only when lectures are being held in a classroom where a teacher and students share the same time and space. The system captures not only hand-writings and slide switching intervals but also audio and video of the people with their spatial location information. These recorded media will be served to users in distance learning process. We designed the system so that it does not interfere with classes. It can archive lectures while they hold classes on regular basis and generate a multimedia archive automatically. Teachers are neither asked to remain at a certain place nor wired with devices they would have to put on. We have implemented our approach and our lecture archive system currently works for six classes per week, since October, 2002.

In this paper, we propose a method for detecting the location of the speaker that is a target of automatic video filming in distance learning and lecture archive. It is required that a face of a speaking student is filmed in a lecture video. For this purpose, it is necessary to detect the location of a speaker. An acoustic sensor such as a microphone array is used widely to detect the location of a sound source. However, it is difficult to detect the location of a sound source precisely using only microphone array because of sound noise in a large space such as a lecture room. In this paper, we propose a method for detecting more precise location of a speaker in the lecture room using not only the microphone array but also visual sensors. The result shows that the precision ratio of detecting the location of a speaker was improved about 20% by our sensor-fusion method.

Recognizing structure of the human body is important for modeling human motion. The human body is usually represented as an articulate model, which consists of the rigid parts and the joint points between them. The structure of the human body is specified by the joint points. We propose a method for estimating the location of joint points from successive volume data. Our joint point estimation method consists of three steps. In the first step, rigid parts are extracted from two successive volume data under the constraint of the rigid transformation. In the second step, the joint points are estimated based on the rigid parts. As the last step, false joint points are eliminated by using more successive data. Applying the method to the simulated data, the locations of joint points of the human are correctly estimated.