This paper presents a novel method for virtual view synthesis that allows viewers to virtually fly through real soccer scenes, which are captured by multiple cameras in a stadium. The proposed method generates images of arbitrary viewpoints by view interpolation of real camera images near the chosen viewpoints. In this method, cameras do not need to be strongly calibrated since projective geometry between cameras is employed for the interpolation. For avoiding the complex and unreliable process of 3-D recovery, object scenes are segmented into several regions according to the geometric property of the scene. Dense correspondence between real views, which is necessary for intermediate view generation, is automatically obtained by applying projective geometry to each region. By superimposing intermediate images for all regions, virtual views for the entire soccer scene are generated. The efforts for camera calibration are reduced and correspondence matching requires no manual operation; hence, the proposed method can be easily applied to dynamic events in a large space. An application for fly-through observations of soccer match replays is introduced along with the algorithm of view synthesis and experimental results. This is a new approach for providing arbitrary views of an entire dynamic event.

This paper introduces two kinds of free viewpoint observation systems for sporting events captured with uncalibrated multiple cameras in a stadium. In the first system (viewpoint on demand system), a user can watch the realistic sporting scenes with the original stadium. In the second system (mixed reality presentation system), a user can virtually watch the scenes overlaid on a desktop stadium model via a video see-through head mounted display (HMD). In the both systems, the user can observe sporting events from his/her favorite viewpoints, where the virtual view images are synthesized and presented by performing view interpolation. As outdoor scenes often have variations in lighting, we develop the systems to handle the changes of lighting condition. If the captured scene contains shadows, we synthesize the virtual view image of the shadows of the background and the foreground independently from real camera images using projective geometry between cameras. The shadows of the foreground objects are then overlaid on the synthesized background of the original stadium or the stadium model in front of the user respectively. The results indicate that the appearance of shadows can produce a realistic mixed reality presentation.

We introduce methods for sports scene analysis and visualization from multiple videos captured with multiple cameras. As the scene analysis, we present a method for tracking multiple soccer players. Tracking is done by integrating the tracking data from all cameras, using the geometrical relationship between cameras called homography. Integrating information from all cameras enables stable tracking on the scene, where the tracking by a single camera often fails in the case of occlusion. We also present a method for free-viewpoint visualization of a soccer game as a technique for visualizing the soccer scene. The free-viewpoint image is synthesized by view interpolation between actual cameras near the virtual viewpoint at each frame. Such free-viewpoint video can also be presented in a system of augmented reality (AR) for immersive visualization of soccer matches

This paper presents a novel observation system for immersive soccer match taken by multiple video cameras at a real stadium. The user sees the soccer field model in front of his/her eyes from the viewpoint through head-mounted display, while the images of players and a soccer ball are also rendered onto the display. For geometric registration between the soccer field model in the real world and the dynamic soccer scene in the rendered images, the viewpoint position of the user is computed by using only natural feature lines in the HMD camera image. Since it is difficult to strongly calibrate the HMD camera and the multiple cameras that capture the real soccer scene, we employ projective geometry for the registration and the rendering. For demonstrating the efficacy of the proposed system, video images of soccer matches taken at real stadium are rendered onto the HMD camera images of a tabletop stadium model. This is a completely new challenge to apply augmented reality to a dynamic event in a large-space.