Derivative, spectral, and other operators implemented in filter kernels and applied with convolution have been well studied in image recognition problems. We explore in this paper a novel system that applies such filters in the actual 3D scene. The system projects images of filters into the scene as controlled illumination. The projected filters are subjected to object surface reflectance (BRDF) and scene composition; and the filter response is captured by camera pixels. Our approach utilizes a pixel-based convolution formulation, exploiting the intrinsic integration performed by individual pixel sensors and diffraction-limited optics. For example, we project a grid of Gaussian derivative filters as images and simply retrieve a filter response as a pixel value. This is a local (pixel-only) computation that is not limited by camera resolution. Our system can serve as a front-end to existing techniques for surface or geometry estimation from texture analysis. A projector and camera pair is the basic equipment requirement.

We introduce a new paradigm for real-time conversion of a real world event into a rich multimedia database by processing data from multiple sensors observing the event. A real-time analysis of the sensor data, tightly coupled with domain knowledge, results in instant indexing of multimedia data at capture time. This yields semantic information to answer complex queries about the content and the ability to extract portions of data that correspond to complex actions performed in the real world. The power of such an instantly indexed multimedia database system, in content-based retrieval of multimedia data or in semantic analysis and visualization of the data, far exceeds that of systems which index multimedia data only after it is produced. We present LucentVision, an instantly indexed multimedia database system developed for the sport of tennis. This system analyzes video from multiple cameras in real time and captures the activity of the players and the ball in the form of motion trajectories. The system stores these trajectories in a database along with video, 3D models of the environment, scores, and other domain-specific information. LucentVision has been used to enhance live television and Internet broadcasts with game analyses and virtual replays in more than 250 international tennis matches.

Remote experience of sporting events has thus far been limited mostly to watching video and the scores and statistics associated with the sport. However, a fast-developing trend is the use of visualization techniques to give new insights into performance, style, and strategy of the players. Automated techniques can extract accurate information from video about player performance that not even the most skilled observer is able to discern. When presented as static images or as a three-dimensional virtual replay, this information makes viewing a game an entirely new and exciting experience. This paper presents one such sports visualization system called LucentVision, which has been developed for the sport of tennis. LucentVision uses real-time video analysis to obtain motion trajectories of players and the ball, and offers a rich set of visualization options based on this trajectory data. The system has been used extensively in the broadcast of international tennis tournaments, both on television and the Internet.

Presents a real-time computer vision system that tracks the motion of a tennis ball in 3D using multiple cameras. Ball tracking enables virtual replays, new game statistics, and other visualizations which result in very new ways of experiencing and analyzing tennis matches. The system has been used in international television broadcasts and webcasts of more than 15 matches. Six cameras around a stadium, divided into four pairs, are currently used to track the ball on serves which sometimes exceed speeds of 225 kmph. A multi-threaded approach is taken to tracking where each thread tracks the ball in a pair of cameras based on motion, intensity and shape, performs stereo matching to obtain the 3D trajectory, detects when a ball goes out of view of its camera pair, and initializes and triggers a subsequent thread. This efficient approach is scalable to many more cameras tracking multiple objects. The ready acceptance of the system indicates the growing potential for multi-camera based real-time tracking in broadcast applications.

This paper develops real time tracking technology for sports broadcast applications. The specific sport chosen here is the game of tennis. The outputs of the tennis tracking system are spatio-temporal trajectories of motion of the players and the ball which can in turn provide a number of statistics about the game. For instance, the distance travelled by a player, the speed and the acceleration at any instant, as well as court coverage patterns can be obtained from the trajectories. The statistics so obtained can be visualized in compelling ways to enhance the appreciation of the athleticism and strategy involved in the sport. We present techniques for tracking the players and the ball in video obtained from stationary cameras. The problem is challenging as the tracking needs to be performed outdoors, players are fast-moving non-rigid objects, and the ball is a small object that can move at speeds in the range of 150 miles an hour. Player trajectories are obtained by dynamically clustering tracks of local features. Ball segmentation and tracking is based on shape and color features of the ball. Real time tracking results are presented on video recorded live by the authors in an international tennis tournament.