The quality achieved by simply scaling a sports video to the limited display resolution of a mobile device is often insufficient. As a consequence, small details like the ball or lines on the playing field become unrecognizable. In this paper, we present a novel approach to analyzing court-based ball sports videos. We have developed new techniques to distinguish actual playing frames, to detect players, and to track the ball. This information is used for advanced video retargeting, which emphasizes essential content in the adapted videos. We evaluate the precision and recall achieved in the analysis and measure the computational time taken. In addition, we compare our new approach to other video retargeting techniques based on scaling, cropping, and seam carving.

Person detection in complex real-world scenes is a challenging problem. State-of-the-art methods typically use supervised learning relying on significant amounts of training data to achieve good detection results. However, labeling training data is tedious, expensive, and error-prone. This paper presents a novel method to improve detection performance by supplementing real-world data with synthetically generated training data. We consider the case of detecting people in crowded scenes within an AdaBoost-framework employing Haar and Histogram-of-Oriented-Gradients (HOG) features. Our evaluations on real-world video sequences of crowded scenes with significant occlusions show that the combination of real and synthetic training data significantly improves overall detection results.

Object tracking is important for video analysis applications. However, tracking through occlusions is a difficult task due to significant appearance changes of the objects. Approaches based on either global features or one kind of local features can not solve the problem completely. In this paper, a multi-cue based tracking approach is introduced. It combines a corner tracking with a color and a shape model to resolve the object tracking problem through occlusions for most scenes (indoor and outdoor).To obtain an objective evaluation of the proposed method, a set of detection and tracking measures are used to perform a quantitative analysis based on a large sequence dataset with ground-truth annotation. The experimental results show that the proposed approach works robustly under varying conditions.

The composition of panoramic images has recently received considerable attention. While panoramic images were first used mainly as a flexible visualization technique, they also found application in video coding, video enhancement, format conversion, and content analysis. The topic has enlarged and diverged into many specialized research directions, which makes it difficult to stay in touch with recent developments. This paper intends to give an overview of the current state of research, including recent developments. Two of the applications of sprite coding and global-motion estimation are presented in more detail to provide some insights into the system aspects.

This paper addresses the automatic analysis of court-net sports video content. We extract information about the players, the playing-field in a bottom-up way until we reach scene-level semantic concepts. Each part of our framework is general, so that the system is applicable to several kinds of sports. A central point in our framework is a camera calibration module that relates thea-prioriinformation of the geometric layout in the form of a court model to the input image. Exploiting this information, several novel algorithms are proposed, including playing-frame detection, players segmentation and tracking. To address the player-occlusion problem, we model the contour map of the player silhouettes using a nonlinear regression algorithm, which enables to locate the players during the occlusions caused by players in the same team. Additionally, a Bayesian-based classifier helps to recognize predefined key events, where the input is a number ofreal-worldvisual features. We illustrate the performance and efficiency of the proposed system by evaluating it for a variety of sports videos containing badminton, tennis and volleyball, and we show that our algorithm can operate with more than 91% feature detection accuracy and 90% event detection.

Most existing feature-matching methods utilize texture correlation for feature matching, which is usually sensitive to contrast changes. This paper proposes a new feature-point matching algorithm that does not rely on the image texture. Instead, only the smoothness assumption, which states that the displacement field in a neighborhood is coherent (smooth), is used. In the proposed method, the collected correspondences of a group of feature points within a neighborhood are efficiently determined such that the coherence measure of the displacement field in the neighborhood is maximized. The experimental results show that the proposed method is invariant to contrast changes and significantly outperforms the conventional block-matching technique.

The matching of planar shapes can be cast as a problem of finding the shortest path through a graph spanned by the two shapes, where the nodes of the graph encode the local similarity of respective points on each contour. While this problem can be solved using dynamic time warping, the complete search over the initial correspondence leads to cubic runtime in the number of sample points. In this paper, we cast the shape matching problem as one of finding the shortest circular path on a torus. We propose an algorithm to determine this shortest cycle which has provably sub-cubic runtime. Numerical experiments demonstrate that the proposed algorithm provides faster shape matching than previous methods. As an application, we show that it allows to efficiently compute a clustering of a shape data base.

We investigate the coding of multiview images obtained from a set of multiple cameras. To exploit the inter-view correlation, two view-prediction tools have been implemented and used in parallel: a block-based motion compensation scheme and a depth image based rendering technique (DIBR). Whereas DIBR relies on an accurate depth image, the block-based motion-compensation scheme can be performed without any geometry information. Our encoder adaptively selects the most appropriate prediction scheme using a rate-distortion criterion for an optimal prediction-mode selection. The attractiveness of the algorithm is that the compression algorithm is robust against inaccurately estimated depth images and requires only one single reference camera for fast random-access to different views. We present experimental results for several multiview sequences, that result in a quality improvement of up to 1.4 dB as compared to H.264 compression.

This paper presents a novel depth-image coding algorithm that concentrates on the special characteristics of depth images: smooth regions delineated by sharp edges. The algorithm models these smooth regions using piecewise-linear functions and sharp edges by a straight line. To define the area of support for each modeling function, we employ a quadtree decomposition that divides the image into blocks of variable size, each block being approximated by one modeling function containing one or two surfaces. The subdivision of the quadtree and the selection of the type of modeling function is optimized such that a global rate-distortion trade-off is realized. Additionally, we present a predictive coding scheme that improves the coding performance of the quadtree decomposition by exploiting the correlation between each block of the quadtree. Experimental results show that the described technique improves the resulting quality of compressed depth images by 1.5-4 dB when compared to a JPEG-2000 encoder.

The capturing of panoramic 360° images has become a popular photographic technique. While a panoramic image gives an impressive view of the environment, many people have difficulties to understand the spatial scene arrangement from this flat image. In this paper, we present a new visualization technique for panoramic images based on a coarse reconstruction of the indoor environment, in which the panorama was captured. Applications of this are, for example, real-estate or hotel advertising, featuring virtual tours through the apartment. We use a semi-automatic reconstruction process, in which the user marks the room corners in the panoramic images. These can be translated into viewing-angle measurements, from which our algorithms can compute the exact sizes of the walls, based on a pre-defined geometric model.

This paper presents a new augmented-reality system designed to generate visual enhancements for TV broadcasted court-net sports. A probabilistic method based on the Expectation Maximization (EM) procedure is utilized to find the optimal feature points, thereby enabling the automatic acquisition of the camera parameters from the TV image with high accuracy. A virtual camera derived from the original camera, helps to synthesize a variety of virtual scenes, such as the scene from the viewpoint of a player, depending on the intention of the user. To preserve the visual nature of the original human motion, the player's shape and texture are extracted from the real video and texture-mapped onto the virtual video. The system was tested over a set of court-net sports videos containing tennis, badminton and volleyball and demonstrated promising results.

Object-oriented coding in the MPEG-4 standard enables the separate processing of foreground objects and the scene background (sprite). Since the background sprite only has to be sent once, transmission bandwidth can be saved. We have found that the counter-intuitive approach of splitting the background into several independent parts can reduce the overall amount of data. Furthermore, we show that in the general case, the synthesis of a single background sprite is even impossible and that the scene background must be sent as multiple sprites instead. For this reason, we propose an algorithm that provides an optimal partitioning of a video sequence into independent background sprites (a multisprite), resulting in a significant reduction of the involved coding cost. Additionally, our sprite-generation algorithm ensures that the sprite resolution is kept high enough to preserve all details of the input sequence, which is a problem especially during camera zoom-in operations. Even though our sprite generation algorithm creates multiple sprites instead of only a single background sprite, it is fully compatible with the existing MPEG-4 standard. The algorithm has been evaluated with several test sequences, including the well-known Table-tennis and Stefan sequences. The total coding cost for the sprite VOP is reduced by a factor of about 2.6 or even higher, depending on the sequence.

Background subtraction is a popular algorithm for video object segmentation. It identifies foreground objects by comparing the input images with a pure background image. In camera-motion compensated sequences, small errors in the motion estimation can lead to large image differences along sharp edges. Consequently, the errors in the image registration finally lead to segmentation errors. This paper proposes a computationally efficient approach to detect image areas having a high risk of showing misregistration errors. Furthermore, we describe how existing change detection algorithms can be modified to avoid segmentation errors in these areas. Experiments show that our algorithm can improve the segmentation quality. The algorithm is memory efficient and suitable for real-time processing.

Semantic analysis of sport sequences requires camera calibration to obtain player and ball positions in real-world coordinates. For court sports like tennis, the marker lines on the field can be used to determine the calibration parameters. We propose a real-time calibration algorithm that can be applied to all court sports simply by exchanging the court model. The algorithm is based on (1) a specialized court-line detector, (2) a RANSAC-based line parameter estimation, (3) a combinatorial optimization step to localize the court within the set of detected line segments, and (4) an iterative court-model tracking step. Our results show real-time calibration of, e.g., tennis and soccer sequences with a computation time of only about 6 ms per frame.

Sports video processing is an interesting topic for research, since the clearly defined game rules in sports provide the rich domain knowledge for analysis. Moreover, it is interesting because many specialized applications for sports video processing are emerging. This paper gives an overview of sports video research, where we describe both basic algorithmic techniques and applications.

We present a new semiautomatic segmentation tool, which is motivated by the intelligent scissors algorithm, but which uses a modified concept of user-interaction. This new interface provides better capabilities for modifying previous segmentation results. The advantage of the new approach is that it enables to gradually increase the quality of the segmentation. The segmentation tool is based on a shortest circular path search within a corridor that is drawn by the user along the object boundary. For this purpose, we present a new algorithm for computing the shortest circular paths. Our algorithm is so fast that it almost achieves the speed of a regular noncircular shortest path search, while still ensuring an optimal solution.

The background subtraction algorithm is a frequently-used object segmentation technique because of its algorithmic simplicity. However, we show that, for general rotational camera-motion, it is impractical, or even impossible, to use a single background image. As a solution, we propose to use multi-sprite backgrounds which enables processing of arbitrary rotational camera-motion. The paper describes a complete video-object segmentation system employing multi-sprites. The system generates object masks and background sprites that are compatible with MPEG-4 object-oriented video-coding tools. Good segmentation results are also obtained for sequences which cannot be processed with ordinary background images.

A video summary is a sequence of video clips extracted from a longer video. Much shorter than the original, the summary preserves its essential messages. In the ECHO (European Chronicles On-line) project, a system was developed to store and manage large collections of historical films for the preservation of cultural heritage. At the University of Mannheim, we have developed the video summarization component of the ECHO system. We discuss the particular challenges the historical film material poses, and how we have designed new video processing algorithms and modified existing ones to cope with noisy black-and-white films.

Object-oriented coding in the MPEG-4 standard enables the separate processing of foreground objects and the scene background (sprite). Since the background sprite only has to be sent once, transmission bandwidth can be saved. This paper shows that the concept of merging several views of a non-changing scene background into a single background sprite is usually not the most efficient way to transmit the background image. We have found that the counter-intuitive approach of splitting the background into several independent parts can reduce the overall amount of data. For this reason, we propose an algorithm that provides an optimal partitioning of a video sequence into independent background sprites (a multi-sprite), resulting in a significant reduction of the involved coding cost. Additionally, our algorithm results in background sprites with better quality by ensuring that the sprite resolution has at least the final display resolution throughout the sequence. Even though our sprite generation algorithm creates multiple sprites instead of a single background sprite, it is fully compatible with the existing MPEG-4 standard. The algorithm has been evaluated with several test-sequences, including the well-known Table-tennis and Stefan sequences. The total coding cost could be reduced by factors of about 2.7 or even higher.

Knowing the background image of a video scene simplifies the general video-object segmentation problem and therefore it is required by several automatic segmentation algorithms. This paper presents a new background estimation algorithm which is applicable to complex video sequences where many objects are simultaneously visible and the background is visible for a short time period only. The algorithm applies a rough segmentation of the input images into foreground and background regions to exclude the foreground objects from background synthesis. This prevents a bias of the synthesized background image towards the color of foreground objects. Experiments show that the obtained background images differ significantly less from the real background than those obtained with previous algorithms.

Clustering techniques have been widely used in automatic video-summarization applications to group shots with comparable content. We enhance the popular k-means clustering algorithm to integrate user-supplied domain-knowledge into the cluster generation step. This provides a convenient way to exclude scenes from the summary which are a-priori known to be irrelevant. Furthermore, we added an additional, time-constrained clustering step preceding the scene clustering step to exclude short ranges with transitional content. This makes the algorithm robust to fading and wipe-effects in the input without requiring explicit cut detection.