The paper proposes an algorithm to robustly reconstruct an accurate billboard model of an individual object including an occluded one in each camera. Each billboard model is utilized to synthesize high-quality, free-viewpoint video especially for outdoor sport scenes in which roughly calibrated cameras are sparsely placed. The two main contributions of the proposed algorithm are (1) robustness to occlusions caused by overlaps of multiple objects in every camera, that is one of the biggest issues for billboard-based method, and (2) applicability to challenging shooting conditions in which accurate 3D model cannot be reconstructed because of calibration errors, small number of cameras and so on. In order to achieve the contributions above, the algorithm does not try to reproduce an accurate 3D model of each object but utilize a "rough 3D model". The algorithm precisely extracts an individual object region in every camera by reconstructing a "rough 3D model" of each object and back-projecting it to every camera. The 3D coordinate for each billboard to be located is calculated based on the position of a rough 3D model. Experimental results compare the visual quality of free-viewpoint videos synthesized with our proposed method and conventional methods and show the effectiveness of our proposed method in terms of the naturalness of positional relationships and the fineness of the surface textures of all the objects.

Free viewpoint technologies that synthesize virtual viewpoint by using multiple actual videos are one of the hottest topics in the video processing field, and would provide immersive experiences for users. To realize this concept, many conventional methods have been proposed. However, these methods require high computational cost to synthesize a virtual viewpoint because they have to calculate huge data to express three-dimensional information, e.g. the shapes of objects, only by using two-dimensional video. This makes them inadequate for end-to-end (from video capture to virtual view rendering) live streaming. To overcome this problem, we propose a simple and fast free-viewpoint synthesis method based on a visual hull, which is a general concept in this field. We calculate the silhouette of an object along planes in virtual space by simple projection from video images to the 3D space, and express the whole shape of the object by integrating the planes. This scheme works very quickly while providing fine quality because it consists of standard functions in general GPU architecture. The experimental results show our method can generate a fine virtual view of an object by using multiple videos in real time.

In this paper, we propose a new method of automatic silhouette extraction of multiple moving objects with high accuracy for free viewpoint stadium sports video synthesis. The proposed method is basically composed of three parts, including a global extraction based on temporal background subtraction, a classification step based on the constraints of extracted candidates of objects, and local refinement based on the statistical information of the chrominance component of each extracted object. Experimental results show that the proposal outperforms the temporal background subtraction model and Gaussian Mixture Model (GMM) in terms of both objective and subjective evaluations. In addition, the quality of the synthesized free viewpoint sports video is also enhanced by adopting more accurate silhouettes of objects that are extracted by our proposed method. Furthermore, as there is no manual operation in the proposed method, the automatic multiple silhouettes extraction has also been fully realized.

Recently, 3DTV has drawn much attention as a new broadcasting service. In spite of technical advances in 3DTV broadcasting services, there are two barriers that hinder the new services from being launched for terrestrial broadcasting: the lack of available bandwidth for transmission of additional-view video via terrestrial channel and backward compatibility with the legacy 2-D HDTV services. As an alternative, a hybrid stereoscopic video coding scheme is proposed for a stereoscopic 3DTV service, where one-view video is transmitted through the legacy broadcasting systems and the other one is delivered via broadband networks to which TV terminals are connected. In this paper, the proposed hybrid stereoscopic video coding scheme incorporates an MPEG-2 codec for backward compatibility with a legacy 2-D HDTV service and an HEVC codec with inter-view prediction coding as an extension to encode additional-view video sequences with high coding efficiency. The proposed inter-view prediction coding scheme in the extended HEVC incorporates an advanced motion and disparity vector prediction (AMDVP) method for enhanced motion- and disparity-compensated coding. The experimental results demonstrate that the proposed hybrid stereoscopic video coding scheme achieves an average coding efficiency of 38.22% (32.86%) in BD-rate or 1.394 dB (1.235 dB) in BD-PSNR for the out-band (in-band) scenario.

This paper presents a spatio-temporal graph-based method of detecting and tracking moving objects by treating the encoded blocks with non-zero motion vectors and/or non-zero residues as potential parts of objects in H.264/AVC bitstreams. A spatio-temporal graph is constructed by first clustering the encoded blocks of potential object parts into block groups, each of which is defined as an attributed subgraph where the attributes of the vertices represent the positions, motion vectors and residues of the blocks. In order to remove false-positive blocks and to track the real objects, temporal connections between subgraphs in two consecutive frames are constructed and the similarities between subgraphs are computed, which constitutes a spatio-temporal graph. We show the experimental results that the proposed spatio-temporal graph-based representation of potential object blocks enables effective detection for the small-sized objects and the objects with small motion vectors and residues, and allows for reliable tracking of the detected objects even under occlusion. The identification of the detected moving objects is determined as rectangular regions of interest (ROIs) for which the ROI sizes and positions are adaptively adjusted to give the best approximation of the real shapes and positions of the objects.

In this paper we present a novel method to detect and track moving objects in H.264/AVC bitstreams by processing motion vector and residue information. The encoded blocks with nonzero motion vectors and residues are first detected as moving object candidates. A spatio-temporal graph in video sequences is then constructed to represent groups of blocks in each frame and their associations to the other groups of blocks in subsequent frames. Identification and refinement of ROIs for moving objects being tracked are done by graph matching and adaptive ROI-size adjustment. The experimental results show that the proposed method can correctly identify real moving objects from frame to frame and can effectively detect small-sized objects and objects with small motion vectors and residues, as well as by recognizing moving objects even under occlusion.