We are the first to apply split rendering to AR to improve the quality of the transparent channel. The proposed method evaluates a cloud-based AR streaming system that offloads photorealistic rendering to a cloud server and splits the rendering between the cloud server and the mobile device (split rendering). Server-side rendering is capable of rendering photorealistic images, but the quality of the image is degraded by coding when the video is compressed and transmitted. In particular, degradation of the transparent channel significantly reduces the subjective image quality. The proposed method avoids this degradation by rendering the transparent channel at the mobile terminal. In addition, the server improves the coding efficiency by padding the transparent areas. Nonlinear quantization of the difference images contributes to improved subjective image quality. Experimentally, we confirmed that the proposed method has a SSIM gain of 0.022 compared to the conventional method.

In this paper, we propose a novel free-viewpoint video synthesis method that adaptively extracts the texture even from occluded areas. The conventional method based on object segmentation and inter-camera interpolation has two major problems. One is that textures of incorrectly segmented objects degrade image quality in the synthesized output of free-viewpoint video. For example, some object textures have missing regions and other object textures include unwanted regions. The other problem is that the inter-camera interpolation often causes inconsistency between the object appearance and corresponding moving direction. In order to overcome these problems, we propose a new texture acquisition scheme based on inter-frame interpolation to handle the case where object segmentation and inter-camera interpolation are both insufficient. In addition, the proposed method enables adaptive selection among three texture acquisition schemes, segmentation, inter-camera interpolation and inter-frame interpolation. This selection is optimally conducted considering the segmentation results and the direction of the virtual view point. The experimental results revealed that the proposed method can acquire an appropriate texture. Consequently, the subjective quality of generated free-viewpoint video is successfully improved while maintaining the original motion property even for occluded objects.

In this paper, we propose robust segmentation of foreground objects, such as human regions from sparsely arranged multi-view cameras. This work is intended for an immersive telepresence system. The system can be realized by segmenting a conferee (i.e. the foreground) from captured video at each conference site and then synthesizing life-sized textures with the background of another space. However, segmentation is a very challenging problem where the background has a similar texture to the foreground or the illumination varies according to time changes. Actually, segmentation accuracies of conventional methods are not sufficient to realize the telepresence system. The proposed method achieves sufficient segmentation quality to realize the system by directly estimating the foreground regions in a three-dimensional space based on the object existence probability for an individual camera and the color similarity among multiple cameras. Experimental results showed the effectiveness of the proposed method regarding foreground segmentation accuracy. Furthermore, confirmation was made that the experience of motion parallax for head movement could be naturally realized.

In this paper, we propose a dynamic camera calibration and object extraction method for sport videos captured with a moving pan-tilt-zoom camera. Such technology realizes an immersive free-viewpoint experience whereby audiences can see real sport scenes from any viewpoint. Camera calibration and object extraction are two of the most important processes for rendering free-viewpoint video, since a 3-dimensional model of each object needs to be reconstructed in every frame based on objects' textures and camera parameters. Most conventional rendering methods only use static cameras whose camera parameters and background models change little. However, since the cameras have to be set apart widely enough to capture the entire scene, the resolution of each object becomes low and is not sufficient for rendering high-quality free-viewpoint video. In order to obtain the texture of an object in high resolution from a moving pan-tilt-zoom camera, our proposed method estimates camera parameters by identifying reliable corresponding feature points between video frames, and also extracts the precise textures of objects using estimated camera parameters. Experimental results revealed that the proposed method successfully estimated precise camera parameters compared to the conventional method. Furthermore, by applying our proposed approach, the free-viewpoint video was rendered without visual defects.

This research proposes a novel method to extract image regions of products from an advertisement video, by analyzing features which are completely independent from the target object. Namely, we focus on how each product is emphasized in the video production, and propose the utilization of low-level visual features which leverage the technical know-how of video producers. By using such features, our proposal can achieve highly accurate detection of temporal and spatial locations of the advertised products, regardless of the product domain. Evaluation of the proposed method has been conducted with the actual advertisement video, in which accuracy of 79.4% in F-measure has been achieved.

TV logo recognition is one of the suggested solutions for preventing unauthorized duplication and redistribution. The major problem of the previous logo recognition is that the matching process against the reference logos requires much time. Since millions of minutes of TV programs are produced and broadcasted every day, an efficient algorithm is necessary. In this paper, we propose lightweight logo recognition using an image moment. An image moment represents a geometrical feature by a numerical low dimensional vector and thus matching can be achieved by a simple distance metric. Since an image moment itself is not so robust to noise, we introduce conspicuousness of a transparent logo according to visual properties of its background and surrounding regions. A validation experiment using the actual TV programs achieves much faster processing than the existing method with only a small degradation in accuracy. In the proposed method, computational complexity is independent of the number of the reference logos and the size of the logo region.

In this paper, we present a pioneering study on genre classification for home video. Analyzing home video is a challenging problem because it is generally unstructured and of low production quality in audio and video. Our approach is to define a set of genres referring to those in the actual video sharing site, and to extract salient low level features from MPEG compressed data which are robust to low production quality content. Experimental results based on ensemble learning show that our proposed method achieves around 0.7 to 0.8 F-measure values with 37.5 times faster processing than real-time playback for QVGA resolution home video.

This paper proposes shot genre classification from MPEG compressed movies, as one of the high-level indexing methods for audio-visual contents. Through statistical analysis of low-level and mid-level audio-visual features on compressed domain, the proposed method can achieve subjectively accurate shot classification within the movies into predefined genre set, which can be applied to various content handling applications, such as summarization, navigation, editing, filtering, and so on. By feeding subjectively evaluated feature set for each shot genre into the linear machine decision tree classifier, each shot is classified at very low cost. The experimental results show that most of the shots in the movies can be classified into subjectively accurate genres, and also the dominant shot genre can correctly resolve each movie genre.

The paper proposes a novel method for automatically summarizing MPEG coded audio-video content in the compressed domain, and describing audio-video summaries in the MPEG-7 standard format. Semantically important features for summary generation are carefully analyzed, and two types of summaries, digest and highlight, are extracted. A digest extraction is based on an audio level estimation and visual features analysis, where digest shots are adaptively determined for user-specified duration by utilizing the MPEG-7 based motion features. A highlight-based summarization from TV broadcast sports programmes is achieved by analyzing audio class and audio level. The extracted summaries and related features are described by the MPEG-7 description tools. Experimental results show that the digests and highlights can be successfully extracted in various kinds of contents without a priori knowledge of content characteristics.

This paper proposes a fast and efficient transcoding algorithm from MPEG-2 video to MPEG-1 video where spatial resolution reduction is involved. We use quantization parameters along with motion information of the incoming MPEG-2 video bitstream to estimate the motion vectors for MPEG-1 transcoding without any additional motion estimation process. Since such information can be directly extracted from the original MPEG-2 video bitstream, very fast transcoding can be achieved. In addition, since the proposed method is not restricted to a specific coding structure, it can be applied to various types of GOP structures. The experimental results show that the proposed method can provide approximate motion vectors to that of full range motion estimation. therefore transcoding efficiency can be increased and picture quality is enhanced when compared with the conventional fast transcoding methods.

Audio information classification becomes a very important task for such purposes as automatic keyword spotting and other content-based audio-visual query systems. In this paper, we describe a fast and accurate audio data classification method on the MPEG coded data domain. Firstly silent segments are detected using a robust approach for different recording conditions. Then the non-silent segments are classified into three types, music, speech, and applause using temporal density, bandwidth and center frequency of subband energy. In order to be robust for a variety of audio sources as much as possible, we use Bayes discriminant function for multivariate Gaussian distribution instead of manually adjusting a threshold for each discriminator. In the experiment, every one-second of MPEG audio data is classified and about 90% of audio and speech segments have been successfully detected. As for the detection speed, less than 20% of MPEG audio decoding processing power is required.

Moving object detection is a useful tool for such purposes as intelligent video browsing/analysis and video surveillance systems. Previously several algorithms have been reported on moving object detection from compressed video data. However, in these algorithms, flat regions are often falsely detected. In addition if multiple moving objects are involved, each moving object may not be identified and therefore object tracking becomes very difficult. We describe a fast moving detection and identification method from MPEG coded data. The experimental results show that each moving object is successfully identified for several surveillance sequences.

Although a GOP (group of picture) length is often fixed in MPEG, this may not guarantee the best picture quality from the point of view of coding efficiency. Furthermore, it is also well known that P-picture interval M=3 does not provide the best quality for all sequences. We propose an MPEG encoding algorithm with scene adaptive dynamic GOP structure to enhance the picture quality in the cases mentioned above. Using macroblock characteristics such as luminance activity and simple motion compensation on the activity domain, N and M values are dynamically determined.

Formerly, once the audio data is compressed, transcoding is used to scale the bit rate, where decoding and re-encoding are taking place. Therefore, data manipulation of coded data has been very complex and time consuming work. We describe three algorithms for bit rate scaling in the coded MPEG data domain. One is a bandwidth limitation method cutting higher frequency components until the target data rate is satisfied. The other two use a re-quantization process where a quantization step in each subband is modified. One of them reflects the psychoacoustic model from bit allocation information obtained in the bitstream in order to improve the bit rate scaling efficiency. The simulation results show that the re-quantization process provides a very high conversion efficiency and a nearly equal sound quality can be obtained as directly coding from PCM by reflecting the psychoacoustic model. It is also shown that a very fast scaling (factor of six) have been achieved when compared with the transcoding method.

This paper proposes a very fast and accurate scene change detection algorithm on MPEG coding parameter domain. The fast operation can be obtained by spatio-temporally subsampling coding information and by exploiting only coding parameters extracted in variable length decoding (VLD) stage, while the accurate detection is accomplished by examining the statistical characteristics of various scene changes on the coding parameter domain. The computer simulation shows that the proposed algorithm can accomplish detection more than 5 times faster than that of real-time playback for MPEG-2 video sequences using a standard workstation. It is also shown that most of abrupt scene changes, dissolve transitions, and wipe transitions have been successfully detected.