This paper proposes a method that enhances the compression performance of the current model under development for the upcoming MPEG standard on Feature Compression for Machines (FCM) [1]. By truncating low-activation feature channels and signaling these truncations in the bitstream, the method reduces bitrate while preserving task accuracy. Experimental results show an average BD-rate reduction of 10.59% across datasets and tasks, demonstrating gains in bitrate efficiency.

The split-inference paradigm divides an artificial intelligence (AI) model into two parts. This necessitates the transfer of intermediate feature data between the two halves. Here, effective compression of the feature data becomes vital. In this paper, we employ Z-score normalization to efficiently recover the compressed feature data at the decoder side. To examine the efficacy of our method, the proposed method is integrated into the latest Feature Coding for Machines (FCM) codec standard under development by the Moving Picture Experts Group (MPEG). Our method supersedes the existing scaling method used by the current standard under development. It both reduces the overhead bits and improves the end-task accuracy. To further reduce the overhead in certain circumstances, we also propose a simplified method. Experiments show that using our proposed method shows 17.09% reduction in bitrate on average across different tasks and up to 65.69% for object tracking without sacrificing the task accuracy.

As numerous edge devices start implementing intelligent components, the challenges of energy consumption, bandwidth efficiency, and privacy gain significance. One proposed solution relies on the paradigm of split inference, which optimizes the delegation of the computational load between edge and remote devices. We developed and implemented the standard-compliant split inference system with an encoder and decoder capable of real-time streaming and processing. Our system outperforms state-of-the-art video compression implementations by an average of 83% bitrate reduction, while preserving privacy. We demonstrate the system's real-time performance on consumer devices, with interactive visualizations of object detection and segmentation, incorporating real-time metrics. Demo video: https://youtu.be/bmCbUo_ZWWU

This work introduces the Perceptual Algorithm for Versatile video Encoding using Neural Networks (PAVEN), a subjective video coding algorithm designed to reduce the bit rate in videos encoded with the Versatile Video Coding (VVC) standard without compromising subjective video quality. The algorithm uses a deep learning model specifically trained by the authors to account for the specific characteristics of video signals. The trained model outperforms others in the literature by more accurately identifying areas of the frames where viewers are most likely to focus their attention. The output of the deep learning model is further processed to merge all disjoint areas and adapt the result to the Coding Tree Unit (CTU) size in VVC, allowing for greater compression in less important areas. The results show an average reduction in bit rate of 7% while maintaining the same subjective video quality, validated through viewer interviews using the Mean Opinion Score (MOS) metric.

This article presents a new approach to predict learning difficulty in applications such as e-learning using eye movement and pupil response. We have developed 12 eye response features based on psycholinguistics, contextual information processing, anticipatory behavior analysis, recurrence fixation analysis, and pupillary response. A key aspect of the proposed approach is the temporal analysis of the feature response to the same concept. Results show that variations in eye response to the same concept over time are indicative of learning difficulty. A Feature Weighted Linguistics Classifier (FWLC) was developed to predict learning difficulty in real time. The proposed approach predicts learning difficulty with an accuracy of 90%.

Efficient storing and retrieval of medical images has direct impact on reducing costs and improving access in cloud-based health care services. JPEG 2000 is currently the commonly used compression format for medical images shared using the DICOM standard. However, new formats such as high efficiency video coding (HEVC) can provide better compression efficiency compared to JPEG 2000. Furthermore, JPEG 2000 is not suitable for efficiently storing image series and 3-D imagery. Using HEVC, a single format can support all forms of medical images. This paper presents the use of HEVC for diagnostically acceptable medical image compression, focusing on compression efficiency compared to JPEG 2000. Diagnostically acceptable lossy compression and complexity of high bit-depth medical image compression are studied. Based on an established medically acceptable compression range for JPEG 2000, this paper establishes acceptable HEVC compression range for medical imaging applications. Experimental results show that using HEVC can increase the compression performance, compared to JPEG 2000, by over 54%. Along with this, a new method for reducing computational complexity of HEVC encoding for medical images is proposed. Results show that HEVC intra encoding complexity can be reduced by over 55% with negligible increase in file size.

The papers in this special section focus on the topic of video analytics. Also known as video content analysis, video analytics refer to the capability of automatically analyzing video to extract knowledge/information and detect and determine temporal and spatial events. The algorithms designed for these analytics can be implemented as software on general-purpose machines, or as hardware in specialized video processing units. Video analytics is still an emerging technology with techniques that are continuously being developed to help make widespread implementation feasible in the years ahead. Such analytics has been typically used in semantic categorization and retrieval of video databases. A goal of this special issue is to focus on video analytics beyond categorization and retrieval. With increasing hardware capability and advances in algorithms used, real-time video analytics is now being used in a wide range of domains including entertainment, health-care, retail, automotive, transport, home automation, emotion analysis, aesthetics, inappropriate content detection, safety and security. For instance, video analytics are increasingly being deployed for real-time alerts in situation monitoring systems such as traffic surveillance (vehicle counting), counting people in lines (some hospitals are using this to get more nurses from a less busy department to serve thewaiting patients) and in manufacturing (for monitoring and counting). From the sensing aspect, 3-D cameras such as RGB-D and LiDAR (Light Detection and Ranging) cameras are becoming more and more affordable, enabling additional areas of research and applications, such as self-driving cars employing video analytics on LiDAR captured data for path planning as well as obstacle detection.

A method is introduced to enhance the High Efficiency Video Coding early skip decision process. The proposed method exploits visual acuity and high-velocity motion perception of the human visual system (HVS). The method determines coding units where the HVS is unlikely to notice the changes and hence can be skipped. In this paper, the focus is mobile devices, such as smart phones, where displays are less than 5 in diagonal length and bandwidth availability is low. The proposed method exploits the mismatch between the video resolution, display resolution, and visual acuity of users. This method has very low impact on subjective quality while reducing encoding time for low bitrate environments. The proposed method reduces both encoding time and bitrate; the results show an average of 21.7% decrease in encoding time, 13.4% decrease in average bitrate, and very low impact on subjective quality ratings. The proposed methods can be adapted to decrease encoding time and bitrate in devices with lower resolution displays.

The High Efficiency Video Coding (HEVC) standard roughly doubles the rate-distortion compression performance of its predecessor, H.264/AVC, at a cost of a high computational complexity. Moreover, intra sequences are commonly used at video editing or post-production, making it necessary a migration from H.264/AVC to HEVC. This paper aims to propose a fast intra transcoding algorithm from H.264/AVC to HEVC.

High Efficiency Video Coding (HEVC) is the new video coding standard recently approved by ISO and ITU. HEVC allows a bit rate reduction greater than a 50% with respect to its predecessor, the H.264/AVC, offering the same perceptual quality, by means of a set of new tools that have been introduced. Compared with the current state of the art in image coding, such as JPEG, JPEG2000 or the new JPEG XR, the new Intra Frame coding performs a high compression process in the "All-Intra" mode. All these improvements are at expense of a high computational cost, making it considerably difficult to implement in real time. Hence, this paper presents a mechanism that can be used by the RDO algorithm to select the optimal coding block size for Intra-Prediction, by using a data mining classifier, based on a previous training. Experimental results show that the proposed algorithm can achieve a 30% of Time Savings over a wide range of high resolution sequences (Class A, B and F), with a negligible loss of coding efficiency.

This is a MM'14 Workshop Summary Abstract for PIVP'14 - 1st International Workshop on Perception Inspired Video Processing. Workshop provided a venue for researchers involved with perceptual video processing and coding to present their current work and to discuss future directions. The workshop program consisted of a keynote talk, oral presentations of full papers and interactive poster session for the short papers. All participants took part in the discussion panel at the end of workshop, exchanging ideas and suggestions for future work.

We present a method that employs human visual system characteristics for perceptually lossless optimization of video coding. Algorithm is robust and applicable for any modern hybrid coder. We decided to evaluate it using the most recent standard (HEVC/H.265) in order to show potential benefits for future applications. Experiments show savings of up to 8% in bitrate without any loss in quality as confirmed by two sets of subjective experiments using DSCQS and ACJ methodology.

The use of permutations in data compression is an aspect that is worthy of further exploration. The work that has been done in video compression based on permutations was primarily oriented toward lossless algorithms. The study of previous algorithms has led to a new algorithm that could be either lossless or lossy, for which the amount of compression and the quality of the output can be controlled. The lossless version of our algorithm performs close to lossy versions of H.264 and it improves on them for the majority of the videos that we analyzed. Our algorithm could be used in situations where there is a need for lossless compression and the video sequences are part of a single scene, e.g., medical videos, where loss of information could be risky or expensive.

This paper presents a system and methodology for crowdsourcing subjective quality evaluations. The goal of this work is to study the feasibility of fast, inexpensive, and reliable subjective quality evaluations with crowdsourcing. We present a web-based system for conducting subjective quality evaluations using crowdsourcing platform. The results were evaluated by comparison with the formal subjective tests conducted by MPEG as a part of video standardization process. High quality compressed videos with known MOS values are used as references in subjective evaluations instead of the original lossless videos to address the bandwidth limitations. The results show that crowdsourcing can be a reliable platform for subjective quality evaluations.

Video signature techniques based on tomography images address the problem of video identification. This method relies on temporal segmentation and sampling strategies to build and determine the unique elements that form the signature. In this paper an extension for these methods is presented; first an implementation for high contrast sub-sampling and up-sampling filters is used to increase tomography edge stability, then a robust temporal video segmentation system is used to replace the original method applied to determine shot changes more accurately, and finally a new feature extraction method, derived from the previously proposed sampling pattern, was implemented and tested, resulting in a highly distinctive set of signature elements. The video set for the test was recorded from live broadcast TV, using as queries 20 different commercials, with the objective of analyzing the audit proficiency of the signatures for new applications. The performance obtain over the version presented in [3] is significant with 99.58% of Recall and 99.35% of Precision prediction.

Today, world is rapidly turning to high definition multimedia. From engineering and programming point of view, this usually means more computation is needed and more memory space is required to achieve these higher qualities. In this paper we explore the use of parallelization opportunities in graphics processors to accelerate video encoding. We evaluate the NVIDIA CUDA[1] toolkit and evaluate the performance of motion estimation in video encoding. The main goal of this paper is to evaluate the capabilities of NVIDIA/CUDA and develop a process for implementing video/multimedia applications. We have discovered that the difference in performance when CUDA is not used properly can be over 100x. We show how we were able to use CUDA capabilities to reduce the motion estimation time from 7000 milli seconds to 70 milli seconds.

Based on our previous efforts, we introduce in this letter a high-efficient MPEG-2 to H.264 transcoder for the baseline profile in the spatial domain. Machine learning tools are used to exploit the correlation between the macroblock (MB) decision of the H.264 video format and the distribution of the motion compensated residual in MPEG-2. Moreover, a dynamic motion estimation technique is also proposed to further speed-up the decision process. Finally, we go a step further on our previous research efforts by combining the two aforementioned speed-up approaches. Our simulation results over more than 40 sequences at common intermediate format and quarter common intermediate format resolutions show that our proposal outperforms the MB mode selection of the rate-distortion optimization option of the H.264 encoder process by reducing the computational requirements by up to 90%, while maintaining the same coding efficiency. Finally, we conduct a comparative study of our approach with the most relevant fast inter-prediction methods for MPEG-2 to H.264 transcoder recently reported in the literature.

Low power consumption in computing systems is a key requirement for devices such as cell phones and cameras. In this paper we present a low power DCT implementation using a highly scalable multiplier. This paper focuses on IDCT with playback applications such as digital photo displays. The proposed solution exploits the fact that the size of the multiplications varies per stage in a multistage IDCT implementation and configuring multipliers to match the needs of each stage saves power. Results are compared with Wallace and array multipliers. We show that using a scalable multiplier and dynamically reconfiguring the width of the multiplier leads to significant power savings (over 72%) with negligible degradation in decoded image quality.

This paper proposes an improved Wyner-Ziv to H.264 video transcoder as part of a framework for mobile to mobile video communications. In this scheme, the user devices keep the low complexity constraints by using the Wyner-Ziv encoding and H.264 decoding algorithms. They shift their complexities to the network where the proposed transcoder is allocated. The main goal of the transcoder is to convert the bitstream and reduce the delay efficiently. The results show that the proposed transcoder reduces the complexity by a factor of 95% with a negligible rate-distortion loss.

Video identification or copy detection is a challenging problem and is becoming increasingly important with the growing popularity of online video services. The problem addressed in this paper is the identification of a given video clip in a given set of videos. For a given query video, the system returns all the instance of the video in the data set. This identification system uses video signatures based on video tomography. A robust and low complexity video signature is designed. The signatures are generated for video shots and not individual frames. This results in a compact signature of 64 bytes per video segment/ shot. The signatures are matched using simple Euclidean distance metric. Both signature generation and matching have a very low complexity. The system was evaluated using the dataset from video copy detection evaluations organized at the CIVR 2007. The results show that the proposed signatures outperform the results reported in the CIVR competition. The results also show that the signature is robust to common video transformations.

In mobile-to-mobile video communications, both the sender and the receiver devices should not have higher complexity requirements to perform complex video compression tasks. The traditional video coding solutions are not suitable to support this communications due to its extremely complex encoding algorithm. On the other hand, the new Wyner-Ziv video coding paradigm reduces the complexity of the encoder at the expenses of a more complex decoder. In this paper, we propose an improved WZ/H.264 video transcoder to support this mobile-to-mobile communications, using the low complexity Wyner-Ziv encoding and the traditional H.264 decoding to be implemented in the end-user devices. The improved transcoder converts the video from the Wyner-Ziv to H.264 and reuses the motion vectors generated in the Wyner-Ziv decoding, in order to reduce the computational complexity of the motion estimation process in the H.264 encoding. Simulations results show a complexity reduction up to 55% with negligible rate-distortion drop.

The current technological advances allow for the easy distribution and duplication of video content through various venues, the most popular being the World Wide Web. Upholding copyright laws has thus become an important task; the result to this problem needs to be implemented in such a way to provide automatic video identification. This paper presents a novel approach to video identification based on the video tomography technique. The proposed video signature was designed and evaluated based on its ability to uniquely identify videos. A video signature should be both independent and robust. This paper focuses on evaluating the independence of the proposed video signatures. The signatures were evaluated using a large test set developed for the MPEG activity on digital video signatures. The results show that the proposed signature works well and exhibits strong independence necessary to support general purpose video identification systems.

The authors develop a decoder/encoder system (transcoder) to solve the consumption constraint in the communications between end-user devices, when a new Wyner–Ziv (WZ)/H.264 framework is defined for being used in mobile-to-mobile environments. This approach is based on leaving to the devices only WZ video encoding and traditional video decoding; the lowest complexity algorithms in both paradigms. The system shifts the burden of complexity to the network, where an improved transcoder that reuses information between both paradigms is allocated. The WZ decoding motion vectors are used to reduce the H.264 motion estimation process. The proposed transcoder offers a complexity reduction up to 60% on average, without any rate distortion drop.

Most of the reported distributed video coding (DVC) architectures have a serious limitation that hinders its practical application. The uses of a feedback channel between the encoder and the decoder require an interactive decoding procedure which is a limitation for applications such as offline processing. On the other hand, the decoder needs an efficient way to estimate the probability of error without assuming the availability of the original video at the decoder. In this paper we continue with our previous works into a more practical DVC architecture which solves both problems based on the use of machine learning. The proposed approach is based on extracting the relationships that exist between the residual frame and the number of requests over this feedback channel. We apply these concepts to pixel-domain Wyner-Ziv coding demonstrating significant savings in bitrates with a little loss of quality.

One of the main concerns of the wide use of video surveillance is the loss of individual privacy. Individuals who are not suspects need not be identified on camera recordings. Mechanisms that protect the identity while ensuring legitimate security needs are necessary. Selectively encrypting objects that reveal identity (e.g., faces or vehicle tags) is necessary to preserve individualspsila right to privacy. This paper presents a compression algorithm independent solution that provides privacy in video surveillance applications. The proposed approach is based on the use of permutation based encryption to hide identity revealing features. The permutation based encryption tolerates lossy compression and allows decryption at a later time. The use of permutation based encryption makes the proposed solution independent of the compression algorithms used. The paper presents the performance of the system when using H.264 video encoding.

Distributed video coding has become increasingly popular in recent years among the researchers in video coding due to its attractive and promising features. DVC proposed a dramatic structural change to video coding by shifting the majority of complexity conventionally residing in the encoder towards the decoder. Nevertheless, these kinds of architectures have some serious limitations that hinder its practical application. The uses of a feedback channel between the encoder and the decoder requires an interactive decoding procedure which is a limitation for certain applications such as offline processing. On the other hand, the decoder needs an efficient way to estimate the probability of error without assuming the availability of the original video at the decoder. In this paper we investigate a first approximation to solve both problems based on the use of machine learning to extract the knowledge that exits between the residual frame and the number of requests over this feedback channel. Exploiting this correlation gives us a more practical architecture without higher complexity encoders. We apply these concepts to pixel-domain Wyner-Ziv coding and the results show a loss of 0.21 dB in the rate-distortion performance.

The H.264 standard achieves much higher coding efficiency than the MPEG-2 standard, due to its improved inter-and intra-prediction modes at the expense of higher computational complexity. Transcoding MPEG-2 video to H.264 is important to enable gradual migration to H.264. However, given the significant differences between the MPEG-2 and the H.264 coding algorithms, transcoding is a much more complex task and new approaches to transcoding are necessary. The main problems that need to be addressed in the design of an efficient heterogeneous MPEG-2/H.264 transcoder are: the inter-frame prediction, the transform coding and the intra-frame prediction. In this paper, we focus our attention on the inter-frame prediction, the most computationally intensive task involved in the transcoding process. This paper presents a novel macroblock (MB) mode decision algorithm for P-frame prediction based on machine learning techniques to be used as part of a very low complexity MPEG-2 to H.264 video transcoder. Since coding mode decisions take up the most resources in video transcoding, a fast MB mode estimation would lead to reduced complexity. The proposed approach is based on the hypothesis that MB coding mode decisions in H.264 video have a correlation with the distribution of the motion compensated residual in MPEG-2 video. We use machine learning tools to exploit the correlation and construct decision trees to classify the incoming MPEG-2 MBs into one of the several coding modes in H.264. The proposed approach reduces the H.264 MB mode computation process into a decision tree lookup with very low complexity. Experimental results show that the proposed approach reduces the MB mode selection complexity by as much as 95% while maintaining the coding efficiency. Finally, we conduct a comparative study with some of the most prominent fast inter-prediction methods for H.264 presented in the literature. Our results show that the proposed approach achieves the best results for video transcoding applications.

Recent developments have given birth to H.264/AVC: a video coding standard offering better bandwidth to video quality ratios than previous standards (such as H.263, MPEG-2, MPEG-4, etc.), due to its improved inter- and intraprediction modes at the expense of higher computation complexity. It is expected that H.264/AVC will take over the digital video market, replacing the use of previous standards in most digital video applications. This creates an important need for heterogeneous video transcoding technologies from older standards to H.264. In this paper, we focus our attention on the interframe prediction, the most computationally intensive task involved in the heterogeneous video transcoding process. This paper presents a novel macroblock (MB) mode decision algorithm for interframe prediction based on data mining techniques to be used as part of a very low complexity heterogeneous video transcoder. The proposed approach is based on the hypothesis that MB coding mode decisions in H.264 video have a correlation with the distribution of the motion compensated residual in the decoded video. We use data mining tools to exploit the correlation and derive decision trees to classify the incoming decoded MBs into one of the several coding modes in H.264. The proposed approach reduces the H.264 MB mode computation process into a decision tree lookup with very low complexity. For general validation purposes, we apply our algorithm to two of the most important heterogeneous video transcoders: MPEG-2 to H.264 and H.263 to H.264. Our results show that the our data-mining based transcoding algorithm is able to maintain a good video quality while considerably reducing the computational complexity by 72% on average when applied in MPEG-2 to H.264 transcoders, and by 62% on average when applied in H.263 to H.264 transcoders. Finally, we conduct a comparative study with some of the most prominent fast interprediction methods for H.264 presented in the literature. Our results show that the proposed data mining-based approach achieves the best results for video transcoding applications.

In this paper, we propose the use of data mining algorithms to create a macroblock partition mode decision algorithm for inter-frame prediction, to be used as part of a high-efficient H.263 to H.264 transcoder. We use machine learning tools to exploit the correlation and derive decision trees to classify the incoming H.263 MC residual into one of the several coding modes in H.264. The proposed approach reduces the H.264 MB mode computation process into a decision tree lookup with very low complexity. Experimental results show that the proposed approach reduces the inter-prediction complexity by as much as 60% while maintaining the coding efficiency.

VC-1 is now one of the three video coding standards for high definition DVD that include MPEG-2 and H.264. The coded is expected to be used in consumer electronic devices such as DVD and camcorders. The H.264 format has begun to see strong acceptance and is used in mobile devices such as iPod and mobile phones. While multi-format DVD players are expected to support the three high definition video coding formats, H.264 is expected to have broader support in devices and video download services. The need to move data among devices with different capabilities creates a need for transcoding. In this paper we present a P-frame transcoder for VC-1 to H.264 transcoding. The transcoder exploits the variable size transform used in VC-1 to select the variable block size for motion compensation in H.264. The transcoder reduces the complexity substantially without significant loss in quality.

The two key problems in video transcoding are complexity reduction and quality management. The complexity of H.264 encoding makes complexity reduction even more important. A good transcoder design reduces complexity with negligible loss in quality. While motion estimation complexity can be easily reduced by reducing the search range or eliminating the coding modes, these approaches will cause significant quality drop. This paper presents reduced complexity motion estimation for MPEG-2 to H.264 transcoding. The proposed approach uses MPEG-2 motion vectors to dynamically reduce the search range and the motion vector refinement window. This approach is based on the hypothesis that longer MPEG-2 motion vectors indicate higher motion and hence a larger search window as well as a larger refinement window is necessary. Experimental results show that the proposed approach reduces the motion estimation complexity by as much as 85% with negligible loss in PSNR. We show that the results hold across video resolutions and bitrates.

H.264 is a highly efficient and complex video codec. The complexity of the codec makes it difficult to use all its features in resource constrained mobile devices. This paper presents a machine learning approach to reducing the complexity of Intra encoding in H.264. Determining the macro block coding mode requires substantial computational resources in H.264 video encoding. The goal of this work to reduce MB mode computation from a search operation, as is done in the encoders today, to a computation. We have developed a methodology based on machine learning that computes the MB coding mode instead of searching for the best match thus reducing the complexity of Intra 16x16 coding by 17 times and Intra 4x4 MB coding by 12.5 times. The proposed approach uses simple mean value metrics at the block level to characterize the coding complexity of a macro block. A generic J4.8 classifier is used to build the decision trees to quickly determine the mode. We present a methodology for Intra MB coding. The results show that intra MB mode can be determined with over 90% accuracy. The proposed can also be used for determining MB prediction modes with an accuracy varying between 70% and 80%.

A novel encryption model for digital videos is presented. The model relies on the encryption-compression duality of certain types of permutations acting on video frames. In essence, the proposed encryption process preserves the spatial correlation and, as such, can be applied prior to the compression stage of a spatial-only video encoder. Several algorithmic modes of the proposed model targeted for different application requirements are presented and analyzed in terms of security and performance. Experimental results are generated for a number of standard benchmark sequences showing that the proposed method, in addition to providing confidentiality, preserves or improves the compression ratio. 1

This paper presents a novel background modeling and subtraction approach for video object segmentation. A neural network (NN) architecture is proposed to form an unsupervised Bayesian classifier for this application domain. The constructed classifier efficiently handles the segmentation in natural-scene sequences with complex background motion and changes in illumination. The weights of the proposed NN serve as a model of the background and are temporally updated to reflect the observed statistics of background. The segmentation performance of the proposed NN is qualitatively and quantitatively examined and compared to two extant probabilistic object segmentation algorithms, based on a previously published test pool containing diverse surveillance-related sequences. The proposed algorithm is parallelized on a subpixel level and designed to enable efficient hardware implementation.

Though H.264 is highly efficient compared to MPEG-2, the wide and deep penetration of MPEG-2 creates a need for co-existence of these technologies and hence creates a need for transcoding. In this paper, we introduce and evaluate a novel macroblock partition mode decision algorithm for inter-frame prediction to be used as part of a high-efficient MPEG-2 to H.264 transcoder. The proposed tools are used to compute an optimal MB coding mode decision with significantly reduced computational complexity. Specifically, we achieve the computational savings by using the following MB information coming from MPEG-2: the MB coding modes in MPEG-2 and the mean and variance of the 16 4×4 sub blocks of the MPEG-2 residual MBs. We use data mining algorithms to develop a decision tree for obtaining the H.264 coding decision modes. Our results show that the proposed algorithm is able to maintain a good picture quality while considerably reducing the number of operations to be performed.

In this paper, we introduce and evaluate a low complexity macroblock partition mode decision algorithm for inter-frame prediction in MPEG-2 to H.264 transcoder. The proposed tools are used to compute an optimal MB coding mode decision with significantly reduced computational complexity. Specifically, we achieve the computational savings by using the following MB information coming from MPEG-2: the MB coding modes, the coded block pattern (CBPC) in MPEG-2, and the mean and variance of the 16 4times4 sub blocks of the MPEG-2 residual MBs. We use data mining algorithms to develop a decision tree for H.264 coding mode decisions. The decision trees are built using RD optimized mode decisions and result in highly efficient mode decisions. The proposed transcoder is 35% faster than the RD optimized H.264 reference transcoder without a significant PSNR degradation. The proposed transcoder performs over 3 dB better than the SAE cost based H.264 transcoding

This paper explores the challenges opportunities in developing and deploying 3D TV services. The 3D TV services can be seen as a general case of the multi-view video that has been receiving significant attention lately. The keys to a successful 3D TV experience are the availability of content, the ease of use, the quality of experience, and the cost of deployment. Recent technological advances have made possible experimental systems that can be used to evaluate the 3D TV services. We have developed a 3D TV prototype and have currently conducting our first user study to evaluate the quality and experience. These experiences have allowed us to identify challenges and opportunities in developing 3D TV services

The high definition video adoption has been growing rapidly for the last two years. The two high definition DVD formats HD-DVD and Blueray have mandated MPEG-2, H.264 and VC-1 as video compression formats. The coexistence of these different video coding standards creates a need for transcoding. In this paper, an efficient transcoding algorithm from VC-1 video to H.264 video is discussed. While there has been recent work on MPEG-2 to H.264 transcoding, the published work on VC-1 to H.264 transcoding is non-existent. There is very limited amount of published work on VC-1. This paper gives a brief overview of VC-1 and discusses the opportunities for low-complexity tools for VC-1 to H.264 transcoding. The paper considers I and P frame transcoding from VC-1 to H.264 and proposes two approaches 1) low cost design and 2) high cost design

The wireless networks are notoriously error prone and all errors cannot be prevented in real-time communications. The problem of error correction becomes even more challenging in mobile multicast/broadcast applications. The mobile devices are being equipped with multiple modems that could work simultaneously; for example, devices with both GSM (WAN) and WLAN networks such as WiFi. These multi-modal devices can use the second network to improve their error resilience. We propose a P2P approach to establish and utilize an error recovery channel on a secondary network for multi-user video applications. The mobile devices within the vicinity can utilize the WLAN network to form a P2P network for error recovery purpose. We developed and evaluated three error recovery models for error recovery over secondary networks. The proposed models balance the response time, bandwidth utilization, fairness, and unnecessary data received

Low bitrates enabled by the H.264/AVC standard come at the cost of significant decoding complexity. The problem is exacerbated when multi-view video coding (MVC), presently being standardized by the MPEG committee, is considered. Increasing the number of reference views will improve the prediction but also increase the encoding and decoding complexity. In this paper we present an alternative to transform coding and arithmetic coding used to code motion compensation residuals in H.264/AVC. Proposed approach is sorting permutation based, resulting in significantly decreased decoding complexity. We evaluate the efficiency of the coding and present initial results obtained for some sequences in published literature

There is a renewed interest in the 3DTV research primarily due to the advances in low cost 3D display technologies. The two views required for 3DTV can be compressed using standard video compression techniques. MPEG-2 is widely used in digital TV applications today and H.264/MPEG-4 AVC is expected to be the leading video technology standard for digital video in the near future. The compression gains and quality of 3DTV will vary depending on the video coding standard used. While inter-view prediction will likely improve the compression efficiency, new approaches such as asymmetric view coding are necessary to greatly reduce bandwidth requirements for 3DTV. This demonstration will show the quality of 3D video experience on autostereoscopic displays using H.264 and MPEG-2. We will also show the benefits of using asymmetric view coding as well as the role of eye dominance on 3DTV experience.

This paper presents a novel macroblock mode decision algorithm for inter-frame prediction based on machine learning techniques to be used as part of a very low complexity MPEG-2 to H.264 video transcoder. Since coding mode decisions take up the most resources in video transcoding, a fast macro block (MB) mode estimation would lead to reduced complexity. The proposed approach is based on the hypothesis that MB coding mode decisions in H.264 video have a correlation with the distribution of the motion compensated residual in MPEG-2 video. We use machine learning tools to exploit the correlation and derive decision trees to classify the incoming MPEG-2 MBs into one of the 11 coding modes in H.264. The proposed approach reduces the H.264 MB mode computation process into a decision tree lookup with very low complexity. The proposed transcoder is compared with a reference transcoder comprised of a MPEG-2 decoder and an H.264 encoder. Our results show that the proposed transcoder reduces the H.264 encoding time by over 95% with negligible loss in quality and bitrate.

Recent advances in video compression and 3D displays have necessitated a further understanding and development of 3D video coding algorithms. The emergence of low cost autostereoscopic displays is expected to drive the growth of 3DTV services. This paper discusses key issues that affect the quality of 3D video experience on autostereoscopic displays. The characteristics of the human visual system can be exploited to compress individual stereo views at different qualities without affecting the perceptual quality of the 3D video. The H.264/AVC video coding algorithm was used to compress each view. We examine the bounds of asymmetric stereo view compression and its relationship to eye-dominance based on a user study. This paper also presents the design and development of a modular video player with stereoscopic and multi-view capabilities including a discussion of useful tools for accelerating the development and enhancing flexibility. The experimental results indicate that eye-dominance influences 3D perception and as a result will impact the coding efficiency of 3D video.

Multimedia Broadcast Multicast Service (MBMS) is currently being standardized by the 3GPP. The MBMS services will enable new class of applications with efficient use of spectrum. MBMS also poses a new set of requirements on video coding and delivery. In this paper we present a short overview of MBMS and a video codec designed for MBMS. The proposed video codec addresses the two key issues in video delivery over MBMS: 1) error resilience and 2) resource adaptive playback. We show that the proposed codec is simple to implement on low-resource terminals such as mobile phones and outperforms MPEG-4 SP coding under lossy conditions.

The emerging H.264 video coding standard offers a flexible coding algorithm for use in a range of applications from broadcast TV to low bitrate mobile applications. The flexibility comes with complexity. The H.264 video coding algorithm is complex and requires substantial amount of resources for encoding and decoding. The resources required to decode an H.264 video varies depending upon the encoding options used. Understanding the resources required to play the video at the receiver is especially important for mobile devices with limited amount of resources. In this paper, we present an overview of the H.264 video coding standard, complexity analysis of H.264 video, and its implications for resource management and adaptive coding for mobile devices. We examine high level bitstream characteristics and its suitability for estimating the bitstream complexity and receiver resource consumption. We show that high level metrics can be used to estimate receiver resource consumption with reasonable accuracy.

The H.264 video coding standard has recently been standardized under the auspices of both the ITU and the ISO video experts groups. This new generation video standard, with its significant bandwidth savings, is expected to replace the use of MPEG-2 video compression in digital video systems. The complete migration to the new video coding algorithm will take several years given the wide scale use of MPEG-2 in the market place today. This creates an important need for transcoding technologies that transcode the widely available MPEG-2 compressed video to H.264 compressed format and vice versa. The high transcoding complexity can be reduced by reusing the information gathered in the video decoding process. However, given the significant differences between the MPEG-2 and the H.264 coding algorithms, transcoding is much more complex compared to MPEG-2 to MPEG-4 transcoding. We give a brief introduction to H.264 and discuss the issues in transcoding and complexity reduction.

Object-based representation of audio-visual (AV) presentations provides a flexible scheme to create interactive content that lends itself to resource-driven adaptation. The content adaptation needs of mobile devices can be met well with the use of object-based AV presentations. The main distinguishing feature of object-based AV presentations is the scene composition at the user terminal. In this paper, we discuss the problem of scheduling the delivery of object-based AV presentations under resource constraints. We explore the similarities with the problem of job sequencing on a single machine. We present a family of algorithms to determine the schedulability of AV presentations, and for unschedulable presentations, we present algorithms to compute a schedule that minimizes the additionally acquired resources. We present algorithms for computing incremental schedules for applications such as content authoring that require immediate feedback on resource consumption. The algorithms can be used to schedule object-based MPEG-4 presentations. We discuss the algorithms and results by considering a relatively complex MPEG-4 presentation with 16 objects, including audio, video, and images.

MPEG-4, under the auspices of the ISO, is specifying tools to enable object-based audio-visual presentations. These include tools to encode individual objects, compose presentations with objects, store these object-based presentations and access these presentations in a distributed manner over networks. The main distinguishing feature of object-based audio-visual presentations is the scene composition at the user terminal. The objects that are part of a scene are composed and displayed at the user end as opposed to encoding the composed scenes as is done in the case of MPEG-2. Such object-based representation and presentation has several benefits including compression efficiency and interaction with individual objects.

We describe the implementation of a streaming client-server system for object-based audio-visual presentations in general and MPEG-4 content in particular. The system augments the MPEG-4 demonstration software implementation (IM1) for PCs by adding network-based operation with full support for the Delivery Multimedia Integration Framework (DMIF) specification, a streaming PC-based server with DMIF support (via Xbind Inc.'s XDMIF suite), and multiplexing software. We describe XDMIF, the first reference implementation of the DMIF specification. The MPEG-4 server is designed for delivering object-based audio-visual presentation. We discuss the issues in the design and implementation of MPEG-4 servers. The system also implements a novel architecture for client-server interaction in object-based audio-visual presentations, using the mechanism of command routes and command descriptors. This new concept of command routes and command descriptors is useful in developing sophisticated interactive applications.