Real-time video encoding requires efficient and accurate quality metrics to optimize performance under strict computational and latency constraints. Traditional low-complexity metrics such as PSNR and SSIM often fall short in perceptual alignment, while accurate metrics such as VMAF are too computationally intensive for real-time deployment.We present x265-pVMAF, a low-complexity perceptual quality metric integrated into the x265 encoding loop. By leveraging machine learning and efficiently extracted encoder features, x265-pVMAF bridges the gap between computational efficiency and perceptual accuracy. It replicates VMAF predictions with a correlation of 0.99, while delivering a 37× speed-up. These results establish x265-pVMAF as a practical solution for real-time video quality assessment in next-generation encoding workflows.

With the growing use of smart cellular devices for entertainment purposes, audio and video streaming services now offer an increasingly wide variety of popular mobile applications that offer portable and accessible ways to consume content. The user interfaces of these applications have become increasingly visual in nature, and are commonly loaded with dense multimedia content such as thumbnail images, animated GIFs, and short videos. To efficiently render these and to aid rapid download to the client display, it is necessary to compress, scale and color subsample them. These operations introduce distortions, reducing the appeal of the application. It is desirable to be able to automatically monitor and govern the visual qualities of these small images, which are usually small images. However, while there exists a variety of high-performing image quality assessment (IQA) algorithms, none have been designed for this particular use case. This kind of content often has unique characteristics, such as overlaid graphics, intentional brightness, gradients, text, and warping. We describe a study we conducted on the subjective and objective quality of images embedded in the displayed user interfaces of mobile streaming applications. We created a database of typical "billboard" and "thumbnail" images viewed on such services. Using the collected data, we studied the effects of compression, scaling and chroma-subsampling on perceived quality by conducting a subjective study. We also evaluated the performance of leading picture quality prediction models on the new database. We report some surprising results regarding algorithm performance, and find that there remains ample scope for future model development.

Video compression standard H.264/AVC was released in 2003, and has been dominating the industry for the past decade. Over the last few years, a number of next-generation standards/formats like VP9 (2012), H.265/HEVC (2013) and AV1 (2018) were introduced, all claiming significant improvement over H.264/AVC. In this paper, we present our evaluation of the performance of these compression standards. Our evaluation is conducted using open-source encoder implementations of these standards, x264 (for H.264/AVC), x265 (for H.265/HEVC), libvpx (for VP9) and aomenc (for AV1). The process is designed to evaluate the attainable compression efficiency for on-demand adaptive streaming applications. Results with two different quality metrics, PSNR and VMAF, are reported. Our results reveal that x265, libvpx and aomenc all achieve substantial compression efficiency improvement over x264.

Objective image quality assessment (IQA) research entails developing algorithms that predict human judgments of picture quality. Validating performance entails evaluating algorithms under conditions similar to where they are deployed. Hence, creating image quality databases representative of target use cases is an important endeavor. Here we present a database that relates to quality assessment of billboard images commonly displayed on mobile devices. Billboard images are a subset of thumbnail images, that extend across a display screen, representing things like album covers, banners, or frames or artwork. We conducted a subjective study of the quality of billboard images distorted by processes like compression, scaling and chroma-subsampling, and compared high-performance quality prediction models on the images and subjective data.

A cloud-based encoding pipeline which generates streams for video-on-demand distribution typically processes a wide diversity of content that exhibit varying signal characteristics. To produce the best quality video streams, the system needs to adapt the encoding to each piece of content, in an automated and scalable way. In this paper, we describe two algorithm optimizations for a distributed cloud-based encoding pipeline: (i) per-title complexity analysis for bitrate-resolution selection; and (ii) per-chunk bitrate control for consistent-quality encoding. These improvements result in a number of advantages over a simple “one-size-fits-all” encoding system, including more efficient bandwidth usage and more consistent video quality.

Parallel encoding systems face the problem of distributed coding decisions, an example of which is rate allocation between different encoding chunks. Distributed encoding is traditionally performed by assigning a fixed bitrate or quality setting to each chunk, after which the chunk uses rate control to achieve that particular target. In this paper, we discuss the use of constant-slope encoding, resulting in chunks which all operate at the same quality-rate trade-off. The (Lagrangian) optimization can be performed independent of the chosen codec or distortion/quality metric. In this paper, we report results for mobile resolutions for VP9 encodes, on both open-source and Netflix catalog original and licensed content titles. BD-rate gains are obtained of 6.8 % on average, and a more constant quality across titles is observed.

In professional markets, there is a growing need to incorporate off-the-shelf consumer-level hardware to reduce costs and speed up development. Typically, these hardware components contain hardware-accelerated video encoders, which are limited to the more popular YUV4:2:0 profiles. For screen content coding (SCC) (e.g., desktop sharing, supervisory control and data acquisition applications, etc.), chroma subsampling can produce specific artifacts that are disturbing in high-quality professional applications. Current research on SCC focuses on developing specific screen content profiles and coding tools, which are not likely to be integrated in consumer electronics. In this paper, a method is explored to improve the chroma quality of SCC using YUV4:2:0 codecs. It is possible to recover parts of the chroma component based on the luma component and the subsampling filter that is used at the encoder device. A 1.63-dB peak signal-to-noise ratio-chroma improvement and 0.051 structural similarity-chroma improvement are measured on average for screen content sequences. The proposed method still provides improvements if compression is used.

High efficiency video coding (HEVC) shows a significant advance in compression efficiency and is considered to be the successor of H.264/AVC. To incorporate the HEVC standard into real-life network applications and a diversity of other applications, efficient bit rate adaptation (transrating) algorithms are required. A current problem of transrating for HEVC is the high computational complexity associated with the encoder part of such a cascaded pixel domain transcoder. This paper focuses on deriving an optimal strategy for reducing the transcoding complexity with a complexity-scalable scheme. We propose different transcoding techniques which are able to reduce the transcoding complexity in both CU and PU optimization levels. At the CU level, CUs can be evaluated in top-to-bottom or bottom-to-top flows, in which the coding information of the input video stream is utilized to reduce the number of evaluations or to early terminate certain evaluations. At the PU level, the PU candidates are adaptively selected based on the probability of PU sizes and the co-located input PU partitioning. Moreover, with the use of different proposed methods, a complexity-scalable transrating scheme can be achieved. Furthermore, the transcoding complexity can be effectively controlled by the machine learning based approach. Simulations show that the proposed techniques provide a superior transcoding performance compared to the state-of-the-art related works. Additionally, the proposed methods can achieve a range of trade-offs between transrating complexity and coding performance. From the proposed schemes, the fastest approach is able to reduce the complexity by 82% while keeping the bitrate loss below 3%.

Communication using internet and digital media is more and more popular. Therefore, the security and privacy of data transmission are highly demanded. One effective technique providing this requirement is information hiding. This technique allows to conceal secret information into a video file, an audio, or a picture. In this paper, we propose a low complexity out-of-the-loop information hiding algorithm for a video pre-encoded with the high efficiency video coding standard. Only selected components such as the motion vector difference and transform coefficients of the video are extracted and modified, bypassing the need of fully decoding and re-encoding the video. In order to reduce the propagation error caused by hiding information, the dependency between video frames is taken into account when distributing the information over the frame. Several embedding strategies are investigated. The experimental results show that the information should be hidden in smaller blocks to reduce quality loss. Using a smart distribution of information across the frames can keep the quality loss under 1 dB PSNR for an information payload of 15 kbps. When such a strategy is used, embedding information in the transform coefficients only slightly outperforms the modification of motion vector differences.

Content providers create different versions of a video to accommodate different end-user devices and network conditions. However, each of these versions requires a resource intensive encoding process. To reduce the computational complexity of the encodings, this paper proposes a fast simultaneous encoder. This encoder takes a single video as input and creates a number of bit streams encoded with different parameters. Only one version of the video is created with a full encode, whereas encoding of the other versions is accelerated by exploiting the correlation with the fully encoded version using machine learning techniques. In a practical scenario, the fast simultaneous encoder achieves a complexity reduction of 67.3% with a bit rate increase of 5.2% compared to performing a full encode of each version.

The High Efficiency Video Coding (HEVC) was developed by the Joint Collaborative Team on Video Coding (JCT-VC) to replace the current H.264/AVC standard which has been widely adopted in the last years. Therefore, there is a lot of legacy content encoded with H.264/AVC and an efficient conversion to HEVC is needed. This paper, presents a Fast Quadtree Level Decision (FQLD) algorithm that greatly reduces the complexity of the transcoding process between H.264/AVC and HEVC. The proposal tries to exploit the information gathered at the H.264/AVC decoder to make decisions on Coding Units (CU) splitting in HEVC using a Naïve-Bayes (NB) probabilistic classifier. Experimental results show that the proposed transcoder can achieve a good tradeoff between coding efficiency and complexity.

The visualization of (ultra) high-resolution compositions created from multiple input bitstreams requires several decoders at the receiving device. Therefore, not all devices can properly display such compositions. To address this problem, the input streams are decoded, merged into a single video, and re-encoded by a transcoder in the network. However, this approach requires a computationally complex re-encoding step. To reduce this complexity, information from the input bit-streams can be reused during transcoding. In HEVC, simply reusing the original encoding information is not compression efficient if the inserted content is not aligned with the grid of coded blocks. In this paper, we applied feature selection based on a decision tree, which was used in a fast HEVC transcoding model for misaligned content. The performance varies depending on the shift and average transform size in the original sequence, resulting in complexity reductions of up to 76%.

Transrating is a useful tool for adapting the bitrate of a video stream. Reducing the complexity of transrating is extremely important, especially for high efficiency video coding (HEVC) based transrating where the encoder is very time consuming. TZSearch is currently the default integer motion estimation (ME) algorithm in the HEVC reference encoder thanks to its excellent performance in reducing the complexity of ME. However, this algorithm is characterized by a fixed search area and search pattern, which can be considered sub optimal for transrating. In this paper, we improve the performance of HEVC transrating by optimizing this TZSearch algorithm. Utilizing the correlation between input and output motion vectors, we propose a fast search scheme including three steps. First, the initial search point is selected. Then, using the rate-distortion cost of this starting point, the search size is determined according to an online-trained Bayes decision rule. Finally, two proposed search algorithms are described for refining the starting point. Experimental results show that our proposed TZSearch scheme can reduce the complexity of transrating while improving the coding performance in terms of bit rate saving.

The capabilities of off-the-shelf consumer electronics are continuously increasing, making them interesting candidates to integrate into industrial visualization systems for reducing the cost of these systems or speed up development. However, in the context of video compression, hardware acceleration on consumer electronics is typically only provided for more common, consumer oriented YUV 4:2:0 profiles. Meanwhile, industrial applications often require YUV 4:4:4 formats due to synthetic visual data. Upscaling YUV 4:2:0 to YUV 4:4:4 using conventional filters results in visual artifacts, specifically for synthetic content. To improve the quality of the chroma components, this paper proposes to extend the well-known guided image filter through content-adaptive selection of the filter radius, and analyze different strategies for limiting complexity. This results in a 3.5 dB improvement of the chroma PSNR compared to conventional filters.

In an IPTV environment, different techniques exist to provide faster random access or equivalently, a faster channel switching experience, but none of them provide backward compatibility and limited overhead at the same time. In this paper, a technique is proposed to increase the random access frequency for the High Efficiency Video Coding (HEVC) standard by using a single-loop scalable version of it. This is achieved by encoding an enhancement layer with a higher frequency of random access points compared to the base layer. With this scalable configuration, a backward compatible base stream remains present. The proposed technique requires 12.4% less bitrate compared to sending the fast and slow switching streams in a non-scalable way. Compared to the almost standardized multi-loop scalable extension of HEVC, 5.5% less bitrate is needed on the core of the IPTV network and 12.5% less bitrate on the access network during steady state conditions. Moreover, the multi-loop scalable extension does not provide backward compatibility with legacy HEVC decoders.

Over the past decades, the importance of multimedia services such as video streaming has increased considerably. Since streaming protocols such as Real Time Streaming Protocol (RTSP) and Real Time Transport Protocol (RTP) require server-side bit-rate adaptation schemes, they are not ideally suited to deal with highly heterogeneous and dynamically changing network conditions. Therefore, research shifted towards client-side adaptation schemes, requiring significantly less investments in server-side infrastructure. HTTP Adaptive Streaming (HAS) is now becoming omnipresent in video streaming services due to many advantages offered by HTTP-based streaming: reliable transmission over TCP, reuse of existing caching infrastructure and compatibility with NATs and firewalls. In HAS, the video content is split temporally into segments which are encoded at different quality rates. The client side heuristic decides at which quality rate each segment should be downloaded, based on measured network statistics, buffer filling level and device characteristics. Traditionally, Advanced Video Coding (AVC) is used to encode the different segments, introducing a significant amount of redundancy across quality representations. Scalable Video Coding (SVC) can cope with these issues of content redundancy by creating dependencies between the base and enhancement layers. Adopting SVC in HAS significantly improves caching and bandwidth efficiency at the server side. Another advantage of SVC, is the ability to gradually upgrade the quality of the video by downloading additional video layers.

One of the drawbacks of current 3DTV systems are the glasses required to separate the 3D views. This problem can be solved by using autostereoscopic 3D displays, which generates a 3D scene from different viewpoints. These viewpoints are synthesized in the display based on depth information of the scene. However, currently no depth information is available for stereoscopic video. Therefore, depth estimation based on the stereoscopic input is required. This depth estimation should be low-complex and real-time for consumer electronic devices. Due to this low-complexity requirement, the quality of such depth maps will be limited. In this paper, we propose a guided depth filter, which filters the estimated depth map by using the downsampled chroma signal of the corresponding texture view as a guide. This allows to refine the borders of objects which reduces visible inpainting artifacts caused by the view synthesis. The proposed algorithm has been evaluated by expert viewing to show that it is independent of the depth map estimation and view synthesis of the autostereoscopic display.

The H.264/AVC video coding standard introduces some improved tools in order to increase compression efficiency. One of these new features is the variable block-size motion estimation. Moreover, H.264/AVC defines different prediction structures which include bi-directional predictions and, more recently, the hierarchical one. These structures also includes new Group Of Picture patterns which outperform the traditional ones. In the literature, several techniques have been proposed over the last few years which are aimed at accelerating the traditional inter prediction process, but there are no many works focusing on bidirectional and hierarchical predictions. In this paper, with the emergence of manycore processors or accelerators, a step forward is taken towards an implementation of an H.264/AVC inter prediction algorithm on a Graphics Processing Unit. The results show a negligible rate distortion drop with a time reduction of up to 92% for the complete H.264/AVC encoder.

To incorporate the newly developed High Efficiency Video Coding (HEVC) standard in real-life network applications, efficient transrating algorithms are required. We propose a fast transrating scheme, based on the early prediction of the partition split-flags in P pictures. Using machine learning techniques, the correlation between co-located partitions at different quantizations is investigated. This results in a model which predicts the split-flag and gives the associated prediction accuracy so that the splitting process in the transcoder is optimized. At each partition depth, the model indicates whether the full rate-distortion cost evaluations should be performed at the current depth, or if the partition can be split immediately. Experimental results show that the proposed transcoder reduces the complexity of the transrating process by 76.04%, while maintaining the coding efficiency of a cascaded decoder-encoder.

When middlebox devices should be able to adapt an encrypted video stream in the network without having the decryption key, format-compliant partial encryption schemes should be applied. In this paper, we propose such encryption schemes for the recently standardized High Efficiency Video Coding (HEVC) standard. By encrypting specific syntax elements like the sign of the residual information, the sign of the motion vector (MV) difference, the MV prediction index, and the MV reference index, format compliance and the possibility for adaptation are offered. Scrambling performance gradually increases when shifting from encrypting the motion information to encrypting the residual sign and finally to the combination thereof. Applying all these techniques has a negligible impact on the compression efficiency.

H.264/AVC applies an adaptive in-loop deblocking filter in order to remove blocking artifacts. The deblocking filter is adaptively controlled by using the boundary strength (BS) parameter. Although the filter achieves advantages in reducing blocking artifacts, there are two problems we should consider to improve the filtering performance. The first problem is that the filter applies just a strong and the strongest filter to filter intra coded blocks. The strong filter can remove blocking artifacts at smooth regions; however, it removes detail or sharpness of pictures at complex regions. The second problem is that H.264/AVC does not pay attention in removing corner outliers which appear when an edge of an object crosses a corner of a block. Therefore, corner outliers clearly appear and reduce the subjective quality of reconstructed pictures. In this paper, we propose an adaptive deblocking filter scheme which removes corner outliers, and which utilizes the intra prediction information of intra coded blocks to adaptively select the BS. Experimental results show that our proposed filter significantly enhances the subjective quality by removing blocking artifacts at smooth regions while maintaining details or sharpness at complex regions. In addition, the objective quality is improved, resulting in PSNR gains of up to 0.16 dB.

Distributed video coding (DVC) is a coding paradigm that entails low complexity encoding by exploiting the source statistics at the decoder. To improve the DVC coding efficiency, this paper proposes a novel adaptive technique for mode decision to control and take advantage of skip mode and intra mode in DVC. The adaptive mode decision is not only based on quality of key frames but also the rate of Wyner-Ziv (WZ) frames. To improve noise distribution estimation for a more accurate mode decision, a residual motion compensation is proposed to estimate a current noise residue based on a previously decoded frame. The experimental results show that the proposed adaptive mode decision DVC significantly improves the rate distortion performance without increasing the encoding complexity. For a GOP size of 2 on the set of test sequences, the average bitrate saving of the proposed codec is 35.5% on WZ frames compared with the DISCOVER codec.

When a video delivery system provides different representations of the same sequence, solutions can be found in simulcast coding, scalable coding, and transcoding. The major downside of transcoding is the additional complexity needed to re-encode the video sequence in its new form. When one fidelity version of the video stream should be transmitted over the network, scalable coding is less efficient compared to single layer coding because of the layering overhead. Finally, simulcast encoding results in large storage requirements. In this paper, we propose an intermediate solution providing transcoding at a low complexity by the aid of control streams. We define a control stream as a regular video stream from which residual information is removed. With these control streams, the complexity of the encoding step in the transcoder can be reduced to decoder complexity. As a result of the removal of residual information, these control streams take up 62.8% less bitrate compared to simulcast coding for a test set based on High Efficiency Video Coding (HEVC). Additionally, when compared to scalable coding, an efficient single layer video stream can be provided without the 16.6% bitrate increase caused by transmitting a layered bitstream.

Nowadays, networks and terminals with diverse characteristics of bandwidth and capabilities coexist. To ensure a good quality of experience, this diverse environment demands adaptability of the video stream. In general, video contents are compressed to save storage capacity and to reduce the bandwidth required for its transmission. Therefore, if these compressed video streams were compressed using scalable video coding schemes, they would be able to adapt to those heterogeneous networks and a wide range of terminals. Since the majority of the multimedia contents are compressed using H.264/AVC, they cannot benefit from that scalability. This paper proposes a technique to convert an H.264/AVC bitstream without scalability to a scalable bitstream with temporal scalability in Main Profile by accelerating the mode decision task of the SVC encoding stage using Machine Learning tools. The results show that when our technique is applied, the complexity is reduced by 87% while maintaining coding efficiency.

Deblocking filtering in the H.264/AVC standard is a computationally complex process because of the filter's high content adaptivity. Furthermore, the deblocking filter introduces a significant number of data dependencies, making parallel processing not obvious. Our previous works analyzed the dependencies of the filter and proposed a massively-parallel implementation, specifically tailored for execution on a single GPU. In this paper, we extend this work by proposing a parallel processing scheme for accelerating deblocking filtering using multiple CPU cores or GPUs. This scheme allows for standard-compliant filtering, regardless of slice configuration. Results show that our multi-GPU implementation using our proposed scheme achieves faster-than real-time deblocking at over 3794 frames per second for 1080p video pictures by using three GPUs. A multi-core CPU implementation using 8 CPU cores allows 1080p deblocking filtering of up to 695 frames per second.

Scalable video coding performance largely depends on the underlying single layer coding efficiency. In this paper, the quality scalability capabilities are evaluated on a base of the new High Efficiency Video Coding (HEVC) standard under development. To enable the evaluation, a multi-loop codec has been designed using HEVC. Adaptive inter-layer prediction is realized by including the lower layer in the reference list of the enhancement layer. As a result, adaptive scalability on frame level and on prediction unit level is accomplished. Compared to single layer coding, 19.4% Bjontegaard Delta bitrate increase is measured over approximately a 30dB to 40dB PSNR range. When compared to simulcast, 20.6% bitrate reduction can be achieved. Under equivalent conditions, the presented technique achieves 43.8% bitrate reduction over Coarse Grain Scalability of the SVC - H.264/AVC-based standard.

Distributed video coding (DVC) is a video coding paradigm that allows for a low-complexity encoding process by exploiting the temporal redundancies in a video sequence at the decoder side. State-of-the-art DVC systems exhibit a structural coding delay since exploiting the temporal redundancies through motion-compensated interpolation requires the frames to be decoded out of order. To alleviate this problem, we propose a system based on motion-compensated extrapolation that allows for efficient low-delay video coding with low complexity at the encoder. The proposed extrapolation technique first estimates the motion field between the two most recently decoded frames using the Lucas–Kanade algorithm. The obtained motion field is then extrapolated to the current frame using an extrapolation grid. The proposed techniques are implemented into a novel architecture featuring hybrid block-frequency Wyner–Ziv coding as well as mode decision. Results show that having references from both temporal directions in interpolation provides superior rate-distortion performance over a single temporal direction in extrapolation, as expected. However, the proposed extrapolation method is particularly suitable for low-delay coding as it performs better than H.264/AVC intra, and it is even able to outperform the interpolation-based DVC codec from DISCOVER for several sequences.

Scalable video coding (SVC) allows encoded bitstreams to be adapted. However, most bitstreams do not incorporate this scalability so bitstreams have to be adapted multiple times to accommodate for varying network conditions or end-user devices. Each adaptation incorporates an additional loss of quality due to transcoding. To overcome this issue, we propose a single transcoding step from H.264/AVC to SVC. Doing so, the resulting bitstream can be freely adapted without any additional quality reduction. Open-loop transcoding architectures can be used for H.264/AVC-to-SVC transcoding with a low complexity, although these architectures suffer from drift artifacts. Closed-loop transcoding, on the other hand, requires a higher complexity. To overcome the drawbacks of both systems, we propose combining both techniques.

Many applications require real-time decoding of high-resolution video pictures, for example, quick editing of video sequences in video editing applications. To increase decoding speed, parallelism can be exploited, yet, block-based image and video coding standards are difficult to decode in parallel because of the high number of dependencies between blocks. This paper investigates the parallel decoding capabilities of the new JPEG XR image coding standard for use on the massively-parallel architecture of the GPU. The potential of parallelism of the hierarchical frequency coding scheme used in the standard is addressed and a parallel decoding scheme is described suitable for real-time decoding of Ultra High Definition (4320p) Motion JPEG XR video sequences. Our results show a decoding speed of up to 46 frames per second for Ultra High Definition (4320p) sequences with high-dynamic range (32-bit/4:2:0) luma and chroma components.

In the current development of HEVC, compression performance improved significantly compared to H.264/AVC for both inter pictures and intra pictures. With intra compression, the main reason for this improvement is the large in crease in intra prediction directions (up to 34). The downside of having a larger number of modes is that they increase the signaling overhead in the bitstream. In this paper, a low complexity intra mode prediction algorithm is proposed which improves the mode prediction accuracy. This is achieved by exploiting the correlation between the prediction directions of the neighboring prediction units and that of the encoded prediction unit. As a result, more efficient intra mode signaling can be achieved with minimal impact on encoder and decoder complexity. On average, 0.33% bitrate improvement is obtained by employing the proposed algorithm. For sequences that are encoded with a high number of directional intra modes, around 1% bitrate improvement is measured.

The deblocking filter in the MPEG-4 AVC/H.264 standard is computationally complex because of its high content adaptivity, resulting in a significant number of data dependencies. These data dependencies interfere with parallel filtering of multiple macroblocks (MBs) on massively parallel architectures. In this letter, we introduce a novel MB partitioning scheme for concurrent deblocking in the MPEG-4 AVC/H.264 standard, based on our idea of deblocking filter independency, a corrected version of the limited error propagation effect proposed in the letter. Our proposed scheme enables concurrent MB deblocking of luma samples with limited synchronization effort, independently of slice configuration, and is compliant with the MPEG-4 H.264/AVC standard. We implemented the method on the massively parallel architecture of the graphics processing unit (GPU). Experimental results show that our GPU implementation achieves faster-than real-time deblocking at 1309 frames per second for 1080p video pictures. Both software-based deblocking filters and state-of-the-art GPU-enabled algorithms are outperformed in terms of speed by factors up to 10.2 and 19.5, respectively, for 1080p video pictures.

Scalable Video Coding (SVC) is a recent extension of the ISO/ITU Advanced Video Coding (H.264/AVC) standard, which allows adapting the bitstream easily by dropping some parts of it named as layers. This adaptation makes possible that the same bitstream meets the requirements for reliable delivery of video to diverse clients over heterogeneous networks using temporal, spatial or SNR scalability, combined or separately. Since the SVC design requires scalability to be provided at the encoder side, the existing contents cannot benefit of it. Efficient techniques for converting contents without scalability to a scalable format are desirable. In this paper, an approach for temporal scalability transcoding from H.264/AVC to SVC is presented and the impact of the GOP size is analyzed. Independently of the GOP size chosen, around a 60% of time saving is achieved while maintaining the coding efficiency. Moreover, this technique could be used to transform an H.264/AVC bitstream without temporal scalability to another H.264/AVC bitstream with hierarchical structures that provides temporal scalability.

Aiming for low-complexity encoding, distributed video coders still fail to achieve the performance of current industrial standards for video coding. One of most important problems in this area is the accurate modeling of the correlation between the predicted signal and the original video. In our previous work we showed that exploiting the quantization distortion can significantly improve the accuracy of a correlation estimator. In this paper we describe how the quantization distortion can be exploited purely at the decoder side without any performance penalty when compared to an encoder-aided system. As a result, the proposed correlation estimator delivers state-of-the-art modeling accuracy while neatly fitting the low-encoder-complexity characteristic of distributed video coding.

In this paper, we present motion-refined transcoding of H.264/AVC streams to SVC in the transform domain. By accurately taking into account both rate and distortion in the different layers on the one hand, and the SVC inter-layer motion prediction mechanisms on the other hand, the proposed transcoding architecture is able to improve rate-distortion performance over existing approaches. We propose a multilayer control mechanism that trades off performance between the different layers, resulting in 0.5 dB gains in the output SVC base layer.

Requantization transcoding is a method for reducing the bit rate of compressed video bitstreams. Most research on requantization is concerned with the architectural design, the selection of a suitable quantizer, or the reduction of requantization errors. In this paper, we propose to incorporate a new dimension in requantization transcoding: the quantization offset. We compare two requantization methods: increasing the quantization step size and decreasing the quantization offset. Furthermore, we propose a novel heuristic for requantization transcoding based on a theoretical rate-distortion analysis. The experimental results for H.264/AVC video show that requantization is improved in rate-distortion sense with gains up to 1 dB for open-loop requantization of B pictures compared to requantization with fixed quantization offset.

When combining non-rectangular (shape-adaptive) partitioning of inter pictures with a rectangular block transform, some of the transforms will be applied to a residual signal which originates from two different predictions. This is a condition that is never encountered in any of the existing video coding standards, and the impact of this on the performance of the transform coder has not been investigated. In this paper we investigate the effect of these mixed-signal blocks on the coding gain for various transforms, and we compare these against the optimal KLT gain. We find that, despite the transformed residual block being composed of two parts that were predicted from different areas of the reference picture, correlation within mixed blocks is very similar to that of normal blocks. The DCT is only marginally suboptimal w.r.t. KLT. KLT has practical issues that will reduce its coding gain or increase the signaling overhead: transform bases need to be quantized and transmitted, or a number of fixed bases needs to be chosen offline. Therefore we recommend DCT be used for all types of blocks.

The scalable extension of H.264/AVC (SVC) was recently standardized, and offers scalability at a minor penalty in rate-distortion efficiency when compared to single-layer H.264/AVC coding. In SVC, a scaled version of the original video sequence can easily be extracted by dropping layers from the stream. However, most of the video content nowadays is still produced in a single-layer format. While decoding and reencoding is a possible solution to introduce scalability in the existing bitstreams, this is an approach which requires a tremendous amount of time and effort. In this paper, we show that transcoding can be used to intelligently derive scalable bitstreams from existing single-layer streams. We focus on SNR scalability, and introduce techniques that are able to create multiple quality layers in the bitstreams. We also discuss bitstream rewriting from SVC to H.264/AVC, and examine how our newly proposed architectures can benefit from the changes that were introduced for bitstream rewriting. Architectures with different rate distribution flexibility and computational complexity are discussed. Rate-distortion performance of transcoding is shown to be comparable to that of reencoding at a fraction of the time needed for the latter.

In previous work, we introduced an H.264/AVC-to-SVC transcoder for creating SVC streams with multiple quality layers from a single-layer H.264/AVC stream. This architecture was able to restrain the error drift due to requantization of the residual coefficients. In this paper, we show that it is possible to further reduce the complexity, and completely eliminate the drift in the enhancement layer, by making use of the bitstream rewriting functionality in SVC. We propose different novel architectures, which are able to flexibly distribute the data among the different created layers.

In this paper, we present a spatial resolution reduction transcoding architecture for H.264/AVC, which extends open-loop transcoding with a low-complexity compensation technique in the reduced-resolution domain. The proposed architecture removes visual artifacts from the transcoded sequence, while keeping complexity significantly lower than more traditional cascaded decoder-encoder architectures. The refinement step of the proposed architecture can be used to further improve rate-distortion performance, at the cost of additional complexity. In this way, a dynamic-complexity transcoder is rendered possible.

In this paper, dynamic rate shaping for H.264/AVC intra-coded video streams is investigated. An analysis of the distortion distinguishes different error components that lead to the degradation of the output video stream. Experimental results show that the propagation error due to intra prediction has a major impact on the visual quality. This results from the highly increased number of dependencies which are invoked by the H.264/AVC intra prediction. In order to eliminate the propagation error, we propose to use single-loop compensation techniques. Both objective and subjective results show that the compensation techniques highly improve the visual quality.

We present a family of new macroblock partitions for H.264/AVC inter prediction. These modes allow a macroblock to be bipartitioned along a horizontal, vertical, or diagonal edge with an arbitrary location within the block. These new partitioning modes allow an encoder to adapt more closely to the local motion characteristics and improve its rate-distortion performance. In order to quantify the improvements we implemented the proposed partitioning modes in the context of the H.264/AVC video coding standard. Over half the coding efficiency of existing more general schemes is attained at a fraction of the number of added partitioning modes, and hence of encoder complexity, of those schemes. The results are consistent across different profiles, indicating that the proposed extension has good compatibility with the existing H.264/AVC tools for motion compensation and residual coding.

To attain efficient coding of sequences with complex motion activity, modern video coding standards allow variable block sizes to be employed in temporal prediction. A block with complex motion can be partitioned into two equal-sized halves or into four quadrants. In this paper we study the impact of allowing blocks to be partitioned in two unequal subpartitions. Additionally, we allow block partitioning along diagonal edges. These provisions allows encoders to better adapt to the local characteristics of the motion activity in a video sequence. We verified experimentally that the presence of partitioning edges that do not coincide with transform boundaries does not negatively impact the decorrelating strength of the residual transform, so the proposed extended partitioning strategies can be applied regardless of the details of the residual coder. Implementing the proposed extended partitioning modes in an H.264/AVC coder at the macroblock and submacroblock level, we observe that coding efficiency gains are greatest in low-resolution sequences, where moving features in the video sequence tend to be more spatially localized. For CIF and QCIF sequences we achieve a bit rate reduction of about 3–6% over a wide fidelity range.

More and more, multimedia services are being accessed via fixed and mobile networks. These services are typically much more sensitive to packet loss, delay and/or congestion than traditional services. In particular, multimedia data is often time critical and, as a result, network issues are not well tolerated and significantly deteriorate the userpsilas quality of experience (QoE). We therefore propose a QoE optimization platform that is able to mitigate problems that might occur at any location in the delivery path from service provider to customer. More specifically, the distributed architecture supports overlay routing to circumvent erratic parts of the network core. In addition, it comprises proxy components that realize last mile optimization through automatic bandwidth management and the application of processing on multimedia flows. In this paper we introduce a transcoding service for this proxy component which enables the transformation of H.264/AVC video flows to an arbitrary bit rate. Through representative experimental results, we illustrate how this addition enhances the QoE optimization capabilities of the proposed platform by allowing the proxy component to compute more flexible and effective bandwidth distributions.

In this paper, we show that it is possible to efficiently transcode single-layer H.264/AVC bitstreams to SNR-scalable SVC streams with CGS layers. Using requantization error compensation techniques, our architecture is able to restrain the drift that arises due to the absence of a closed prediction loop at the different dependency layers. Implementation results show that transcoding can generate SVC bit-streams with rate-distortion performance approaching that of the rate-distortion optimal encoder within 1 to 2 dB. Gains of more than 2 dB are obtained when compared to open-loop requantization.

Requantization transcoding is a fast technique for video bit-rate reduction. In our previous work, we proposed a hybrid architecture for the requantization of H.264/AVC bitstreams. In this architecture, the B pictures were open-loop transcoded. When B pictures are not used as reference for coding neighboring pictures, this is an acceptable approach, since the requantization errors, originating from the B pictures, can not propagate to other pictures. In this paper, however, we compare different transcoding methods in case the coded video streams are composed of hierarchical B pictures. Here, the necessity for compensating the accumulated requantization error in reference B pictures needs to be investigated. We show that extended compensation techniques result in trans-coded sequences that approach the rate-distortion optimal decoder-encoder cascade within 1 dB. Compensation of intra-coded macroblocks only is proven to be a low-complexity alternative, with only minor concessions in rate-distortion performance.