When displaying Standard Dynamic Range (SDR) video on High Dynamic Range (HDR) displays a reverse tone mapping operation can be employed to expand the dynamic range of the SDR video to that offered by the display. This paper presents a subjective performance evaluation of existing reverse Tone Mapping Operators (rTMOs). The presented study evaluates the performance of rTMOs acting on both well exposed SDR video and video that exhibits exposure variation, as is often the case with user generated content (e.g. captured on mobile phones). The paper highlights flickering artefacts that arise in such cases and evaluates the effect on perceived quality of an adapted de-flickering method.

Rate control has been regarded as an indispensable video coding tool for virtually any application involving video transmission. With the advent of many flexible tools introduced in the current state-of-the-art High Efficiency Video Coding (HEVC) standard, previous Rate-Distortion (RD) models used for rate control become insufficiently accurate. To overcome this issue, a new RD model has been recently proposed based on a robust correspondence between the rate and Lagrange multiplier λ. However, existing methods based on this model tend to perform sub-optimally after the scene change. In this paper, a two-pass rate control method is proposed, targeting Ultra High Definition Television (UHDTV) applications. In the first pass, a fast encoder with a reduced set of coding tools is used to obtain the data used for rate allocation and model parameter initialisation utilised during the second pass. Multiple encoding steps required to derive this information are avoided with the proposed variable quantization parameter framework. Experimental evaluation showed that the proposed two-pass rate control method achieves on average 4.4% BD-rate loss, compared with variable bit-rate encoding. That significantly outperforms the state-of-the-art HEVC rate control method with an average BD-rate loss of 8.8%.

The Range Extensions (RExt) of the High Efficiency Video Coding (HEVC) standard have recently been approved by both ITU-T and ISO/IEC. This set of extensions targets video coding applications in areas including content acquisition, postproduction, contribution, distribution, archiving, medical imaging, still imaging, and screen content. In addition to the functionality of HEVC Version 1, RExt provide support for monochrome, 4:2:2, and 4:4:4 chroma sampling formats as well as increased sample bit depths beyond 10 bits per sample. This extended functionality includes new coding tools with a view to provide additional coding efficiency, greater flexibility, and throughput at high bit depths/rates. Improved lossless, near-lossless, and very high bit-rate coding is also a part of the RExt scope. This paper presents the technical aspects of HEVC RExt, including a discussion of RExt profiles, tools, applications, and provides experimental results for a performance comparison with previous relevant coding technology. When compared with the High 4:4:4 Predictive Profile of H.264/Advanced Video Coding (AVC), the corresponding HEVC 4:4:4 RExt profile provides up to ~25$ %, ~32%, and ~36% average bit-rate reduction at the same PSNR quality level for intra, random access, and low delay configurations, respectively.

The human visual system (HVS) exhibits nonlinear sensitivity to the distortions introduced by lossy image and video coding. This effect is due to the luminance masking, contrast masking, and spatial and temporal frequency masking characteristics of the HVS. This paper proposes a novel perception-based quantization to remove nonvisible information in high dynamic range (HDR) color pixels by exploiting luminance masking so that the performance of the High Efficiency Video Coding (HEVC) standard is improved for HDR content. A profile scaling based on a tone-mapping curve computed for each HDR frame is introduced. The quantization step is then perceptually tuned on a transform unit basis. The proposed method has been integrated into the HEVC reference model for the HEVC range extensions (HM-RExt), and its performance was assessed by measuring the bitrate reduction against the HM-RExt. The results indicate that the proposed method achieves significant bitrate savings, up to 42.2%, with an average of 12.8%, compared with HEVC at the same quality (based on HDR-visible difference predictor-2 and subjective evaluations).

Contouring artefacts affect the visual experience of some particular types of compressed Ultra High Definition (UHD) sequences characterised by smoothly textured areas and gradual transitions in the value of the pixels. This paper proposes a technique to adjust the quantisation process at the encoder so that contouring artefacts are avoided. The devised method does not require any change at the decoder side and introduces a negligible coding rate increment (up to 3.4% for the same objective quality). This result compares favourably with the average 11.2% bit-rate penalty introduced by a method where the quantisation step is reduced in contour-prone areas.

Ultra High Definition TV (UHDTV) services are being trialled while UHD streaming services have already seen commercial débuts. The amount of data associated with these new services is very high thus extremely efficient video compression tools are required for delivery to the end user. The recently published High Efficiency Video Coding (HEVC) standard promises a new level of compression efficiency, up to 50% better than its predecessor, Advanced Video Coding (AVC). The greater efficiency in HEVC is obtained at much greater computational cost compared to AVC. A practical encoder must optimise the choice of coding tools and devise strategies to reduce the complexity without affecting the compression efficiency. This paper describes the results of a study aimed at optimising HEVC encoding for UHDTV content. The study first reviews the available HEVC coding tools to identify the best configuration before developing three new algorithms to further reduce the computational cost. The proposed optimisations can provide an additional 11.5% encoder speed-up for an average 3.1% bitrate increase on top of the best encoder configuration.

High Dynamic Range (HDR) technology (capture and display), can offer high levels of immersion through a dynamic range that meets and exceeds that of the Human Visual System (HVS). This increase in immersion comes at the cost of higher bitrate requirements, which necessitate the development of efficient HDR-relevant coding solutions. Efficient perception-based compression of HDR imagery requires models that capture accurately the various masking effects experienced by the HVS under HDR conditions, so that bits are not wasted coding redundant imperceptible information. In this paper we present two psychovisual experiments that we carried out with the aid of a high dynamic range display, in order to determine potential differences between Standard Dynamic Range (SDR) and HDR edge masking (EM) and luminance masking (LM) effects. The EM experimental results indicate that the visibility threshold is higher for the case of HDR content than SDR, especially on the dark background side of an edge. The LM experimental results suggest that the HDR visibility threshold is higher compared to SDR for both dark and bright luminance backgrounds.

With the increase of video resolutions used in multimedia applications, solutions for improved compression are sought. The next generation video compression standard, High Efficiency Video Coding (HEVC) is being developed by ITU-T and ISO/IEC MPEG with the goal to provide significant improvements over H.264/AVC. To further increase compression capabilities of HEVC, this paper proposes an additional bitrate saving mechanism. In this context, an Intensity Dependant Spatial Quantization (IDSQ) perceptual tool is proposed which exploits the intensity masking of the human visual system and perceptually adjusts quantization. The proposed IDSQ allows for adaptation to the video characteristics and its design meets low complexity implementation requirements. The proposed IDSQ has been tested in the HEVC test model and its performance is reported under “just noticeable distortion” or “perceptually lossless” conditions. In this setup, bitrate reductions of up to 25% are reported with an average of 3.4% over an exhaustive test setup.

When two video sequences are combined in each frame, an alpha channel signal is commonly used. Alpha channels are usually required in professional and studio applications for video editing and post-production processing. However, in other applications related to video broadcasting, alpha channels may be required also at the receiver side. One example is the insertion of sign language interpreters in broadcasted videos to help people with hearing difficulties to follow the programs. The interpreter may be distracting for normal hearing audiences so a solution where the video of the interpreter is switchable would be preferable. In this case, the frame composition, i.e. main program plus interpreter, is done at receiver and therefore the alpha channel needs also to be transmitted. In such an application scenario, it is important to devise efficient alpha channel coding algorithms which require low computational complexity. This paper proposes a lossless encoding algorithm which compresses homogeneous regions with few bits and involving a low amount of computational complexity for alpha channel coding. When compared the HEVC codec in lossless modality the proposed algorithm provides higher compression ratios (up to 54% improvement) and reduced computational complexity. Moreover, the proposed algorithm also outperforms conventional lossless coding techniques such as the Lempel-Ziv-Welch algorithm.

Video content containing computer generated objects is usually denoted as screen content and is becoming popular in applications such as desktop sharing, wireless displays, etc. Screen content images and videos are characterized by high frequency details such as sharp edges and high contrast image areas. On these areas classical lossy encoding tools - spatial transform plus quantization - may significantly compromise their quality and intelligibility. Therefore, lossless coding is used instead and improved coding tools should be specifically devised for screen content. In this context this paper proposes a residual differential pulse code modulation (RDPCM) applied to inter predicted residuals and tested in the context of the HEVC range extension development. The proposed method exploits the spatial correlation present in blocks containing edges or text areas which are poorly predicted by motion compensation. In addition to the baseline inter RDCPM, two improvements to the compression efficiency and the overall throughput are presented and assessed. When compared to HEVC lossless coding as specified in Version 1 of the standard, the proposed algorithm achieves up to 8% average bitrate reduction while not increasing the overall decoding complexity.

The Human Visual System (HVS) shows non-linear sensitivity to the distortion introduced by lossy image and video coding. This non-linear sensitivity is due to luminance masking, contrast masking and the spatial and temporal frequency masking phenomena of the HVS. This paper proposes a perception-based quantization method that exploits luminance masking in the HVS in order to enhance the performance of the High Efficiency Video Coding (HEVC) standard for the case of High Dynamic Range (HDR) video content. A profile scaling based on a tone-mapping curve computed for each HDR frame is introduced. The quantization step is then perceptually tuned on a Transform Unit (TU) basis. The proposed method has been integrated into the reference codec considered for the HEVC range extensions and its performance was assessed by measuring the bitrate reduction against the codec without perceptual quantization. The HDR-VDP-2 image quality metric was employed to measure the compressed picture quality. For the same quality level, an average bitrate reduction of 9% is achieved across all tested HDR sequences.

To meet the still growing compression efficiency needs for high definition content, ITU and MPEG are now defining the High Efficiency Video Coding (HEVC) standard. The HEVC codec still relies on a hybrid motion compensated predictive video coding architecture as its H.264/AVC ancestor though novel coding tools are introduced. These novel coding tools provide a highly uncorrelated prediction residue for which classical frequency decomposition methods as the discrete cosine transform may not provide an effective energy compaction. Therefore this paper proposes a transform skip mode which allows skipping one or both directions where the transform is applied. The proposed transform skip mode is integrated in the HEVC codec and is able to provide bitrate reductions of up to 5% at the same objective quality when compared with the HEVC reference codec.

The under development High Efficiency Video Coding (HEVC) standard employs several powerful coding tools to obtain improved compression efficiency regarding the state-of-the-art H.264/AVC standard. To efficiently exploit the temporal and spatial redundancies, HEVC adopts a very flexible quadtree coding structure, allowing the encoder to use a block partition better matching the image features. The best combination of HEVC block partitioning and coding modes is found by means of a rate distortion optimization process. During this minimization, the encoder tests all the possible coding modes and block partitions and keeps those providing the smallest RD cost. Due to the large number of available modes and partitions, the minimization involves extremely high complexity, which may not be suitable for real-time application scenarios. To reduce the complexity, two novel fast RDO techniques are proposed in this paper: Top Skip and Early Termination. The experimental evaluation reveals that the proposed techniques significantly reduce the encoding time, notably up to 45% regarding the high complexity HEVC reference software codec, with a negligible quality loss never larger than 0.1 dB.

A novel scheme for depth sequences compression, based on a perceptual coding algorithm, is proposed. A depth sequence describes the object position in the 3D scene, and is used, in Free Viewpoint Video, for the generation of synthetic video sequences. In perceptual video coding the human visual system characteristics are exploited to improve the compression efficiency. As depth sequences are never shown, the perceptual video coding, assessed over them, is not effective. The proposed algorithm is based on a novel perceptual rate distortion optimization process, assessed over the perceptual distortion of the rendered views generated through the encoded depth sequences. The experimental results show the effectiveness of the proposed method, able to obtain a very considerable improvement of the rendered view perceptual quality.

To face the still growing video compression needs, ITU and MPEG have jointly started a standardization project called High Efficiency Video Coding (HEVC) which aims to improve the compression efficiency of the state-of-the-art H.264/AVC standard for high and ultra high definition video. The HEVC codec still relies on the usual motion compensated, predictive, block based transform coding architecture with block sizes higher than 8×8 being now considered. Furthermore, the codec is also equipped with a Wiener in-loop filter which, together with the H.264/AVC deblocking filter, further reduces the distortion between the original and decoded frames introduced by lossy coding. While this filter allows indeed reducing the sum of square differences between the original and reconstructed frames, it is not specifically designed to reduce the ringing artifacts resulting from the use of large transform block sizes. In this context, this paper proposes to combine an adaptive bilateral filter together with the HEVC Wiener filter. The proposed combined filter allows an average bitrate reduction of about 7% regarding the HEVC codec without the Wiener filter and a 1.5% reduction regarding the HEVC codec with only the Wiener filter, always for the same quality. Moreover, the combined filter also reduces the ringing artifacts according to an objective metric specifically designed to quantify this type of coding artifact.

The High Efficiency Video Coding (HEVC) standardization project mainly aims at improving the compression efficiency beyond the state-of-the-art H.264/AVC standard for high and ultra high definition video contents. The HEVC codec reference software, still under development, encodes video content using a fixed quantization step, thus leading to a variable bitrate stream which may not be suitable for the many multimedia applications where a constant bandwidth is required. Therefore, a rate control algorithm must be integrated in the HEVC codec taking into account that the novel coding tools further reduce the bitrate associated to texture data while increase the bitrate associated to the auxiliary information, e.g. motion vectors and coding modes. In this context, this paper studies the integration of a quadratic modeling rate control algorithm in the HEVC codec. Experimental results reveal that the rate control enabled HEVC codec provides a lower rate-distortion performance regarding a fixed quantization step HEVC codec, notably spending up to 20.12% more bitrate for the same objective quality. However, the integrated rate control algorithm offers lower bitrate fluctuations along the video contents than the HEVC codec with fixed quantization step.

Although great achievements have been obtained in the past, recent developments towards ultra high definition video have been asking for further compression efficiency gains regarding the H.264/AVC state-of-the-art. As an answer to these needs, ITU-T and MPEG started a new standardization project called High Efficiency Video Coding. This paper extends and integrates a perceptual visual model in the High Efficiency Video Coding experimental codec to improve its RD performance. The designed extensions are related to the new Integer Discrete Cosine Transform block sizes as well as the new Mode Dependent Directional Transform. Moreover, an estimation mechanism for the perceptual model at the decoder side is also integrated to avoid the rate burden of sending to the decoder the model parameters. The RD performance is assessed using a methodology taking into account the variability of subjective scores which is reflected into a nonlinear sensitivity of the subjective quality metric. The RD performance obtained for the perceptually driven codec shows a bitrate reduction of up to 43% regarding the non-perceptual video codec.

In this paper, we consider a scenario where 3D scenes are modeled through a View+Depth representation. This representation is to be used at the rendering side to generate synthetic views for free viewpoint video. The encoding of both type of data (view and depth) is carried out using two H.264/AVC encoders. In this scenario we address the reduction of the encoding complexity of depth data. Firstly, an analysis of the Mode Decision and Motion Estimation processes has been conducted for both view and depth sequences, in order to capture the correlation between them. Taking advantage of this correlation, we propose a fast mode decision and motion estimation algorithm for the depth encoding. Results show that the proposed algorithm reduces the computational burden with a negligible loss in terms of quality of the rendered synthetic views. Quality measurements have been conducted using the Video Quality Metric.

The compression efficiency of the state-of-art H.264/AVC video coding standard must be improved to accommodate the compression needs of high definition videos. To this end, ITU and MPEG started a new standardization project called High Efficiency Video Coding. The video codec under development still relies on trans form domain quantization and includes the same in-loop deblocking filter adopted in the H.264/AVC standard to reduce quantization blocking artifacts. This deblocking filter provides two offsets to vary the amount of filtering for each image area. This paper proposes a perceptual optimization of these offsets based on a quality metric able to quantify the blocking artifacts impact on the perceived video quality. The proposed optimization involves low computational complexity and provides quality improvements with respect to a non-perceptually optimized H.264/AVC deblocking filter. Moreover, the proposed optimization allows up to 92% of complexity reduction regarding a brute force perceptual optimization which exhaustively tests all the possible offsets values.

Recent advances in three dimensional display technologies and the growing efforts put in the production of three dimensional videos are intensely demanding for new, more efficient three dimensional content representation formats. One promising for mat is the so-called multiview plus depth format where multiple views of the observed scene are represented together with a per pixel depth map. Naturally, this format requires to efficiently code not only the video views but also the depth data. In this context, this paper proposes a novel depth map codec encoding the depth data by means of a binary tree triangular decomposition and reconstructing the depth map values by means of a tri angle based planar approximation. For depth maps related to typical three dimensional video contents, the proposed depth map codec outperforms both the H.264/AVC standard with all intra coding modes enabled and the JPEG standard. In particular, for the same objective reconstruction quality, the proposed codec allows an average bitrate reduction of 60% and 90% regarding H.264/AVC Intra and JPEG coding, respectively.

The characteristics of the human visual system may be further exploited in lossy video coding to improve the video compression efficiency beyond the state-of-the-art H.264/AVC standard. Although the literature is rich in solutions to model the human visual system characteristics, the performance and real benefits brought by these models have not been fully integrated and assessed yet. Moreover, the rate-distortion (RD) performance is usually measured by means of methodologies that do not account for the implicit variability of the observers when rating the video quality. In this context, the novelty brought by this paper is threefold: first, it proposes novel perceptual video coding tools, notably decoder side just noticeable distortion (JND) model estimation to perceptually allocate the available rate with the finest level of granularity while avoiding the extra rate associated to coding the varying quantization steps. Second, it proposes an integrated, powerful H.264/AVC-based perceptual video coding architecture embedding a state-of-the-art JND model based on spatio-temporal human visual system masking mechanisms; this model is exploited for both the aforementioned rate allocation as well as to perceptually weight the distortion used in the motion estimation and RD optimization. Finally, it proposes a relative assessment methodology to measure the RD performance of a perceptual video codec (PVC) with respect to another codec taken as reference. The methodology considers the implicit observers variability when rating video quality which leads to a nonlinear sensitivity of the objective metrics used for quality assessment. The obtained RD performance, measured according to this methodology, shows an average bitrate reduction of up to 30% when the proposed PVC is compared with the H.264/AVC High profile at the same objective quality level. Moreover, the proposed perceptual codec outperforms an alternative perceptual codec recently published in the literature.

This paper describes a publicly available database of subjective scores, relative to quality assessment of 156 video streams encoded with H.264/AVC and corrupted by simulating packet losses over an error-prone network. The data has been collected in controlled test environments at the premises of two academic institutions. A detailed statistical analysis of subjective results has been performed, showing high consistency of the collected scores. In addition to subjective scores, we have made available to the research community both the uncompressed files and the H.264/AVC bitstreams of each video sequence, in order to provide a common database of benchmark data to test and compare the performance of full-reference, reduced-reference and no-reference video quality assessment algorithms.

This paper proposes a decoder side estimation for a just noticeable distortion perceptual model integrated in the H.264/AVC codec to improve its RD performance. The proposed estimation avoids having to send the perceptual weights used during the quantization of each frequency coefficient. Experiments with high definition videos show that the novel perceptual video codec is able to reduce the bitrate up to about 34% at the cost of, at most, 1.2% objective multi scale structural similarity quality loss. Moreover, the RD performance with the estimation is comparable to the performance achieved if the original perceptual model is used at the decoder without considering the associated rate.

The reduced-reference (RR) approximation of a full-reference (FR) video quality assessment method is a convenient way to build evaluation metrics which are both intrinsically well correlated with human judgments and feasible to implement in a network scenario, without the need to explore the perceptual significance of new video features through mean opinion score tests. In this paper, we propose a RR approximation of the video structural similarity index (VSSIM), a FR metric which is known to be well descriptive of the video quality perceived by users. We focus on the visual degradation produced by channel transmission errors: first, at the encoder, a small set of salient structural video features is assembled and transmitted through the RR channel to the end-user; then, at the decoder the feature vector is combined with a fine-granularity, no-reference estimate of the channel-induced distortion to produce the VSSIM approximation. By uniformly quantizing the feature vector and compressing it using a context-adaptive, variable length encoder, we show that good correlation coefficients with ground-truth VSSIM (rho = 0.85) may be achieved spending, respectively, less than 12 and 27 kbps for a video sequence with CIF or SD resolution.

In this paper we evaluate NORM, a no-reference video quality monitoring algorithm we proposed in a previous work, for the prediction of the subjective quality of H.264/AVC video transmitted over a noisy packet-switched network. NORM produces an estimate of the mean square error distortion at the macroblock level between the noiseless and noisy sequence, without having access to the former. The output of NORM can be readily converted into a no-reference estimate of the PSNR at the sequence level. We carried out an extensive subjective evaluation campaign on CIF and 4CIF resolution sequences encoded with H.264/AVC and transmitted over a channel that drops packets at different packet loss rates, to obtain the differential mean opinion scores. Our results show that the estimated PSNR achieves good correlation with the subjective scores, very close to the ones achieved by the PSNR computed in full-reference mode, i.e. as if the noiseless sequence would be available at the decoder.

When video is transmitted over a packet-switched network, the sequence reconstructed at the receiver side might suffer from impairments introduced by packet losses, which can only be partially healed by the action of error concealment techniques. In this context we propose NORM (NO-Reference video quality Monitoring), an algorithm to assess the quality degradation of H.264/AVC video affected by channel errors. NORM works at the receiver side where both the original and the uncorrupted video content is unavailable. We explicitly account for distortion introduced by spatial and temporal error concealment together with the effect of temporal motion-compensation. NORM provides an estimate of the mean square error distortion at the macroblock level, showing good linear correlation (correlation coefficient greater than 0.80) with the distortion computed in full-reference mode. In addition, the estimate at the macroblock level can be successfully exploited by forward quality monitoring systems that compute quality objective metrics to predict mean opinion score (MOS) values. As a proof of concept, we feed the output of NORM to a reduced-reference quality monitoring system that computes an estimate of the structural similarity metric (SSIM) score, which is known to be well correlated with perceptual quality.

This paper proposes a model for estimating, at the decoder side, the distortion induced by the transmission over an error-prone channel, when error-free reconstructed frames are not available as a reference. The proposed estimation model considers explicitly the temporal error concealment algorithm adopted at the decoder. In the evaluation of the induced distortion, we model the effects of the absence of motion vectors and prediction residuals in the decoding process. In addition, we take into account the error propagation along successive frames. Experimental results conducted over real video sequences coded with the state-of-art H.264/AVC video coding standard validate the proposed model. In fact, the distortion estimated when no reference is available is strongly correlated both at the frame and group of pictures level with the actual distortion. This technique represents an effective no-reference video quality monitoring tool that can be embedded in any H.264/AVC compliant decoder.

Channel-induced distortion estimation is an important aspect in the delivery of video contents over IP networks: the QoS requirements of both content providers and content users conflict with the intrinsic best-effort nature of packet-switched networks, which may introduce annoying artifacts in the received streams due to channel errors or jitter. In this paper we propose a Reduced-Reference video quality assessment method, based on objective quality metrics, which enables distortion estimation at the macroblock level. The content provider transmits a small feature vector for each frame, starting from random projections computed for each macroblock. In order to reduce the bit rate of the transmitted feature vector, we encode it using Distributed Source Coding (DSC) tools. The content user decodes the feature vector using the received sequence as side information. Additionally, the end-user may take advantage of some prior information about the support of the errors in the frame in such a way that the required bit length of the transmitted feature vector is further reduced. In our experiments, using 4 random projections, the use of DSC enables a bit saving of 70% w.r.t. scalar quantization and transmission of the original feature vector; when also the a priori error map is available at the decoder, the average length of the transmitted partial reference can be further reduced by another 5% of average.