Specific regions of interest (ROIs) within a video are of greater interest than the remainder of the video for many use cases. The Packed Regions Information (PRI) Supplemental Enhancement Information (SEI) message enables packing of rectangular ROIs from an original picture into a smaller resolution picture for video coding, reducing pixel rate and bitrate. The SEI message signals metadata describing the size and position of the ROIs in the coded picture and in the original picture. Decoders may use the metadata to reconstruct target pictures at the original resolution from the decoded pictures containing the packed regions. The PRI SEI message is under consideration for potential inclusion in a future version of the Versatile Supplemental Enhanced Information (VSEI) standard. Experimental results are provided for use of the PRI SEI message with test conditions for machine analysis of coded video content, showing reductions in bitrate and pixel rate.

This article introduces the ISO/IEC MPEG Immersive Video (MIV) standard, MPEG-I Part 12, which is undergoing standardization. The draft MIV standard provides support for viewing immersive volumetric content captured by multiple cameras with six degrees of freedom (6DoF) within a viewing space that is determined by the camera arrangement in the capture rig. The bitstream format and decoding processes of the draft specification along with aspects of the Test Model for Immersive Video (TMIV) reference software encoder, decoder, and renderer are described. The use cases, test conditions, quality assessment methods, and experimental results are provided. In the TMIV, multiple texture and geometry views are coded as atlases of patches using a legacy 2-D video codec, while optimizing for bitrate, pixel rate, and quality. The design of the bitstream format and decoder is based on the visual volumetric video-based coding (V3C) and video-based point cloud compression (V-PCC) standard, MPEG-I Part 5.

In this Special Section of the IEEE Transactions on Circuits and Systems for Video Technology, it is our honor to introduce the Versatile Video Coding (VVC) standard, the latest of the historic partnership collaborations between the International Telecommunication Union Telecommunication Standardization Sector (ITU-T), the International Organization for Standardization (ISO), and the International Electrotechnical Commission (IEC) in the field of video coding standardization.

Versatile Video Coding (VVC), a.k.a. ITU-T H.266 | ISO/IEC 23090-3, is the new generation video coding standard that has just been finalized by the Joint Video Experts Team (JVET) of ITU-T VCEG and ISO/IEC MPEG at its$19^{\mathrm {th}}$meeting ending on July 1, 2020. This paper gives an overview of the VVC high-level syntax (HLS), which forms its system and transport interface. Comparisons to the HLS designs in High Efficiency Video Coding (HEVC) and Advanced Video Coding (AVC), the previous major video coding standards, are included. When discussing new HLS features introduced into VVC or differences relative to HEVC and AVC, the reasoning behind the design differences and the benefits they bring are described. The HLS of VVC enables newer and more versatile use cases such as video region extraction, composition and merging of content from multiple coded video bitstreams, and viewport-adaptive 360° immersive media.

Multiplane Images (MPI) is a new approach for storing volumetric content. MPI represents a 3D scene within a view frustum with typically 32 planes of texture and transparency information per camera. MPI literature to date has been focused on still images but applying MPI to video will require substantial compression in order to be viable for real world productions. In this paper, we describe several techniques for compressing MPI video sequences by reducing pixel rate while maintaining acceptable visual quality. We focus on using traditional video compression codecs such as HEVC. While certainly a new codec algorithm specifically tailored to MPI would likely achieve very good results, no such devices exist today that support this hypothetical MPI codec. By comparison, hundreds of millions of real-time HEVC decoders are present in laptops and TVs today.

Standardization for digital video compression has shown significant evolution over the last three decades. Starting in 1988 with ITU-T H.261 as the first such standard that was practical for consumer use, ISO/IEC MPEG-1 and H.262/MPEG-2 video (the latter jointly standardized by ITU-T and ISO/IEC) were developed very soon thereafter, creating the first wave of broad usage of digital technology in consumer video, such as broadcast and disc player applications. Later, the H.264/MPEG-4 Advanced Video Coding (AVC) standard was again developed jointly by ITU-T and ISO/IEC experts, with its High Profile becoming dominant from 2004 in HD broadcast and storage, as well as network-based streaming services and private capture of video. With ever-increasing demands for higher quality and the advent of flat-panel displays, the H.265/MPEG-H High Efficiency Video Coding (HEVC) standard became the next generation of video compression standard; with its first version defined in 2013, HEVC has been especially instrumental for the recent deployment of Ultra High Definition (UHD, a.k.a. 4K video). As time has moved forward, video content has continued to become an increasing presence in our lives, with an ever-growing diversification of usage models and continuing demands for higher quality. For example, flat panels evolved towards support of high dynamic range (HDR) video with a wider color gamut, and new modalities for consuming video have appeared, such as head-mounted displays (HMDs).

Augmented reality (AR) and virtual reality (VR) applications have seen rising popularity in recent years. Omnidirectional 360° video is a video format often used in AR and VR applications. To address the industry needs, a new HEVC edition recently published includes several supplemental enhancement information (SEI) messages to enable the carriage of omnidirectional video using HEVC. However, further improvement in 360° video compression efficiency is needed. In order to address this challenge, the Joint Video Exploration Team (JVET) of ITU-T VCEG and ISO/IEC MPEG has been investigating 360° video coding technologies, including projection formats, pre- and post-processing technologies, as well as 360°-video-specific coding tools since 2016. The joint call for proposals (CfP) recently issued by ITU-T VCEG and ISO/IEC MPEG on video compression technologies beyond HEVC included a category on 360° video. Twelve CfP responses in the 360° video category were received. This paper describes technologies relevant to 360° video for VVC. A summary of projection formats, pre- and post-processing methods, and 360°-video specific coding tool modifications in these proposals is provided.

MPEG is developing an immersive video coding standard aims to deliver 6DoF (degrees of freedom) visual experience enabling motion parallax and omnidirectional viewing capabilities. A reference software has been made available to implement the immersive standard. To make it robust for various immersive content and deliver sharper synthesis with suppressed artifacts, a multi-pass add-on tool has been implemented for rendering at the decoder side. The tool runs view synthesis multiple times with different selection of views or patches of views in each pass based on the distance to the desired viewing position and orientation and merges the intermediate synthesized views to output a coherent and complete desired view. We share the multi-pass rendering results operating on whole views or on atlases (i.e. collection of patches from views) and compare them to the traditional single-pass synthesis results. We report significant subjective and objective improvements (1.7 ~ 4.2 dB gain in PSNR) for the multi-pass technique in the whole views case while having its synthesis results being atlas-dependent in the immersive coding case.

360-degree video is emerging as a new way of offering immersive visual experience. The quality evaluation of 360- degree video is more difficult compared to the quality evaluation of conventional video. However, to ensure successful development of 360-degree video coding technologies, it is essential to precisely measure both objective and subjective quality. In this paper, an overview of the 360-degree video quality evaluation framework established by the joint video exploration team (JVET) of ITU-T VCEG and ISO/IEC MPEG is provided. This framework aims at reproducing the different processes in the 360-degree video processing workflow that are related to coding. The results of different experiments conducted using the JVET framework are reported to illustrate the impact on objective and subjective quality with different projection formats and codecs.

The emergence of consumer level capturing and display devices for 360 degree video creates new and promising segments in entertainment, education, professional training, and other markets. In order to avoid market fragmentation and ensure interoperability of 360 degree video ecosystems, industry and academia cooperate in standardization efforts in this field. In the video coding domain, 360 degree video invalidates many established procedures, e.g., concerning evaluation of the visual quality, while the specific content characteristics offer potential for higher compression efficiency beyond the current standards. Likewise, 360 degree video puts stricter demands on the system level aspects of transmission but may also offer the potential to enhance existing transport schemes. The Joint Collaborative Team on Video Coding (JCT-VC) as well as the Joint Video Exploration Team (JVET) already started investigations into 360 degree video coding while numerous activities in the Systems subgroup of the Moving Picture Experts Group (MPEG) started to investigate application requirements and delivery aspects of 360 degree video. This paper reports on the current status of the outlined standardization efforts.

This paper provides an overview of Scalable High efficiency Video Coding (SHVC), the scalable extensions of the High Efficiency Video Coding (HEVC) standard, published in the second version of HEVC. In addition to the temporal scalability already provided by the first version of HEVC, SHVC further provides spatial, signal-to-noise ratio, bit depth, and color gamut scalability functionalities, as well as combinations of any of these. The SHVC architecture design enables SHVC implementations to be built using multiple repurposed single-layer HEVC codec cores, with the addition of interlayer reference picture processing modules. The general multilayer high-level syntax design common to all multilayer HEVC extensions, including SHVC, MV-HEVC, and 3D HEVC, is described. The interlayer reference picture processing modules, including texture and motion resampling and color mapping, are also described. Performance comparisons are provided for SHVC versus simulcast HEVC and versus the scalable video coding extension to H.264/advanced video coding.

This paper proposes changing of picture orientation on a picture-by-picture basis, without requiring the encoding of an Instantaneous Decoder Refresh (IDR) picture. It can be implemented by small modifications to H.264/MPEG-4 AVC and the design of the upcoming High Efficiency Video Coding (HEVC) standard, or most other video coding technologies employing inter picture prediction. To enable picture orientation changes, a high layer syntax element is needed to indicate the rotation to be applied to the decoded pictures. Modifications to the decoding process are also required to avoid the use of IDR pictures. For some applications with handheld cameras that may be held at different orientations, the proposed modifications allow shifting the rotation function to the decoding end, reducing overall system complexity. Additionally, selection by the encoder of the coded picture orientation may result in better coding efficiency. In this respect, an algorithm for selecting the coded picture orientation limited to intra coding is also described, and experimental results show coding efficiency improvements up to 3.4% BD-rate gain.

A scalability extension design is proposed to the upcoming High Efficiency Video Coding (HEVC) standard, to support spatial and SNR scalability. The design attempts to minimize changes needed for the enhancement layer codec from the single layer HEVC codec, re-using existing tools exactly wherever possible. Experimental results are provided for the 2 layer spatial scalability case with 1:2 ratio in each dimension, although the design can be extended to any spatial ratios and to more than 2 layers. Coding gains of up to 24.4% vs. HEVC simulcast coding are shown.

In this paper, an unequal error protection (UEP) scheme is proposed for DVB-H based on its link layer forward error correction. The scheme preserves the existing DVB-H protocol stack and maintains its original time diversity. The UEP scheme is further applied to DVB-H video streaming service. The H.264 based temporal scalable coding is employed to facilitate UEP application and provides a set of UEP configurations to accommodate different channel conditions. Experimental results show graceful video quality degradation provided by the proposed framework.

Flicker at periodic Intra-coded frames is a commonly seen video coding artifact, especially in low or medium bit rate coding. In this paper, a novel two-pass scheme is proposed to effectively remove I-frame flicker. The scheme involves a first pass of simplified P-frame coding to derive the no-flicker reference of the current frame, which is then taken as the target frame and coded with small quantization scales in the second pass of actual I-frame coding. Extensive experiment results demonstrate that our scheme successfully removes I-frame flicker, and greatly enhances the overall perceptual quality of coded video. Another contribution of this paper is a novel flicker distortion metric, which is superior than the existing metric, and can accurately measure flicker for any single frame.

This paper investigates the perceptual quality of decoded video bitstreams after packet losses. We focus on low-resolution and low-bit rate video coded by the H.264/AVC encoder, and packet loss patterns likely in 3G wireless networks. We examine the impact of several factors on the perceptual quality, including the error length (the error propagation duration after a loss), the loss severity (measured by the drop in PSNR due to a loss), and loss location (forgiveness effect). We further propose an objective quality measure based on the findings of our study.

This paper addresses the low complexity end-to-end distortion (ED) estimation problem for error resilient video coding. Unlike the existing "look-back-only" ED estimation paradigm, we propose a new hybrid paradigm involving both "look-back" and "look-ahead" estimation. For low complexity, our "look-back" estimation accurately accounts for the error propagation (EP) distortion from the last two frames only, while the impacts of "look-back" ignored frame losses are compensated by "look-ahead" frame-level EP approximation. The proposed hybrid scheme is then applied in ED-based RD optimization (ED-RDO) of both motion estimation (ME) and coding mode selection (MS). Results show that our hybrid estimation scheme yields accurate ED estimates, and when applied in ED-RDO ME and MS, significant performance gain is achieved over the other existing low complexity solutions.

In conversational video streaming applications, periodic I-frame coding is not allowed due to strict low delay requirement. For this scenario, we propose a novel hybrid paradigm, which takes advantage of both accurate pixel-level estimation and efficient block-level estimation. Specifically, error propagated (EP) distortion from the last N coded frames are exactly calculated per pixel for high accuracy, while EP effect beyond the frame limit N are estimated per macroblock for low complexity. By varying the frame limit, flexible tradeoffs between complexity and performance can be achieved, rendering our scheme especially suitable for wireless video streaming, where stringent power resource is always a challenge. Simulation results justify the effectiveness of the proposed hybrid ED estimation scheme.

In this paper, an error resilience video coding scheme is proposed based on the redundant slice feature in H.264/AVC. With the proposed scheme, the source redundancy in a coded bitstream can be adjusted flexibly according to changing channel conditions such as available bandwidth and data loss rate, without the need of complicated re-encoding or transcoding. To achieve good resilience performance, an algorithm is further developed to select a proper set of redundant slices and provide unequal error protection gain. Experimental results demonstrate its improved adaptive capability and similar performance compared to other error resilience coding scheme.

In this paper, an algorithm is proposed to improve the efficiency of flexible macroblock ordering (FMO) coding in H.264/AVC by optimal frame selection. More specifically, through a distortion-rate analysis, the algorithm selects the best frames to be coded with FMO, with the constraint of a reduced total bit rate. With the algorithm, the error resilience capability provided by the original FMO is maximally retained despite the bit rate reduction. This effectively results in a bit rate saving which can be used for other purposes. The algorithm can be combined with other error resilience techniques. From the experimental results, up to 70% reduction of the extra bit rate has been observed.

This paper proposes a new concealment-aware motion estimation and mode selection scheme for error resilient video coding, where a new distortion metric is adopted to better accounts for the impact of "worst-case" distortion on the overall perceptual video quality than the existing statistically defined end-to-end distortion metric does. Effective "motion copy" error concealment is assumed at the decoder. The proposed scheme requires no end-to-end distortion estimation, and thus, achieves low complexity. Simulation result with an H.264 codec shows that our proposed scheme improves subjective decoder reconstructed video quality, especially for sequences of medium motion.

In this paper, an error concealment algorithm is proposed to conceal an entirely lost frame in an H.264/AVC compressed video bitstream. For a lost frame, the algorithm reconstructs the lost motion information of this frame based on its temporal neighbors. The algorithm is block based and thus has low computation and implementation complexity. It is designed for those hard-to-conceal video sequences, which have abundant active and chaotic motions. Experimental results show that the algorithm compares favorably to other simple error concealment algorithms

This paper analyzes the distortion in decoded video caused by random packet losses in the underlying transmission network. A recursion model is derived that relates the average channel-induced distortion in successive P-frames. The model is applicable to all video encoders using the block-based motion-compensated prediction framework (including the H.261/263/264 and MPEG1/2/4 video coding standards) and allows for any motion-compensated temporal concealment method at the decoder. The model explicitly considers the interpolation operation invoked for motion-compensated temporal prediction and concealment with sub-pel motion vectors. The model also takes into account the two new features of the H.264/AVC standard, namely intraprediction and inloop deblocking filtering. A comparison with simulation data shows that the model is very accurate over a large range of packet loss rates and encoder intrablock rates. The model is further adapted to characterize the channel distortion in subsequent received frames after a single lost frame. This allows one to easily evaluate the impact of a single frame loss.

In this paper we present an extension of the reduced resolution update mode, originally supported by H.263, into the H.264 (MPEG-4 AVC/JVT) video coding standard. This new mode provides the opportunity of increasing the coding picture rate, while maintaining subjective quality. This is done by encoding an image at a reduced resolution, while performing prediction using a high resolution reference. This allows the final image to be reconstructed at full resolution and with good quality, although the required bitrate for encoding the image is reduced considerably. In our paper, considering that H.264 contains several new improved motion compensation tools compared to its counterpart, this concept had to be adapted accordingly to fit within the specifications of the new standard or its extensions. This includes new syntax elements, and certain semantic and encoder/decoder architecture modifications to inter and intra prediction modes.

This paper analyzes transmission-error induced distortion in decoded video. A recursion model is derived that relates the distortion in successive P-frames. The model takes into account of non-integer motion vectors used for motion-compensated temporal prediction and concealment, unconstrained intra prediction, and in-loop deblocking filtering. Experimental data show that the model is quite accurate over a large range of packet loss rates and encoder intra rates.

Video denoising is an important feature of many modem video encoding architectures since it can considerably enhance coding efficiency while at the same time improving objective and subjective quality. In this paper, a new scheme for video denoising based on spatio-temporal filtering is presented, which can be used as a preprocessing stage within a video encoder. Spatial filtering is performing through the adaptive selection and combination of a wavelet based and a 2D Wiener filter, while for temporal filtering we employ bidirectional block based motion compensation using the Enhanced Predictive Zonal Search (EPZS) algorithm. Experimental results are presented which show a significant improvement in video quality when the denoised video sequence is encoded with an H.264 encoder.

The H.264/MPEG-4 AVC video coding standard is very promising due to its high coding efficiency. In this paper, we propose a new constant bit rate control method based on the rate-distortion model of TMN8. The novelty of our approach is to use simple preprocessing to achieve the target bit rate more accurately, better target bit allocation and buffer management for both frame-level and macroblock-level rate control, improved perceptual quality, and adoption of virtual frame skipping. Simulations show that our method can meet the target bitrate very accurately, even for content with scene changes and scene transitions, and achieves both good subjective and objective quality. In addition, our approach adds little complexity to the encoder, and can therefore be used for a real-time encoder.

The JVT/H.264 video coding standard achieves considerably higher coding efficiency than previous standards. Unfortunately this comes at a cost in considerably increased complexity at the encoder mainly due to motion estimation and mode decision. In this paper, we propose a new scheme to jointly optimize motion estimation and mode decision. Simulation results show that we achieve up to 90% complexity reduction while maintaining coding efficiency.

Packet video will become a significant portion of emerging and future wireless/Internet traffic. However, network congestion and wireless channel error yields tremendous packet loss and degraded video quality. In this paper, we propose a new complete user datagram protocol (CUDP), which utilizes channel error information obtained from the physical and link layers to assist error recovery at the packet level. We propose several maximal distance separable (MDS) code-based packet level error control coding schemes and derive analytical formulas to estimate the equivalent video frame loss for different versions of user datagram protocol (UDP). We validate the proposed packet coding and CUDP protocol using MPEG-coded video under various Internet packet loss and wireless channel profiles. Theoretic and simulation results show that the video quality can be substantially improved by utilizing the frame error information at UDP and application layer.

Packet video will become a significant portion in emerging and future wireless/Internet traffic. However, network congestion and wireless channel error yields tremendous packet loss and degraded video quality. We propose a new protocol stack design, which utilizes channel error information obtained from the physical and link layer to assist the packet level error recovery. The proposed scheme is applied to MPEG coded video and tested using various packet loss and wireless channel profiles. The video quality can be substantially improved by using the channel error information at the upper protocol stack such as the UDP and application layer.

BSTWCTThis paper presents results from a study of stieaming h~EG compressed video over the pubHc Inteme& using the RTP and UDP tianspoti protocok.TmTofie minute video etips were h~EG coded at rates of 384 Kbps and 1 l~ps.The resultant coded streams were transmitted at their respective data ra$esbe~een four sites in the United States and Europa hIeasurements w'ere taken be~een sites during aHhours of the day for seveml weeks at a time to generate a clear pictire of the time varying nature of Internet errors.The paper concentrates on ne~ork loss/error characteristics that specifica~y affect the quali~of the received ~~EG compressed stieams.Due to the nature of h~EG data streams, losses in certain pati of the data stieam are more disturbing when viewed than losses in other parts of the data stieam.For instance, since N~EG video is inter-fmme coded, artifacts due to netiork loss/emors, can persist for many frames.Thus, a meaningful measure of received video qua~ty requires a more thorough analysis of netiork errors than average error rates.Our results include patterns of packet lossover tim~conditional packet 10SS Perm~

A scheme for noise reduction of image sequences by adaptively switching, on a block-by-block basis, between simple (nondisplaced) frame averaging and motion-compensated frame averaging is represented. The resulting noise reduction approaches that achievable with simple frame averaging, while maintaining the good image resolution achievable for motion compensated frame averaging.>