Efficient representation and coding of fine-granular motion information is one of the key research areas for exploiting inter-frame correlation in video coding. Representative techniques towards this direction are affine motion compensation (AMC), decoder-side motion vector refinement (DMVR), and subblock-based temporal motion vector prediction (SbTMVP). Fine-granular motion information is derived at subblock level for all the three coding tools. In addition, the obtained inter prediction can be further refined by two optical flow-based coding tools, the bi-directional optical flow (BDOF) for bi-directional inter prediction and the prediction refinement with optical flow (PROF) exclusively used in combination with AMC. The aforementioned five coding tools have been extensively studied and finally adopted in the Versatile Video Coding (VVC) standard. This paper presents technical details of each tool and highlights the design elements with the consideration of typical hardware implementations. Following the common test conditions defined by Joint Video Experts Team (JVET) for the development of VVC, 5.7% bitrate reduction on average is achieved by the five tools. For test sequences characterized by large and complex motion, up to 13.4% bitrate reduction is observed. Additionally, visual quality improvement is demonstrated and analyzed.

Block-based partitioning is one of the fundamental techniques in video coding. Geometric-based block partitioning is a well-studied method to enable better spatial adaptation to the signal properties. This paper introduces the most recent proposal of advanced geometric-based inter prediction (GIP) made to the state-of-the-art are video coding standard - Versatile Video Coding (VVC). Implemented in the latest test model VTM-6.0 to generalize the existing triangle partition mode (TPM) and evaluated with the Joint Video Experts Team (JVET) Common Test Conditions (CTC) sequences, the proposed advanced GIP scheme provides luma BD-rate reduction of 0.56% for random access (RA) and 1.37% for low-delay (LB) test cases with 2% encoder runtime increase and negligible decoder runtime increase. Furthermore, BD-rate reductions up to 2.92% and 3.49% for RA and LB test cases can be achieved in the absence of multiple related VVC inter prediction tools.

This paper describes a new video coding tool in the Versatile Video Coding standard (VVC) named as luma mapping with chroma scaling (LMCS). Experimental compression performance results for LMCS and non-normative examples for deriving LMCS parameter values are also provided. LMCS has two main components: 1) a process for mapping input luma code values to a new set of code values for use inside the coding loop; and 2) a luma-dependent process for scaling chroma residue values. The first process, luma mapping, aims at improving the coding efficiency for standard and high dynamic range video signals by making better use of the range of luma code values allowed at a specified bit depth. The second process, chroma scaling, manages relative compression efficiency for the luma and chroma components of the video signal. The luma mapping process of LMCS is applied at the pixel sample level, and is implemented using a piecewise linear model. The chroma scaling process is applied at the chroma block level, and is implemented using a scaling factor derived from reconstructed neighboring luma samples of the chroma block.

Affine Motion Model (AMM) based inter prediction, which can represent complex motions such as zooming, rotation or shearing, has been adopted into the Versatile Video Coding (VVC) standard. AMM is defined by Control Point Motion Vectors (CPMVs) in VVC. On the other hand, Adaptive Motion Vector Resolution (AMVR) has also been adopted into VVC standard due to a favorable trade-off between the Motion Vector (MV) precision and the bit consumption on MV Differences (MVDs). However, AMVR is only applied to the Translational Motion Model (TMM), and AMM cannot benefit from it. In this paper, it is proposed to extend AMVR to AMM. Specifically, 1-pixel, 1/4-pixel and 1/16-pixel MV precisions are allowed and can be selected adaptively by each affine-inter mode coded Coding Unit (CU). Simulation results reportedly show that the proposed method can achieve 0.32% BD-rate saving on average under the Random Access configuration.

Affine motion compensation (AMC) has been adopted in the latest working draft of the Versatile Video Coding (VVC) standard jointly developed by ITU-T VCEG and ISO/IEC MPEG. The AMC in the VVC working draft is implemented as a sub-block based MC rather than pixel-based MC in order to reduce the memory access bandwidth and computation complexity, which loses prediction efficiency. The proposed algorithm in this paper is to improve the MC granularity by using pixel-based optical flow refinement. The affine prediction efficiency will be close to the pixel-based MC refinement without increasing external memory access. The experiment results show that on average the proposed algorithm can achieve 0.86% and 0.87% BD rate saving for random access and low delay B configurations, respectively, comparing to the VTM-4.0. The BD rate saving is up to 3.10%.

This paper proposes an object-level rate control algorithm to jointly controlling the bit rates of multiple video objects. Utilizing noncooperative game theory, the proposed rate control algorithm mimics the behaviors of players representing video objects. Each player competes for available bits to optimize its visual quality. The algorithm finds an ¿optimal solution¿ in that it conforms to the mixed strategy Nash equilibrium, which is the probability distribution of the actions carried by the players that maximizes their expected payoffs (the number of bits). The game is played iteratively, and the expected payoff of each play is accumulated. The game terminates when all of the available bits for the specific time instant have been distributed to video object planes (VOPs). The advantage of the proposed scheme is that the bidding objects divide the bits among themselves automatically and fairly, according to their encoding complexity, and with an overall solution that is strategically optimal under the given circumstances. To minimize buffer fluctuation and avoid buffer overflow and underflow, a proportional-integral-derivative (PID) control based buffer policy is utilized.

Motion estimation is a multistep process that encompasses techniques such as motion vector prediction, determination of search range and search patterns, and identification of termination criteria. Each of these techniques has several diversions that may suit a particular set of video characteristics. It would be hard to conceive a universal algorithm that can perform well for all kinds of video contents. However, if important characteristics of a video sequence can be identified and utilized for adjusting various steps of motion estimation, one can design an adjustable algorithm that can tune its parameters to suit the video at hand. A multistage motion estimation algorithm that includes a pre-stage for analyzing the motion characteristics of a video sequence online is proposed. This stage predicts the motion vector field (MVF) from the previous coded frame, and clusters macroblocks into background and foreground regions based on the predicted MVF. The information from the pre-stage are passed on to the next stage, which includes a mathematical model for block distortion surface to estimate the distance from the current search point to the global optimal position. This allows the motion estimation stage to adjust its search strategies accordingly. The search is performed on a precise area according to the statistical properties of the motion vector prediction error, separately for foreground and background regions. The proposed algorithm, including the pre-stage, is fast and, therefore, is suitable for online and real-time encoding. Extensive simulation results obtained for several video sequences affirm the effectiveness of the proposed algorithm. The self-tuning property enables the algorithm to perform well for several types of benchmark sequences, yielding better video quality without exceeding the time complexity as compared to the other predictive motion estimation algorithms.

Object-based video coding, such as MPEG-4, supports the encoding and manipulating of individual video object. Rate control algorithm for multiple video object coding shall adapt to the perceptual importance of the video object. For this purpose, we present a joint rate control algorithm for coding multiple video objects. The proposed algorithm is aiming to obtain superior quality for the video objects with higher perceptual importance, while squeezing bits from the video objects with lower perceptual importance. The quality gaps between video objects is regulated to the target level through the proportional-derivative feedback design.

Object based coding can potentially achieve a higher degree of compression and better visual quality. This paper proposes a multiple object rate control algorithm for video transmissions over wireless systems. The algorithm adopts an efficient bit allocation strategy to intelligently allocates more bits to important objects, while adapts to time-varying wireless channels. Simulation results for both wireless downlink and uplink channels demonstrate that the proposed algorithm effectively improves perceptual quality on important objects, significantly reduces VOP (Video Object Plane in MPEG-4) skipping and thereby, maintains the motion continuity in wireless video transmissions.

This paper presents a game theory based technique for optimizing the bit rate control in video coding. Game theory, by virtue of its enormous potential for solving constrained optimization problems, has been effectively utilized in several branches of natural and social sciences. But this paper is the first attempt in using game theory for video compression. The objective is to optimize the perceptual quality while guaranteeing "fairness" in bit allocation among macroblocks (MBs). The proposed technique is a dual-level rate control algorithm: At the first level, the algorithm allocates the target bits to frames based on their coding complexity; a method to estimate the coding complexity of the remaining frames is proposed. At the second level, MBs of a frame play cooperative games such that each MB competes for a fair share of resources (bits) to optimize its quantization scale while considering the human visual system (HVS) perceptual property. We formulate the rate control problem by defining players, strategies and objective function. Since the whole frame is an entity perceived by viewers, MBs compete cooperatively under a global objective of achieving the best quality with the given bit constraint. The major advantage of the proposed approach is that the cooperative game leads to an optimal and fair bit allocation strategy based on the Nash bargaining solution. Another advantage is that it allows multi-objective optimization with multiple decision makers (e.g., MBs) in order to achieve accurate bit rate with good perceptual quality while maintaining a stable buffer level. Several extensions of the work are possible.

Motion estimation (ME) is a multistep process that involves not one, but a combination of techniques, such as motion starting point, motion search patterns, and adaptive control to curb the search, avoidance of search stationary regions, etc. The collective efficiency of these techniques is what makes a ME algorithm robust and efficient across the board. This paper proposes a ME algorithm that is an embodiment of several effective ideas for finding the most accurate motion vectors (MVs) with the aim to maximize the encoding speed as well as the visual quality. The proposed algorithm takes advantage of the correlation between MVs in both spatial and temporal domains, controls to curb the search, avoids of search stationary regions, and uses switchable shape search patterns to accelerate motion search. The algorithm yields very similar quality compared to the full search but with several hundred times faster speed. We have evaluated the algorithm through a comprehensive performance study that shows that the proposed algorithm achieves substantial speedup without quality loss for a wide range of video sequences, compared with the ME techniques recommended by the MPEG-4 committee.

The embedded multireference frames selection with variable block-size motion compensation model drastically increases the computational complexity of the H.264/AVC video coding standard. This paper proposes an adaptive hierarchical motion estimation (ME) algorithm for H.264/AVC with the objective of minimizing the complexity while maximizing the visual quality. The proposed algorithm is based on a framework that exploits the "history" of the motion intensity from a video sequence in order to control ME. The complexity and memory requirement for this meta information is low. The algorithm determines the motion intensity of a video sequence at three levels and accordingly employs different ME techniques. The results certify that the history-based hierarchical information can be very effective in improving the efficiency of ME.

The embedded motion compensation model of the new H.264/AVC video coding standard dramatically increases the computational complexity of motion estimation. We propose a fast motion estimation algorithm using a hierarchical motion intensity structure to lower the computational complexity of the motion estimation in H.264/AVC. The proposed algorithm is mainly based on a multi-level motion intensity structure. It determines the motion intensity at three levels and, accordingly, uses different motion estimation techniques to find a more accurate, and faster, motion vector (MV). Experimental results show that the proposed algorithm provides promising performance in terms of the computational speedup and video reconstruction quality.

A multistage motion estimation scheme is proposed. The scheme extracts video characteristics by first performing an online video analysis separately for foreground and background regions. Motion parameters are extracted and passed to the next stage. The next stage includes a mathematical model for the block distortion surface (BDS) that enables the algorithm to accordingly adjust its search technique. The search is performed on a precise search area adaptive to the statistical property of the motion vector prediction error. Due to its self-tuning property, not only does the proposed scheme adapt to scenes by yielding better visual quality but it also yields a lower computational complexity, compared with the other predictive motion estimation algorithms on standard benchmark sequences.

Video scenes containing multiple objects can potentially achieve higher degree of compression and better visual quality with individual coding for each object. Video objects are not always synchronous, implying each object may have a separate temporal resolution. This paper proposes a rate control algorithm for multiple video object encoding. Using a novel bit allocation strategy, the algorithm achieves accurate target bit rate, provides good visual quality, and decreases buffer overflow/underflow. Experimental results for both synchronous and asynchronous multiple video object encoding demonstrate that, when compared with the existing rate control scheme recommended by the MPEG-4 standard, the proposed algorithm provide better temporal-spatial tradeoff with more accurate rate regulation.