In this paper, we propose a multi-agent reinforcement learning based bit allocation method towards quality stability for gaming video coding in Versatile Video Coding (VVC). The bits allocated to regions-of-interests (ROI) are critical to obtain a subjectively optimal visual quality but also constrained subject to the frame-level bit budgets. A multi-objective partially observable stochastic game is formulated by combining the frame-level and ROI-level bit allocation process, which optimizes both the quality and fluctuation simultaneously. The proposed method is implemented in VVC and verified with gaming video. A multi-agent reinforcement learning method is utilized for training the agents and obtaining reasonable bit allocation actions. In comparison to the reference methods, the proposed method achieves a more consistent quality at both the frame-level and ROI-level, while improving the quality of ROI.

The Bj⊘ntegaard Delta rate (BD-rate) measurements have been used as the primary metrics to evaluate performance of video codecs. However, current BD-rate calculation methods are only applicable under the condition that the rate-distortion (R-D) values maintain a monotonic relationship, as this prerequisite is essential for computing integral along the distortion axis. To address this limitation, we propose a curve-fitting based BD-rate solution that guarantees the reconstructed R-D curve to be monotonic. Considering different use cases, we provide a four parameters logistic curve and a constraint cubic curve to approximate the underlying R-D curve. Computation of BD-rate and BD-metric using fitted R-D curve are elaborated in detail. Experimental results indicate that the proposed solutions work well on non-monotonic data. Furthermore, we verified through quantitative analysis that curve-fitting solutions provide more precise measurements of coding efficiency compared to interpolation methods. This improved accuracy contributed by the proposed methods is attributed to the higher resilience to the inherent randomness present in observed data. The proposed method has been adopted by the MPEG WG4 VCM study group for standardization activities. The source code was released at https://multimedia.tencent.com/resources/tvd.

Recently, Video Coding for Machines (VCM) has gained more and more attention due to its efforts in machine vision tasks. As a crucial track in VCM, feature compression preserves and transmits critical feature information for machine vision. Most existing studies employ dimensionality reduction to the raw multi-scale feature before compression. However, feature sparsity is left insufficiently considered in removing redundancy in compressed features. In this letter, we propose a novel framework for image feature compression for machines, where the multi-scale feature is hierarchically transformed into a sparse representation for compression. The multi-scale feature is first fused by convolutional neural networks and the attention mechanism. To introduce sparsity into the fused feature, informative channels are identified by a channel-wise binary mask where activated elements are sampled from the importance distribution of channels learned from feature content. Then, the fused feature is masked to generate a sparse representation for compression. Experiments conducted on two machine tasks show significant improvements in our model over state-of-the-art methods.

In video coding, inter prediction aims to reduce temporal redundancy by using previously encoded frames as references. The quality of reference frames is crucial to the performance of inter prediction. This paper presents a deep reference frame generation method to optimize the inter prediction in Versatile Video Coding (VVC). Specifically, reconstructed frames are sent to a well-designed frame generation network to synthesize a picture similar to the current encoding frame. The synthesized picture serves as an additional reference frame inserted into the reference picture list (RPL) to provide a more reliable reference for subsequent motion estimation (ME) and motion compensation (MC). The frame generation network employs optical flow to predict motion precisely. Moreover, an optical flow reorganization strategy is proposed to enable bi-directional and uni-directional predictions with only a single network architecture. To reasonably apply our method to VVC, we further introduce a normative modification of the temporal motion vector prediction (TMVP). Integrated into the VVC reference software VTM-15.0, the deep reference frame generation method achieves coding efficiency improvements of 5.22%, 3.61%, and 3.83% for the Y component under random access (RA), low delay B (LDB), and low delay P (LDP) configurations, respectively. The proposed method has been discussed in Joint Video Exploration Team (JVET) meeting and is currently part of Exploration Experiments (EE) for further study.

Geographic routing for connected vehicles enables vehicles and roadside infrastructure to exchange information about traffic conditions and road hazards based on their geographic positions. Its security is thus critical to traffic efficiency and road safety. In this paper, we conduct a security analysis of one standardized geographic routing protocol - GeoNetworking-and unfortunately find that its packet forwarding algorithms are vulnerable to two simple attacks. The first inter-area interception attack disturbs the victim vehicle's routing decision making and intercepts packets transmitted from one area to another. The second intra-area blockage attack intervenes packet forwarding within an area by impersonating a packet forwarder in a contention based flooding process; The attacker injects fake packets to its nearby peers and prevents vehicles within an area from receiving the broadcast packets. We use an open-source simulator to evaluate the effectiveness of proof-of-concept attacks and assess their attack damages under the settings released in public field tests. The first attack achieves an inter-area interception rate up to 99.9% (>35% in all test cases); The second attack reaches an intra-area packet blockage rate between 35% and 39%, which implies that about one-third vehicles within an area fail to receive broadcast packets. These attacks cause unnecessary traffic jams and collisions which could be avoided if GeoNetworking is properly secured. We further propose standard-compatible solutions to mitigating both attacks and conduct a preliminary evaluation to validate their effectiveness.

In this paper, we propose a deep reference frame generation method that aims to enhance bi-direction inter prediction under random access configuration in the latest video coding standard, Versatile Video Coding. Specifically, a pair of neighboring reconstructed frames are selected from decoded picture buffer and put into an optical-flow-based interpolation network to synthesize a new frame, similar to the current to-be-coded frame. Subsequently, this synthesized frame is incorporated into two-sided picture reference lists as additional reference frames. The proposed method is employed in both the encoding and decoding processes to eliminate bitstream signaling for supplementary information. The Small Ad-hoc Deep-Learning Library is utilized for implementing the proposed method. Experimental results demonstrate 3.67%/7.34%/6.51% coding efficiency improvements for Y/U/V components under the random access configuration when compared to the Versatile Video Coding NNVC reference software VTM-11_NNVC-5.0.

In video coding, bi-directional inter prediction aims to remove temporal redundancy via two-sided previously coded frames as reference. High-quality reference frames are essential to reduce the prediction residuals and improve coding efficiency performance. In this paper, we propose a deep learning-based reference frame interpolation method to enhance bi-prediction by introducing a synthetic frame to reference picture lists. Specifically, reconstructed frames are fed into a well-designed interpolation and filtering network to synthesize a picture which can be regarded as an additional reference of to-be-coded frame. Then, the picture is inserted at the appropriate place in reference picture lists to provide a more reliable reference for subsequent motion estimation and motion compensation. Experimental results show that the proposed method achieves 2.03%/6.96%/6.40% coding efficiency improvements for Y/U/V components under random access configuration, when compared with the VVC reference software VTM-15.0.

As one of the most important parts of virtual reality applications, panoramic video has become very popular. Differing from the traditional plane video, panoramic video is projected onto the 2D plane for processing, while viewed in the spherical domain. Quality of the projected plane cannot represent the real visual quality in VR viewing. In this paper, a no-reference (NR) quality assessment method is proposed for panoramic video. The proposed method extracts spatial and temporal video features from spherical domain to better model the perceived visual quality, alleviating the influence of non-uniform projection. Experiments conducted on a subjective quality database for panoramic video show that the proposed method can achieve better performance compared with the NR method designed for traditional 2D video.

In this paper, we propose a two stage optimal bit allocation scheme for HEVC hierarchical coding structure. The two stage, i.e., the frame-level and the CTU-level bit allocation, are separately conducted in the traditional rate control methods. In our proposed method, the optimal allocation in the second stage is firstly considered, and then the allocation strategy in the second stage is deemed as a foreknowledge in the first stage and applied to guide the frame-level bit allocation. With the formulation, the two stage bit allocation problem can be converted to a joint optimization problem. By solving the formulated optimization problem, the two stage optimal bit allocation scheme is established, in which more appropriate number of bits can be allocated to each frame and each CTU. The experimental results show that our proposed method can bring higher coding efficiency while satisfying the constraint of bit rate precisely.

In this paper, we propose a reinforcement learning based region of interest (ROI) bit allocation method for gaming video coding in Versatile Video Coding (VVC). Most current ROI-based bit allocation methods rely on bit budgets based on frame-level empirical weight allocation. The restricted bit budgets influence the efficiency of ROI-based bit allocation and the stability of video quality. To address this issue, the bit allocation process of frame and ROI are combined and formulated as a Markov decision process (MDP). A deep reinforcement learning (RL) method is adopted to solve this problem and obtain the appropriate bits of frame and ROI. Our target is to improve the quality of ROI and reduce the frame-level quality fluctuation, whilst satisfying the bit budgets constraint. The RL-based ROI bit allocation method is implemented in the latest video coding standard and verified for gaming video coding. The experimental results demonstrate that the proposed method achieves a better quality of ROI while reducing the quality fluctuation compared to the reference methods.

In this article, a new CTU-level bit allocation scheme aimed at subjectively optimized video coding for video conferencing applications is presented, in which the non-cooperative Stackelberg game is used for formulating and solving the bit allocation problem during the encoding process. Videos are divided into the Region of interests (ROI) which attracts people more and the non-ROI. The two regions are defined as the players in the game, where the ROI is the leader who takes the priority in strategy making and the non-ROI follows the leader's strategy. Based on the formulated game, the bit allocation problem can be expressed as a utility optimization problem. By solving the corresponding utility optimization problem, the bit allocation strategy between the ROI and the non-ROI will be established. Then the bits will be allocated to each CTU by a Newton-method-based algorithm for encoding, in which a trade-off between the ROI's quality and the overall quality can be achieved. Both the objective and subjective experimental results show that our proposed bit allocation method can improve the quality of ROI significantly with an acceptable overall quality degradation, leading to a better visual experience.

Rate control plays an important role in practical video coding. In this paper, we present a rate control method for Versatile Video Coding (VVC). We conduct some statistical analysis to investigate the influence of the skip block upon the rate-distortion parameter estimation and quality dependency among frames. Based on the analysis, we propose the rate-distortion parameter updating strategy and establish a quality dependency factor based frame-level bit allocation scheme. Experimental results show that the proposed method can achieve up to 7.31%/5.44% BD-rate improvements in Low-Delay/Random-Access configuration upon previous rate control method in VVC respectively. The presented rate control algorithm has already been adopted into VVC VTM platform by Joint Video Experts Team (JVET).

One of the key challenges in no-reference video quality assessment (NR-VQA) is the absence of the reference video to measure the similarity or difference between the distorted video and the original one. In this paper, an encoder-decoder model is proposed to predict pixel-by-pixel similarity maps from the distorted video. The model takes multiple frames as input since correlated pixels of adjacent frames can be exploited to recover the similarity map of the middle frame of the distorted video clip. In addition, to further exploit the temporal perception mechanism of the human visual system (HVS), which is relevant to the perceptual video distortion measurement, visual persistence and temporal memory effects are considered in the spatio-temporal pooling network design. Experimental results demonstrate that our proposed method outperforms state-of-the-art NR-VQA metrics.

In the hybrid video coding framework, inter prediction is an efficient tool to exploit temporal redundancy. Since the performance of inter prediction depends on the content of reference frames, coding efficiency can be significantly improved by having more effective reference frames. In this paper, we propose an enhanced inter coding scheme by generating artificial reference frames with deep neural network. Specifically, a new reference frame is interpolated from two-sided previously reconstructed frames, which can be regarded as the prediction of the to-be-coded frame. The synthesized frame is merged into reference picture list for motion estimation to further decrease the prediction residual. We integrate the proposed method into HM-16.20 under random access configuration. Experimental results show that the proposed method can significantly boost the coding performance, which provides 4.6% BD-rate reduction on average compared to HEVC baseline.

Region of Interests (ROI) coding has shown advantages in subjectively optimized video coding. In this paper, we propose a new CTU-Ievel rate allocation scheme based on the Stackelberg Game model to enhance the visual quality of ROI which formulates the rate allocation process as a noncooperative Stackelberg Game between the ROI and the non-ROI. In this game, ROI is the leader who takes the priority. Based on the formulated game, the rate allocation problem can be expressed as a utility optimization problem. By solving the corresponding utility optimization problem, a novel CTU-level rate allocation strategy is established, in which a trade-off between the ROI's quality and the overall quality can be achieved. The experimental results show that our proposed scheme can improve the quality of ROI significantly with a negligible overall quality degradation.