Recent studies on video rate control (RC) have introduced accurate and high performance methods but have not explored the optimal RC solution. The optimal solution is crucial for improving RC methods and providing labels for supervised learning. To find approximate optimal RC solutions within a limited time, we are the first to propose a reinforcement learning based search method for finding approximate optimal RC solutions within a limited time. Specifically, the RC problem for a video is first modeled as a Markov decision process (MDP). Then, with the MDP model, we develop a search method based on the deep Q-network method, which consists of exploration and exploitation steps. During exploration, an agent is created, consisting of two multilayer perceptrons and a replay memory, and trained within the MDP to estimate the value function, while superior RC solutions are recorded throughout the training process. After training, RC solution is estimated by the trained agent using the value function and a greedy strategy during the exploitation step. Finally, the approximate optimal RC solution is determined based on the RC solutions from both two steps. In addition, the time complexity of proposed method is controllable, specifically,$O(m n)$where$m$denotes the number of training epoch. Experimental results show that the bit-rate error and compression quality of the solutions found by proposed method approach the optimal solutions, with differences of only less than 0.005% and 0.399%, respectively, and are achieved in a significantly shorter time compared to the brute force search.

Since the introduction of hierarchical picture prediction structure in the advanced video coding (AVC), the hierarchical coding structure (HCS) has been widely adopted and continuously improved in video coding standards. Correspondingly, the HCS-based bit allocation methods in rate control have also emerged endlessly. Considering that pictures in higher temporal levels (TLs) of HCS usually refer to pictures in lower TLs, most methods tend to allocate more bits to pictures in lower TLs. However, these methods do not fully consider the correlation of picture quality in different TLs, which leads to the bit allocation waste and the coding performance degradation. To address this issue, we propose a fast bit allocation refinement method that can adapt to different video rate control approaches. Fig. 1 shows the overall framework of the proposed method. In general, our method is to appropriately adjust the bit allocation of pictures in lower TLs according to the relationship between the quality of pictures in different TLs. Specifically, based on the hyperbolic rate-distortion (RD) model and initial allocated bits, the quality of picture in higher TLs is first predicted and then used to estimate the quality of picture in lower TLs. Subsequently, the bits of picture in lower TLs are derived using estimated quality and its RD model. Finally, the final allocated bits of picture in lower TLs are adjusted by comparing the estimated and initial allocated bits. Experimental results show that our method can improve the coding performance of different rate control methods without introducing latency and encoding complexity.