VR streaming and VR cloud gaming require high-resolution video streaming to provide users with high quality visual experience and maximize their interaction and immersion. Consequently, the video streams have a high bitrate and require a large amount of available bandwidth. Foveated video encoding (FVE) reduces bandwidth demand by selectively allocating higher quality to perceptually relevant regions based on human visual characteristics. However, scenes with high spatial detail or rapid motion introduce spatial and temporal frame complexities. Conventional video encoding assesses complexity and manages quality and bitrate through an internal rate-control mechanism. However, the SoTA FVE methods perform the quality allocation process after the rate control has already run. This may lead to rate violations and, consequently, under-or overutilization of available bandwidth, which in turn causes increased latency and/or reduced visual quality. In this paper, we present a real-time complexity-adaptive FVE method that minimizes computational latency using a GPU-accelerated compute shader. By prioritizing key spatial and temporal complexity variables, our weighted complexity estimation optimizes quality assignment. Our method outperforms the complexity-agnostic FVE benchmark with 44-78% greater bitrate stability, 21% lower latency, and 7% higher perceptual quality. It also surpasses the partial complexity-aware FVE benchmark, delivering 18-94% better network utilization alongside a 7% latency reduction and a 4-5% gain in quality. Our method also ensures generalizability across diverse scenarios.

VR cloud gaming promises immersive experiences, yet its realization is critically challenged by the trade-off between stringent latency requirements and high visual quality under unpredictable network conditions. Existing heuristic adaptive bitrate and foveation approaches lack adaptability to highly dynamic mobile networks. This results in a suboptimal trade-off between bandwidth usage and visual quality. While data-driven approaches (i.e., reinforcement learning, RL) have been successful in video streaming, their application to VR cloud gaming poses particular challenges. The stringent demands for high resolution and frame rate, and ultra-low latency are compounded by the necessity for fine-grained, per-frame inference to adapt to rapid changes in user gaze and network conditions. This work introduces FovRL, an RL framework for jointly optimizing foveation parameters and bitrate allocation in response to real-time network throughput. Our work pioneers the application of RL for real-time foveated encoding in immersive VR cloud gaming. Evaluations over real-world networks reveal that FovRL enhances bitrate adaptability to deliver superior perceptual visual quality, while maintaining latency comparable to the SoTA.

In cloud virtual reality (VR), delivering high perceptual visual quality under constrained wireless bandwidth remains a pivotal challenge. Foveated video encoding (FVE) copes with this by human vision-driven quality allocation, reducing bandwidth demand while preserving high visual fidelity for immersive VR streaming. However, existing FVE methods are content-independent, relying exclusively on gaze position and predefined quality degradation profiles. Thus, they fall short in modeling the intricate mechanisms of human attention. This leads to suboptimal quality distribution for visual saliency stimuli, causing perceptible artifacts and degraded perceptual quality. In this paper, we introduce a novel AI-driven streaming framework that incorporates visual saliency cues, defined as the innate capacity of scene elements to attract attention, into the video encoding pipeline. To meet the stringent low-latency demands of immersive VR, we propose a lightweight deep neural network for saliency inference, reducing computational complexity (FLOPs) by 48× compared to prior models while maintaining comparable accuracy. We integrate our pipeline into an open-source cloud VR gaming platform and conduct comprehensive experiments. Evaluation results demonstrate that our approach enhances perceptual visual quality by 22.98% compared to SoTA systems. Our IRB-approved user study shows that the saliency-guided FVE pipeline achieves superior visual quality and spatial smoothness while significantly reducing noticeable artifacts introduced by gaze-exclusive FVE. The project source code is available at https://github.com/WuZemyp/SaliencyFov.

Virtual reality (VR) cloud gaming is increasingly developing in the gaming industry. Yet, the performance of the congestion control algorithms on top of which these systems build remains under-explored. In this study, we implement two industry-standard network congestion control algorithms, Google Congestion Control (GCC) and Network-Assisted Dynamic Adaptation (NADA), according to their Requests for Comments (RFCs), and integrate them into an open-source VR gaming system (ALVR). Including ALVR's congestion control (ALVR-ABR), we conduct extensive experiments on real-world networks to evaluate each algorithm's frame latency, target-to-receiving bitrate gap, dropped frames, image quality, and fairness among heterogeneous competing flows. GCC decreases frame latency by 352ABR present significant gaps between the selected and received bitrate, causing substantial congestion-induced frame drops, while GCC has a minimal gap, resulting in minor frame drops, suggesting its suitability for game-player interaction. GCC exhibits a 2.7ABR, respectively, indicating slight immersion degradation. However, only NADA ensures a fair bandwidth share against loss-based flows due to its bitrate response to loss-induced congestion signals and lower sensitivity to delay gradients compared to GCC.

VR cloud gaming enables users to play high-end VR games on lightweight devices by offloading rendering tasks to cloud servers. Despite video compression, high-definition video streaming requires substantial data transfer rates. Foveated rendering (FR) and video encoding (FVE) leverage the non-uniform perception of the human visual system to reduce computing and bandwidth demand. They enhance visual quality in central gaze regions and reduce it in the periphery. However, bandwidth variation may hinder the provision of smooth VR gaming experiences. We present FovOptix, a system that combines FR with adaptive FVE to deliver video stream at a lower yet adaptive bitrate while not compromising the perceived video quality. FovOptix is based on a game-agnostic open-source to ensure reproducibility and compatibility with various games. We evaluate FovOptix against benchmarks using 5G mobile network traces. FovOptix achieves a latency reduction of 3% compared to the Google standard and a significant +100% reduction compared to other solutions. Additionally, it enhances the visual quality within the player's region of interest. Consequently, FovOptix attains the highest playability and gaming scores while minimizing the severity of motion sickness. FovOptix thus offers smooth and accessible VR cloud gaming for a wider range of players.