Cloud-based Virtual Reality (VR) gaming enables immersive experiences without the need for costly high-end consumer hardware. However, it imposes substantial bandwidth requirements due to the need to stream high-resolution, high frame rate, and stereoscopic frames to maintain immersion and prevent motion sickness. Existing techniques like foveated rendering and encoding face challenges such as reliance on costly eye-tracking hardware and sensitivity to sudden gaze shifts. In addition, prior super-resolution methods can improve fidelity but are often too computationally heavy for practical deployment. To address these challenges, we propose iSR, a system that integrates stereo-aware colorization and super-resolution to reduce transmission cost while preserving visual quality. The key idea of iSR is that it first downsamples both stereo views to reduce the total number of transmitted pixels. Then, it transmits one view in full color and the other in monochrome. This removes redundant chrominance information and further reduces the required bandwidth. On the client side, iSR reconstructs full-color, high-resolution stereo frames by transferring chroma between views and enhancing spatial resolution. Extensive experiments across multiple VR games show that iSR achieves substantial bitrate reductions while maintaining high visual fidelity. These results highlight its potential for enabling high-quality VR streaming in bandwidth-limited environments.

We present GameLab, an open-source, AI-enabled cloud gaming testbed built on WebRTC. Unlike existing open-source stacks and deployment-oriented pipelines (e.g., GamingAnywhere, Sunshine/-Moonlight, and Unity Render Streaming), GameLab is designed for AI-in-the-loop systems research: it provides programmable interfaces on both the server and client, with well-defined hook points to plug in machine learning modules on demand (e.g., super-resolution, denoising, object detection, QoE estimation, and learned rate control). GameLab also collects detailed transport and application traces for online and offline analyses of gaming sessions and network behavior. To enable reliable objective evaluation in interactive settings, GameLab embeds compact QR-based frame identifiers into the video stream, allowing accurate computation of full-reference quality metrics, such as PSNR, SSIM, and VMAF, even under frame loss, reordering, and duplication. Finally, GameLab supports GPU-based visualization of frames and model outputs for interactive inspection without costly GPU-to-CPU transfers.

Many recent video games require powerful hardware to render them. To reduce such high hardware requirements, upscalers have been proposed in the literature and industry. Upscalers save computing resources by first rendering games at lower resolutions and frame rates and then upscaling them to improve players' quality of experience. Current upscalers, however, are tightly coupled with the rendering logic of video games, which requires updating the source code of each game for every upscaler. This increases the development cost and limits the use of upscalers. The tight coupling also stifles the deployment of upscalers in cloud gaming platforms to reduce the required computing resources. We propose decoupling upscalers from game renderers, which allows utilizing various upscalers with games without changing their source code. It also accelerates deploying upscalers in cloud gaming. Decoupling upscalers from renderers is, however, challenging because of the diversity of upscalers, their dependency on information at different rendering stages, and the strict timing requirements of video games. We present an efficient solution that addresses these challenges. We implement the proposed solution and demonstrate its effectiveness with two popular upscalers. We also develop a cloud gaming system in the emerging Media-over-QUIC (MoQ) protocol and implement the proposed approach with it. Our experiments show the potential savings in computing resources while meeting the strict timing constraints of video games.

This article presents a review on the process of estimating player engagement in video gaming. To stay ahead of their competitors in entertainment, game developers need to understand, estimate, and maximize player engagement. We address the multidimensional nature of engagement, encompassing cognitive, emotional, and behavioral aspects across various gaming domains. We present a taxonomy of the diverse modalities for quantifying engagement, including physiological signals, observable behaviors, and gameplay data. We identify the challenges of conducting representative subjective studies in this domain and summarize various methods for establishing ground truth measurements. By synthesizing existing research, we provide insights into modeling techniques, highlight research gaps, and offer practical guidelines for implementing engagement measurement strategies. This review aims to aid researchers and industry professionals in navigating the complexities of player engagement estimation, ultimately contributing to enhanced game design, marketing, and user retention in the competitive gaming landscape.

Cloud gaming enables users to play games on virtually any device. This is achieved by offloading the game rendering and encoding to cloud datacenters. As game resolutions and frame rates increase, cloud gaming platforms face a major challenge to stream high quality games due to the high bandwidth and low latency requirements. In this paper, we propose a new video encoding pipeline, called DeepGame, for cloud gaming platforms to reduce the bandwidth requirements with limited to no impact on the player quality of experience. DeepGame learns the player's contextual interest in the game and the temporal correlation of that interest using a spatio-temporal deep neural network. Then, it encodes various areas in the video frames with different quality levels proportional to their contextual importance. DeepGame does not change the source code of the video encoder or the video game, and it does not require any additional hardware or software at the client side. We implemented DeepGame in an open-source cloud gaming platform and evaluated its performance using multiple popular games. We also conducted a subjective study with real players to demonstrate the potential gains achieved by DeepGame and its practicality. Our results show that DeepGame can reduce the bandwidth requirements by up to 36% compared to the baseline encoder, while maintaining the same level of perceived quality for players and running in real time.

Cloud gaming allows users with thin-clients to play complex games on their end devices as the bulk of processing is offloaded to remote servers. A thin-client is only required to have basic decoding capabilities which exist on most modern devices. The result of the remote processing is an encoded video that gets streamed to the client. As modern games are complex in terms of graphics and motion, the encoded video requires high bandwidth to provide acceptable Quality of Experience (QoE) to end users. The cost incurred by the cloud gaming service provider to stream the encoded video at such high bandwidth grows rapidly with the increase in the number of users. In this paper, we present a content-aware video encoding method for cloud gaming (referred to as CAVE) to improve the perceptual quality of the streamed video frames with comparable bandwidth requirements. This is a challenging task because of the stringent requirements on latency in cloud gaming, which impose additional restrictions on frame sizes as well as processing time to limit the total latency perceived by clients. Unlike many of the previous works, the proposed method is suitable for the state-of-the-art High Efficiency Video Coding (HEVC) encoder, which by itself offers substantial bitrate savings compared to prior encoders. The proposed method leverages information from the game such as the Regions-of-Interest (ROIs), and optimizes the quality by allocating different amounts of bits to various areas in the video frames. Through actual implementation in an open-source cloud gaming platform, we show that the proposed method achieves quality gains in ROIs that can be translated to bitrate savings between 21% and 46% against the baseline HEVC encoder and between 12% and 89% against the closest work in the literature.

A typical video encoder includes into the generated bit stream Instantaneous Decoder Refresh (IDR) units. This allows random access playback at the receiver side as well as graceful recovery from potential channel errors. Such forced IDR units typically come in repetitive patterns, which may negatively impact the perceived subjective quality if not handled properly. The reason is that the restricted encoding process of an IDR unit results in a different (regardless higher or lower) quality of reconstructed signal compared to the surrounding non-IDR ones. This causes eye-capturing irritating periodical artifacts when it occurs in patterns. This phenomenon gets to be even more pronounced when the intra refresh feature is enabled, since it forces IDR and nonIDR units to co-exist within the same picture, making the quality difference more noticeable. This paper proposes a method to hide such undesired patterns that naturally accompany the intra refresh feature. Two ideas are presented; the first one imposes restrictions that prevent unwanted fluctuations in the quantization levels between different regions of the picture, while the second hides the repetitive pattern by randomly forcing IDR blocks within specific regions of the refreshed picture. Results show that the proposed method results in improvements in subjective quality.

Due to recent demand for playback of high quality video on mobile devices, there is the need for a scalable, error resilient framework with the ability to adjust to the network and receiver's specification. To cope with the bandwidth fluctuations in the network, the framework's scalability allows for high bit-rate video to be transcoded to a low bit-rate format while preserving quality as much as possible. The new High Efficiency Video Coding (HEVC/H.265) standard allows for high compression rates, however, it is computationally intensive. We propose a novel Development Infrastructure for Video coding/transcoding Applications (DIVA). This framework is capable of providing different quality of services (e.g., Bronze, Silver, Gold), and has some error-resilient capability. Taking advantage of IBM Platform Symphony, the computationally intensive task of HEVC encoding can be distributed on available local or cloud resources. Our experiments illustrate the feasibility of this approach.

The video tool library (VTL) is one of the major normative components of the reconfigurable video coding (RVC) standard. It specifies the set of functional units (FUs) that may be interchangeably combined and connected to form different video codecs, with various compression performances and implementation complexities. In this paper, an efficient AVC baseline encoder that is described in CAL is introduced. The encoder is composed of many modules that also exist in other codecs of the same or different standards. This makes them highly reusable within the RVC framework. The main modules of the designed encoder include: Inter Prediction, Intra Prediction, and Entropy Coding. Brief descriptions of the designed modules, accompanied with CAL design issues are provided.

The main objective of the MPEG Reconfigurable Video Coding (RVC) standard is to establish a framework for a more flexible usage of standard video coding technology. The framework not only supports multiple standards and new coding configurations, but also provides an incremental and modular approach to innovation in video compression development and design. This paper provides an overview of the main objectives of RVC, standard accompanied with a presentation of the components of the framework for both normative parts and supporting tools useful for the final implementation of RVC codecs. These elements include: the Video Tool Library (VTL), the new standard RVC-CAL language used for the specification of the library, the Bitstream Syntax Description (BSD) used for the specification of the compressed bitstreams, as well as the Functional unit Network Description (FND) that constitutes the specification of a modular library. Technologies and tools that support the RVC standard are also briefly introduced.

This paper proposes to represent the floating-point multipliers required to perform IDCT implementations using a rational Diophantine (i.e. ratio of integers) approximation with a common denominator, which is not necessarily a power of two. A case study to support this proposal is presented by applying the proposed scheme to Chenpsilas IDCT algorithm. Results show better performance when applying the proposed scheme compared to the traditional shift process. Similar studies can be obtained for any other potential up-scaling factor, and by modifying any other potential IDCT fast algorithm.

This paper presents an efficient real time variable block size motion estimation (VBSME) ASIC chip, which represents a step to an H.264/AVC full encoder on chip. The proposed architecture is a SIMD architecture integrated with embedded SRAMs on one chip. The architecture has been prototyped using the TSMC 0.18 mum CMOS technology. It processes 31 CIF frames per second with 122 MHz clock frequency. It can operate at frequencies of up to 156 MHz. The prototyped architecture has 3.345 mm2core area including 2372 bytes of SRAMs and it consumes 283.96 mW @ 122MHz

The recently approved digital video standard known as H.264 promises to be an excellent video format for use with a large range of applications. Real-time encoding/decoding is a main requirement for adoption of the standard to take place in the consumer marketplace. Transformation and quantization in H.264 are relatively less complex than their correspondences in other video standards. Nevertheless, for real-time operation, a speedup is required for such processes. Especially after the recent proposal to use an 8/spl times/8 integer approximation of discrete cosine transform (DCT) to give significant compression performance at standard definition (SD) and high definition (HD) resolutions. This paper discusses a high-performance hardware implementation of the H.264 simplified 8/spl times/8 transformation and quantization. The results show that the architecture satisfies the real-time constraints required by different digital video applications.

This paper presents a hardware prototype of the H.264 transformation. The proposed architecture uses only add and shift operations to reduce the computational requirements for the 4/spl times/4 transform. The architecture is developed to be used in high-resolution applications such as high definition television (HDTV) and digital cinema. The developed architecture is prototyped and simulated using ModelSim 5.4/spl reg/. It is synthesized using Leonardo Spectrum/spl reg/. The results show that the architecture satisfies the real-time constraints required by different digital video applications.

This paper presents a VLSI prototype for the 2times2 Hadamard transform that is applied to the DC coefficients of the four 4times4 blocks of each chroma component as described in the MPEG-4 part 10 advanced video coding (AVC) standard. A VLSI prototype fir the quantization process that is accompanied with the transform operation is given as well. The implemented transform represents a level in the hierarchical transform adopted in the new AVC standard. The transform is computed using add operations only. This reduces the computational requirements of the design. The architecture is prototyped and simulated using ModelSim 5.4reg. It is synthesized using Leonardo Spectrumreg. The results show that the architecture satisfies the real-time constraints required by high definition television (HDTV)