Adaptive bitrate (ABR) schemes enable streaming clients to adapt to time-varying network/device conditions for a stall-free viewing experience. Most ABR schemes use manually tuned heuristics or learning-based methods. Heuristics are easy to implement but do not always perform well, whereas learning-based methods generally perform well but are difficult to deploy on low-resource devices. To make the most out of both worlds, we earlier developedAhaggar, a learning-based scheme executing on the server side that provides quality-aware bitrate guidance to streaming clients running their own heuristics.Ahaggar's novelty is the meta reinforcement learning approach taking network conditions, clients' statuses and device resolutions, and streamed content as input features to perform bitrate guidance.Ahaggaruses the new Common Media Client/Server Data (CMCD/SD) protocols to exchange the necessary metadata between the servers and clients. WhileAhaggarwas a significant step forward, in this study, we focus on three open areas, namely, ($i$) exploring the performance ofAhaggarin a heterogeneous environment including bothAhaggarand non-Ahaggarclients with varied network conditions and device resolutions, and ($ii$) quantifying the impact of device resolutions on QoE withAhaggar. We thoroughly investigate these areas and report our findings. We also ($iii$) discuss theAhaggardesign choices. Experiments on an open-source system show thatAhaggaradapts to unseen conditions fast and outperforms its competitors in several viewer experience metrics.

Adaptive bitrate (ABR) schemes enable streaming clients to adapt to time-varying network/device conditions to achieve a stall-free viewing experience. Most ABR schemes use manually tuned heuristics or learning-based methods. Heuristics are easy to implement but do not always perform well, whereas learning-based methods generally perform well but are difficult to deploy on low-resource devices. To make the most out of both worlds, we develop Ahaggar, a learning-based scheme running on the server side that provides quality-aware bitrate guidance to streaming clients running their own heuristics. Ahaggar's novelty is the meta reinforcement learning approach taking network conditions, clients' statuses and device resolutions, and streamed content as input features to perform bitrate guidance. Ahaggar uses the new Common Media Client/Server Data (CMCD/SD) protocols to exchange the necessary metadata between the servers and clients. Experiments on an open-source system show that Ahaggar adapts to unseen conditions fast and outperforms its competitors in several viewer experience metrics.

This paper introduces the Auxiliary I-Frame Splicing method to reduce bandwidth waste in adaptive streaming. This method involves fetching a high-quality I-frame and splicing it into the already downloaded low-quality segment, resulting in a higher-quality rendering at a lower overhead than replacing the entire low-quality segment. In our experiments with three videos and four quantization parameters, the results show that the bandwidth can be saved up to 87% while still increasing the peak signal-to-noise ratio score by 20% and the video multi-method assessment fusion score by 73%. In the demo, we demonstrate the visual differences between the original and spliced videos.

Bandwidth prediction is critical in any Real-time Communication (RTC) service or application. This component decides how much media data can be sent in real time. Subsequently, the video and audio encoder dynamically adapts the bitrate to achieve the best quality without congesting the network and causing packets to be lost or delayed. To date, several RTC services have deployed the heuristic-based Google Congestion Control (GCC), which performs well under certain circumstances and falls short in some others. In this paper, we leverage the advancements in reinforcement learning and propose BoB (Bang-on-Bandwidth) — a hybrid bandwidth predictor for RTC. At the beginning of the RTC session, BoB uses a heuristic-based approach. It then switches to a learning-based approach. BoB predicts the available bandwidth accurately and improves bandwidth utilization under diverse network conditions compared to the two winning solutions of the ACM MMSys'21 grand challenge on bandwidth estimation in RTC. An open-source implementation of BoB is publicly available for further testing and research.

Omnidirectional MediA Format (OMAF) is the first worldwide virtual reality (VR) standard to store and distribute immersive media, completed in 2019. Later, in 2021, the second edition of this standard (OMAF v2) was published. The second edition kept all the features defined in the first OMAF edition while introducing some new ones, such as overlays and multi-viewpoints. OMAF v2’s Tile Index Segments that contain metadata to track fragment data per segment and quality levels create a bandwidth overhead. During the OMAF v2 standardization, multiple methods for the track fragment run representation were studied to deal with this overhead. This paper presents the implementation of one of these methods, the compressed box method using the DEFLATE algorithm (OMAF v2*). It also provides comprehensive test results of OMAF v1, OMAF v2 and OMAF v2* with various combinations of three tile grids (6x4, 8x6 and 12x8), three segment durations (300 ms, 900 ms and 3 s), two videos (RollerCoaster and Timelapse), two bitrate groups (each group with four different bitrates) and two HTTP versions (HTTP/1.1 and H2).

Our earlier Low-on-Latency (dubbed as LoL) solution offered an accurate bandwidth prediction and rate adaptation algorithm tailored for live streaming applications that targeted an end-to-end latency of up to two seconds. While LoL was a significant step forward in multi-bitrate low-latency live streaming, further experimentation and testing showed that there was room for improvement in three areas. First, LoL used hard-coded parameters computed from an offline training process in the rate adaptation algorithm and this was seen as a significant barrier in LoL’s wide deployment. Second, LoL’s objective was to maximize a collective QoE function. Yet, certain use cases have specific objectives besides the singular QoE and this had to be accommodated. Third, the adaptive playback speed control failed to produce satisfying results in some scenarios. Our goal in this paper is to address these areas and make LoL sufficiently robust to deploy. We refer to the enhanced solution as LoL$^+$, which has been integrated to the official dash.js player in v3.2.0.

Viewport-dependent delivery (VDD) is a technique to save network resources during the transmission of immersive videos. However, it results in a non-zero motion-to-high-quality delay (MTHQD), which is the delta time from the moment where the current viewport has at least one low-quality tile to when all the tiles in the new viewport are rendered in high quality. MTHQD is an important metric in the evaluation of the VDD systems. This paper improves an earlier concept called viewport margins by introducing head-motion awareness. The primary benefit of this improvement is the reduction (up to 64%) in the average MTHQD.

HTTP adaptive streaming is a technique widely used in the internet today to stream live and on-demand content. Server and client-side algorithms play an important role in achieving a better user experience in terms of metrics such as latency, rebufferings and rendering quality. In this doctoral study, we propose and evaluate a number of new algorithms for both non-immersive and immersive media in different settings ranging from low-latency live to on-demand streaming.

There are two main factors that determine the viewer experience during the live streaming of sports content: latency and stalls. Latency should be low and stalls should not occur. Yet, these two factors work against each other and it is not trivial to strike the best trade-off between them. One of the best tools we have today to manage this trade-off is the adaptive playback speed control. This tool allows the streaming client to slow down the playback when there is a risk of stalling and increase the playback when there is no risk of stalling but the live latency is higher than desired. While adaptive playback generally works well, the artifacts due to the changes in the playback speed should preferably be unnoticeable to the viewers. However, this mostly depends on the portion of the audio/video content subject to the playback speed change. In this paper, we advance the state-of-the-art by developing a content-aware playback speed control (CAPSC) algorithm and demonstrate a number of examples showing its significance. We make the running code available and provide a demo page hoping that it will be a useful tool for the developers and content providers.

In September 2020, the Consumer Technology Association (CTA) published the CTA-5004: Common Media Client Data (CMCD) specification. Using this specification, a media client can convey certain information to the content delivery network servers with object requests. This information is useful in log association/analysis, quality of service/experience monitoring and delivery enhancements. This paper is the first step toward investigating the feasibility of CMCD in addressing one of the most common problems in the streaming domain: efficient use of shared bandwidth by multiple clients. To that effect, we implemented CMCD functions on an HTTP server and built a proof-of-concept system with CMCD-aware dash.js clients. We show that even a basic bandwidth allocation scheme enabled by CMCD reduces rebuffering rate and duration without noticeably sacrificing the video quality.

In its annual developers conference in June 2019, Apple has announced a backwards-compatible extension to its popular HTTP Live Streaming (HLS) protocol to enable low-latency live streaming. This extension offers new features such as the ability to generate partial segments, use playlist delta updates, block playlist reload and provide rendition reports. Compared to the traditional HLS, these features require new capabilities on the origin servers and the caches inside a content delivery network. While HLS has been known to perform great at scale, its low-latency extension is likely to consume considerable server and network resources, and this may raise concerns about its scalability. In this paper, we make the first attempt to understand how this new extension works and performs. We also provide a 1:1 comparison against the low-latency DASH approach, which is the competing low-latency solution developed as an open standard.

Live streaming remains a challenge in the adaptive streaming space due to the stringent requirements for not just quality and rebuffering, but also latency. Many solutions have been proposed to tackle streaming in general, but only few have looked into better catering to the more challenging low-latency live streaming scenarios. In this paper, we re-visit and extend several important components (collectively called Low-on-Latency, LoL) in adaptive streaming systems to enhance the low-latency performance. LoL includes bitrate adaptation (both heuristic and learning-based), playback control and throughput measurement modules.