This paper tackles the challenge of designing and implementing a complete, operational, and extensible web-based virtual reality (VR) system for stored six degrees-of-freedom (6DoF) content. We present a unique, open-source volumetric video streaming system, termed WIDE-VR, that leverages cutting-edge web technologies, including a WebGL-based rendering engine, Draco's real-time inbrowser decoder, and HTTP/3 over QUIC transport. We also explore various adaptive streaming and viewport prediction strategies to achieve improvements in bandwidth efficiency, reduced rebuffering, and minimized quality fluctuations. Our experimental results demonstrate the system's capabilities and potential in delivering immersive, high-quality VR experiences directly via web platforms, fostering broader accessibility and scalability for VR applications.

Volumetric video streaming technologies are the future of immersive media services such as virtual, augmented, and mixed-reality experiences. The challenges surrounding such technologies are tremendous due to the high network bandwidth needed to produce high-quality and low-latency streams. Many techniques and solutions have been proposed across the streaming workflow to mitigate such challenges. To better understand and organize these developments, this survey adopts an architectural framework to showcase current and emerging techniques and solutions for volumetric video streaming while highlighting some of their characteristic challenges and opportunities.

This innovative practice WIP paper describes the use of three large language model-based pre-trained AI (LLM-based AI), Inflection's Pi, Azure OpenAI GPT-3.5-Turbo and Azure OpenAI GPT-4 models to provide insightful and timely feedback across written, oral, and team-based assessment tasks in a capstone engineering design course. These LLM-based AI could analyze the content of artefacts produced in formative verbal and written tasks, ensuring that the students include relevant information in their report, and the report requirements such as structure, grammar, style, mechanics etc. are met. The models were also used to analyze the meeting transcripts of student teams, thus allowing the teamwork process and contributions from each member of the student team to be monitored closely. The integration of LLM-based pre-trained AI increased the timeliness and effectiveness of formative feedback on students' design and teamwork processes, thereby fostering a more adaptive and personalized learning experience. Any missteps or misunderstandings on the part of the students regarding the task requirements, as well as any issues arising within team interactions can be promptly communicated to the instructor for immediate resolution. While LLM-based pre-trained AI holds significant promise in transforming feedback practice, it is important to acknowledge that there are still limitations and barriers to practical implementation in the classroom. The feedback produced by LLM-based pre-trained AI lacks nuanced contextual understanding. The integration of LLM-based pre-trained AI into feedback practices holds transformative potential for education. On one hand, the models offer the promise of responsive and personalized learning, and the potential to foster critical thinking and problem-solving skills through interactive and adaptive learning platforms. Conversely, the reliability of the models as a feedback tool and students' receptions to their use remains ambiguous. We conclude the paper by proposing some strategies to overcome these limitations and support academics in applying LLM-based pre-trained AI in feedback practice.

Adaptive bitrate (ABR) schemes enable streaming clients to adapt to time-varying network/device conditions for a stall-free viewing experience. Most ABR schemes use manually tuned heuristics or learning-based methods. Heuristics are easy to implement but do not always perform well, whereas learning-based methods generally perform well but are difficult to deploy on low-resource devices. To make the most out of both worlds, we earlier developedAhaggar, a learning-based scheme executing on the server side that provides quality-aware bitrate guidance to streaming clients running their own heuristics.Ahaggar's novelty is the meta reinforcement learning approach taking network conditions, clients' statuses and device resolutions, and streamed content as input features to perform bitrate guidance.Ahaggaruses the new Common Media Client/Server Data (CMCD/SD) protocols to exchange the necessary metadata between the servers and clients. WhileAhaggarwas a significant step forward, in this study, we focus on three open areas, namely, ($i$) exploring the performance ofAhaggarin a heterogeneous environment including bothAhaggarand non-Ahaggarclients with varied network conditions and device resolutions, and ($ii$) quantifying the impact of device resolutions on QoE withAhaggar. We thoroughly investigate these areas and report our findings. We also ($iii$) discuss theAhaggardesign choices. Experiments on an open-source system show thatAhaggaradapts to unseen conditions fast and outperforms its competitors in several viewer experience metrics.

In today’s Internet, HTTP Adaptive Streaming (HAS) is the mainstream standard for video streaming, which switches the bitrate of the video content based on an Adaptive BitRate (ABR) algorithm. An effective Quality of Experience (QoE) assessment metric can provide crucial feedback to an ABR algorithm. However, predicting such real-time QoE on the client side is challenging. The QoE prediction requires high consistency with the Human Visual System (HVS), low latency, and blind assessment, which are difficult to realize together. To address this challenge, we analyzed various characteristics of HAS systems and propose a non-uniform sampling metric to reduce time complexity. Furthermore, we design an effective QoE metric that integrates resolution and rebuffering time as the Quality of Service (QoS), as well as spatiotemporal output from a deep neural network and specific switching events as content information. These reward and penalty features are regressed into quality scores with a Support Vector Regression (SVR) model. Experimental results show that the accuracy of our metric outperforms the mainstream blind QoE metrics by 0.3, and its computing time is only 60% of the video playback, indicating that the proposed metric is capable of providing real-time guidance to ABR algorithms and improving the overall performance of HAS. The QoE model is released on https://github.com/lcysyzxdxc/ASPECT.

Adaptive bitrate (ABR) schemes enable streaming clients to adapt to time-varying network/device conditions to achieve a stall-free viewing experience. Most ABR schemes use manually tuned heuristics or learning-based methods. Heuristics are easy to implement but do not always perform well, whereas learning-based methods generally perform well but are difficult to deploy on low-resource devices. To make the most out of both worlds, we develop Ahaggar, a learning-based scheme running on the server side that provides quality-aware bitrate guidance to streaming clients running their own heuristics. Ahaggar's novelty is the meta reinforcement learning approach taking network conditions, clients' statuses and device resolutions, and streamed content as input features to perform bitrate guidance. Ahaggar uses the new Common Media Client/Server Data (CMCD/SD) protocols to exchange the necessary metadata between the servers and clients. Experiments on an open-source system show that Ahaggar adapts to unseen conditions fast and outperforms its competitors in several viewer experience metrics.

We present VOLVQAD, a volumetric video quality assessment dataset consisting 7,680 ratings on 376 video sequences from 120 participants. The volumetric video sequences are first encoded with MPEG V-PCC using 4 different avatar models and 16 quality variations, and then rendered into test videos for quality assessment using 2 different background colors and 16 different quality switching patterns. The dataset is useful for researchers who wish to understand the impact of volumetric video compression on subjective quality. Analysis of the collected data are also presented in this paper.

Bandwidth prediction is critical in any Real-time Communication (RTC) service or application. This component decides how much media data can be sent in real time. Subsequently, the video and audio encoder dynamically adapts the bitrate to achieve the best quality without congesting the network and causing packets to be lost or delayed. To date, several RTC services have deployed the heuristic-based Google Congestion Control (GCC), which performs well under certain circumstances and falls short in some others. In this paper, we leverage the advancements in reinforcement learning and propose BoB (Bang-on-Bandwidth) — a hybrid bandwidth predictor for RTC. At the beginning of the RTC session, BoB uses a heuristic-based approach. It then switches to a learning-based approach. BoB predicts the available bandwidth accurately and improves bandwidth utilization under diverse network conditions compared to the two winning solutions of the ACM MMSys'21 grand challenge on bandwidth estimation in RTC. An open-source implementation of BoB is publicly available for further testing and research.

Low latency live streaming over HTTP using Dynamic Adaptive Streaming over HTTP (LL-DASH) and HTTP Live Streaming (LL- HLS) has emerged as a new way to deliver live content with an respectable video quality and short end-to-end latency. Satisfying these requirements while maintaining viewer experience in practice is challenging, and adopting conventional adaptive bitrate (ABR) schemes directly to do so will not work. Therefore, recent solutions including LoL+, L2A, Stallion, and Llama re-think conventional ABR schemes to support low-latency scenarios. These solutions have been integrated with dash.js [9] that supports LL-DASH. However, their performance in LL-HLS remains in question. To bridge this gap, we implement and integrate existing LL-DASH ABR schemes in the hls.js video player [18] which supports LL-HLS. Moreover, a series of real-world trace-driven experiments have been conducted to check their efficiency under various network conditions including a comparison with results achieved for LL-DASH in dash.js. Our version of hls.js is publicly available at [3] and a demo at [4].

Our earlier Low-on-Latency (dubbed as LoL) solution offered an accurate bandwidth prediction and rate adaptation algorithm tailored for live streaming applications that targeted an end-to-end latency of up to two seconds. While LoL was a significant step forward in multi-bitrate low-latency live streaming, further experimentation and testing showed that there was room for improvement in three areas. First, LoL used hard-coded parameters computed from an offline training process in the rate adaptation algorithm and this was seen as a significant barrier in LoL’s wide deployment. Second, LoL’s objective was to maximize a collective QoE function. Yet, certain use cases have specific objectives besides the singular QoE and this had to be accommodated. Third, the adaptive playback speed control failed to produce satisfying results in some scenarios. Our goal in this paper is to address these areas and make LoL sufficiently robust to deploy. We refer to the enhanced solution as LoL$^+$, which has been integrated to the official dash.js player in v3.2.0.

More users have a growing interest in low latency over-the-top (OTT) applications such as online video gaming, video chat, online casino, sports betting, and live auctions. OTT applications face challenges in delivering low latency live streams using Dynamic Adaptive Streaming over HTTP (DASH) due to large playback buffer and video segment duration. A potential solution to this issue is the use of HTTP chunked transfer encoding (CTE) with the common media application format (CMAF). This combination allows the delivery of each segment in several chunks to the client, starting before the segment is fully available in real-time. However, CTE and CMAF alone are not sufficient as they do not address other limitations and challenges at the client-side, including inaccurate bandwidth measurement, latency control, and bitrate selection.

In September 2020, the Consumer Technology Association (CTA) published the CTA-5004: Common Media Client Data (CMCD) specification. Using this specification, a media client can convey certain information to the content delivery network servers with object requests. This information is useful in log association/analysis, quality of service/experience monitoring and delivery enhancements. This paper is the first step toward investigating the feasibility of CMCD in addressing one of the most common problems in the streaming domain: efficient use of shared bandwidth by multiple clients. To that effect, we implemented CMCD functions on an HTTP server and built a proof-of-concept system with CMCD-aware dash.js clients. We show that even a basic bandwidth allocation scheme enabled by CMCD reduces rebuffering rate and duration without noticeably sacrificing the video quality.

Live streaming remains a challenge in the adaptive streaming space due to the stringent requirements for not just quality and rebuffering, but also latency. Many solutions have been proposed to tackle streaming in general, but only few have looked into better catering to the more challenging low-latency live streaming scenarios. In this paper, we re-visit and extend several important components (collectively called Low-on-Latency, LoL) in adaptive streaming systems to enhance the low-latency performance. LoL includes bitrate adaptation (both heuristic and learning-based), playback control and throughput measurement modules.