International audience

In this demonstration, we present a real-time videoconferencing application built on the MOQtail open-source MOQ Transport (MOQT) protocol library. Competing with WebRTC on end-to-end latency (targeting around 100 ms with a small jitter buffer), the application uniquely offers a time-travel feature: participants can instantly rewind and play another participant's last X seconds (configurable) of content to recover context missed due to late arrivals or interruptions, all while the live meeting continues uninterrupted. This capability is achieved via a single transport in which the relay caches recent media objects for on-demand retrieval. Technically, our design leverages reliable QUIC streams to manage delivery, implementing a dynamic strategy that prioritizes low latency over perfect reliability, thereby ensuring real-time performance. This demonstration, which is released as open-source code, highlights MOQT's potential to unify real-time communications with seamless media time-shifting in a single, robust protocol stack.

This special issue presents recent advances in multimedia systems research showcased at ACM Multimedia Systems 2024 and its co-located workshops. The selected papers span adaptive and immersive video streaming, low-latency and scalable delivery architectures, and innovations in video coding and processing. Together, they illustrate the rapid progress and broad impact of emerging techniques across the multimedia stack.

Streaming systems that employ video super-resolution (SR) often rely on a single, generic neural network model for all content types, resulting in suboptimal visual quality across diverse scenes. To address this limitation, we propose a metadata-guided hot swapping mechanism that enables the dynamic selection of specialized, fine-tuned SR models during streaming. The system uses content type change signals transmitted via an auxiliary metadata track, which is prioritized over media tracks to ensure early arrival. This allows the client to preload the appropriate neural network before the content type changes, minimizing startup delay and improving responsiveness. The end-to-end workflow includes content detection through source mapping, high-priority metadata transmission, message extraction, neural network preloading and SR application.

The increasing demand for real-time communication (RTC) applications necessitates robust and reliable systems. Seamless media delivery depends on an accurate assessment of the network conditions, with bandwidth estimation (BWE) being crucial for maintaining system reliability and achieving good quality of experience (QoE) for the users. BWE poses a significant challenge due to dynamic network conditions, limited information availability and computational complexity. The Second Bandwidth Estimation Challenge, organized within ACM MMSys 2024, aims to enhance RTC user QoE by developing a deep learning-based bandwidth estimator using offline reinforcement learning. This paper presents our solution, ranked second in the grand challenge. This solution employs an actor-critic approach to achieve accurate real-time BWE by relying solely on observed network statistics. Due to the offline setting of the challenge, the critic network is trained separately from the actor network to estimate the action quality without interacting with the real environment. Furthermore, the quality prediction by the critic is adjusted by a predefined conservation factor to address overshooting the bandwidth values. The solution's source code is publicly available at https://github.com/streaming-university/FARC.

Low-Latency Dynamic Adaptive Streaming over HTTP (LL-DASH) leverages Common Media Application Format's (CMAF) chunked packaging and HTTP's chunked transfer encoding capabilities to cater to the demand for media delivery with latencies of those achieved by the traditional broadcast. Nevertheless, to achieve lower latencies in a scalable and reliable way, the Internet Engineering Task Force (IETF) recently started developing a QUIC-based solution, termed Media-over-QUIC Transport (MOQT). This paper presents the first testbed to compare LL-DASH and MOQT, comparing their performance and usability under different network conditions.

Adaptive bitrate (ABR) schemes enable streaming clients to adapt to time-varying network/device conditions for a stall-free viewing experience. Most ABR schemes use manually tuned heuristics or learning-based methods. Heuristics are easy to implement but do not always perform well, whereas learning-based methods generally perform well but are difficult to deploy on low-resource devices. To make the most out of both worlds, we earlier developedAhaggar, a learning-based scheme executing on the server side that provides quality-aware bitrate guidance to streaming clients running their own heuristics.Ahaggar's novelty is the meta reinforcement learning approach taking network conditions, clients' statuses and device resolutions, and streamed content as input features to perform bitrate guidance.Ahaggaruses the new Common Media Client/Server Data (CMCD/SD) protocols to exchange the necessary metadata between the servers and clients. WhileAhaggarwas a significant step forward, in this study, we focus on three open areas, namely, ($i$) exploring the performance ofAhaggarin a heterogeneous environment including bothAhaggarand non-Ahaggarclients with varied network conditions and device resolutions, and ($ii$) quantifying the impact of device resolutions on QoE withAhaggar. We thoroughly investigate these areas and report our findings. We also ($iii$) discuss theAhaggardesign choices. Experiments on an open-source system show thatAhaggaradapts to unseen conditions fast and outperforms its competitors in several viewer experience metrics.

Adaptive bitrate (ABR) schemes enable streaming clients to adapt to time-varying network/device conditions to achieve a stall-free viewing experience. Most ABR schemes use manually tuned heuristics or learning-based methods. Heuristics are easy to implement but do not always perform well, whereas learning-based methods generally perform well but are difficult to deploy on low-resource devices. To make the most out of both worlds, we develop Ahaggar, a learning-based scheme running on the server side that provides quality-aware bitrate guidance to streaming clients running their own heuristics. Ahaggar's novelty is the meta reinforcement learning approach taking network conditions, clients' statuses and device resolutions, and streamed content as input features to perform bitrate guidance. Ahaggar uses the new Common Media Client/Server Data (CMCD/SD) protocols to exchange the necessary metadata between the servers and clients. Experiments on an open-source system show that Ahaggar adapts to unseen conditions fast and outperforms its competitors in several viewer experience metrics.

With its advantages over TCP, QUIC created a new field for developing media-aware low-latency delivery solutions. The problem space is being examined by the new Media over QUIC (moq) working group in the IETF. In this paper, we study one of the initial proposals in detail, do a gap analysis and create an open-source testbed by introducing new essential features.

This paper introduces the Auxiliary I-Frame Splicing method to reduce bandwidth waste in adaptive streaming. This method involves fetching a high-quality I-frame and splicing it into the already downloaded low-quality segment, resulting in a higher-quality rendering at a lower overhead than replacing the entire low-quality segment. In our experiments with three videos and four quantization parameters, the results show that the bandwidth can be saved up to 87% while still increasing the peak signal-to-noise ratio score by 20% and the video multi-method assessment fusion score by 73%. In the demo, we demonstrate the visual differences between the original and spliced videos.

Bandwidth prediction is critical in any Real-time Communication (RTC) service or application. This component decides how much media data can be sent in real time. Subsequently, the video and audio encoder dynamically adapts the bitrate to achieve the best quality without congesting the network and causing packets to be lost or delayed. To date, several RTC services have deployed the heuristic-based Google Congestion Control (GCC), which performs well under certain circumstances and falls short in some others. In this paper, we leverage the advancements in reinforcement learning and propose BoB (Bang-on-Bandwidth) — a hybrid bandwidth predictor for RTC. At the beginning of the RTC session, BoB uses a heuristic-based approach. It then switches to a learning-based approach. BoB predicts the available bandwidth accurately and improves bandwidth utilization under diverse network conditions compared to the two winning solutions of the ACM MMSys'21 grand challenge on bandwidth estimation in RTC. An open-source implementation of BoB is publicly available for further testing and research.

Omnidirectional MediA Format (OMAF) is the first worldwide virtual reality (VR) standard to store and distribute immersive media, completed in 2019. Later, in 2021, the second edition of this standard (OMAF v2) was published. The second edition kept all the features defined in the first OMAF edition while introducing some new ones, such as overlays and multi-viewpoints. OMAF v2’s Tile Index Segments that contain metadata to track fragment data per segment and quality levels create a bandwidth overhead. During the OMAF v2 standardization, multiple methods for the track fragment run representation were studied to deal with this overhead. This paper presents the implementation of one of these methods, the compressed box method using the DEFLATE algorithm (OMAF v2*). It also provides comprehensive test results of OMAF v1, OMAF v2 and OMAF v2* with various combinations of three tile grids (6x4, 8x6 and 12x8), three segment durations (300 ms, 900 ms and 3 s), two videos (RollerCoaster and Timelapse), two bitrate groups (each group with four different bitrates) and two HTTP versions (HTTP/1.1 and H2).

Our earlier Low-on-Latency (dubbed as LoL) solution offered an accurate bandwidth prediction and rate adaptation algorithm tailored for live streaming applications that targeted an end-to-end latency of up to two seconds. While LoL was a significant step forward in multi-bitrate low-latency live streaming, further experimentation and testing showed that there was room for improvement in three areas. First, LoL used hard-coded parameters computed from an offline training process in the rate adaptation algorithm and this was seen as a significant barrier in LoL’s wide deployment. Second, LoL’s objective was to maximize a collective QoE function. Yet, certain use cases have specific objectives besides the singular QoE and this had to be accommodated. Third, the adaptive playback speed control failed to produce satisfying results in some scenarios. Our goal in this paper is to address these areas and make LoL sufficiently robust to deploy. We refer to the enhanced solution as LoL$^+$, which has been integrated to the official dash.js player in v3.2.0.

Viewport-dependent delivery (VDD) is a technique to save network resources during the transmission of immersive videos. However, it results in a non-zero motion-to-high-quality delay (MTHQD), which is the delta time from the moment where the current viewport has at least one low-quality tile to when all the tiles in the new viewport are rendered in high quality. MTHQD is an important metric in the evaluation of the VDD systems. This paper improves an earlier concept called viewport margins by introducing head-motion awareness. The primary benefit of this improvement is the reduction (up to 64%) in the average MTHQD.

The growing prevalence of visual disinformation has become an important problem to solve nowadays. Cheapfake is a new term used for the altered media generated by non-AI techniques. In their recent COSMOS work, the authors developed a self-supervised training strategy that detected whether different captions for a given image were out-of-context, meaning that even though pointing to the same object(s) in the image, the captions implied different meanings. In this paper, we propose four methods to improve the detection accuracy of COSMOS. These methods range from differential sensing and fake-or-fact checking that detect contradicting or fake captions to object-caption matching and threshold adjustment that modify the baseline algorithm for improved accuracy.

There are two main factors that determine the viewer experience during the live streaming of sports content: latency and stalls. Latency should be low and stalls should not occur. Yet, these two factors work against each other and it is not trivial to strike the best trade-off between them. One of the best tools we have today to manage this trade-off is the adaptive playback speed control. This tool allows the streaming client to slow down the playback when there is a risk of stalling and increase the playback when there is no risk of stalling but the live latency is higher than desired. While adaptive playback generally works well, the artifacts due to the changes in the playback speed should preferably be unnoticeable to the viewers. However, this mostly depends on the portion of the audio/video content subject to the playback speed change. In this paper, we advance the state-of-the-art by developing a content-aware playback speed control (CAPSC) algorithm and demonstrate a number of examples showing its significance. We make the running code available and provide a demo page hoping that it will be a useful tool for the developers and content providers.

In September 2020, the Consumer Technology Association (CTA) published the CTA-5004: Common Media Client Data (CMCD) specification. Using this specification, a media client can convey certain information to the content delivery network servers with object requests. This information is useful in log association/analysis, quality of service/experience monitoring and delivery enhancements. This paper is the first step toward investigating the feasibility of CMCD in addressing one of the most common problems in the streaming domain: efficient use of shared bandwidth by multiple clients. To that effect, we implemented CMCD functions on an HTTP server and built a proof-of-concept system with CMCD-aware dash.js clients. We show that even a basic bandwidth allocation scheme enabled by CMCD reduces rebuffering rate and duration without noticeably sacrificing the video quality.

Today's HTTP adaptive streaming solutions use a variety of algorithms to measure the available network bandwidth and predict its future values. Bandwidth prediction, which is already a difficult task, must be more accurate when lower latency is desired due to the shorter time available to react to bandwidth changes, and when mobile networks are involved due to their inherently more frequent and potentially larger bandwidth fluctuations. Any inaccuracy in bandwidth prediction results in flawed adaptation decisions, which will in turn translate into a diminished viewer experience. We propose an Automated Model for Prediction (AMP) that encompasses techniques for bandwidth prediction and model auto-selection specifically designed for low-latency live steaming with chunked transfer encoding. We first study statistical and computational intelligence techniques to implement a suite of bandwidth prediction models that can work accurately under a broad range of network conditions, and second, we introduce an automated prediction model selection method. We confirm the effectiveness of our solution through trace-driven live streaming experiments.

In its annual developers conference in June 2019, Apple has announced a backwards-compatible extension to its popular HTTP Live Streaming (HLS) protocol to enable low-latency live streaming. This extension offers new features such as the ability to generate partial segments, use playlist delta updates, block playlist reload and provide rendition reports. Compared to the traditional HLS, these features require new capabilities on the origin servers and the caches inside a content delivery network. While HLS has been known to perform great at scale, its low-latency extension is likely to consume considerable server and network resources, and this may raise concerns about its scalability. In this paper, we make the first attempt to understand how this new extension works and performs. We also provide a 1:1 comparison against the low-latency DASH approach, which is the competing low-latency solution developed as an open standard.

Enhancing user experience in streaming applications is an important problem. Delivering the best quality possible for the given network conditions is not an easy task. In the case of a streaming client running in a multi-homed network, this problem becomes more complicated. In the simplest form, one network can be picked randomly or based on some criteria, and the streaming client solely uses that network. In another form, multiple networks can be simultaneously used by the streaming client and due to the aggregation, doing so may deliver better and more stable quality, and at a lower latency than using either of the networks individually. However, using multiple networks simultaneously is not trivial in certain scenarios. In this demo, we present a gateway-based solution where the gateway is connected to two different networks and dynamically decides which network(s) to use for streaming while hiding all the decision complexity from the streaming clients behind it. In other words, our solution is transparent to the streaming clients, which means any existing client can benefit from this solution without any changes.

The nature of viewing is changing due to the huge volumes of content being produced including user content generated by amateurs and the proliferation of personalized services. The type of content being produced is not only for entertainment (movies/TV) purposes, but also can be instructional and directive (classroom, documentary and adult content). This leads to a type of viewing that is non-linear and requires increased random access into the content to be viewed effectively. Those who produce the content (or aggregate a number of existing ones) may not necessarily index it sufficiently for a variety of reasons. In-advance indexing would not work anyway in case the indexing used time-varying factors such as popularity, viewing frequency or duration. In this demo, we tackle this problem and present a new seekbar design for the dash.js player that allows the users to navigate the content more effectively and find the points of interest within the content faster. This new seekbar uses auxiliary metadata to show informative icons or color the parts of the media timeline differently to inform the users.

Live streaming remains a challenge in the adaptive streaming space due to the stringent requirements for not just quality and rebuffering, but also latency. Many solutions have been proposed to tackle streaming in general, but only few have looked into better catering to the more challenging low-latency live streaming scenarios. In this paper, we re-visit and extend several important components (collectively called Low-on-Latency, LoL) in adaptive streaming systems to enhance the low-latency performance. LoL includes bitrate adaptation (both heuristic and learning-based), playback control and throughput measurement modules.

Volumetric media has the potential to provide the six degrees of freedom (6DoF) required by truly immersive media. However, achieving 6DoF requires ultra-high bandwidth transmissions, which real-world wide area networks cannot provide today. Therefore, recent efforts have started to target efficient delivery of volumetric media, using a combination of compression and adaptive streaming techniques. It remains, however, unclear how the effects of such techniques on the user perceived quality can be accurately evaluated. In this paper, we present the results of an extensive objective and subjective quality of experience (QoE) evaluation of volumetric 6DoF streaming. We use PCC-DASH, a standards-compliant means for HTTP adaptive streaming of scenes comprising multiple dynamic point cloud objects. By means of a thorough analysis, we investigate the perceived quality impact of the available bandwidth, rate adaptation algorithm, viewport prediction strategy and user's motion within the scene. We determine which of these aspects has more impact on the user's QoE, and to what extent subjective and objective assessments are aligned.

HTTP adaptive streaming with chunked transfer encoding can offer low-latency streaming without sacrificing the coding efficiency. This allows media segments to be delivered while still being packaged. However, conventional schemes often make widely inaccurate bandwidth measurements due to the presence of idle periods between the chunks and hence this is causing sub-optimal adaptation decisions. To address this issue, we earlier proposed ACTE (ABR for Chunked Transfer Encoding) [6], a bandwidth prediction scheme for low-latency chunked streaming. While ACTE was a significant step forward, in this study we focus on two still remaining open areas, namely, (i) quantifying the impact of encoding parameters, including chunk and segment durations, bitrate levels, minimum interval between IDR-frames and frame rate on ACTE, and (ii) exploring the impact of video content complexity on ACTE. We thoroughly investigate these questions and report on our findings. We also discuss some additional issues that arise in the context of pursuing very low latency HTTP video streaming.

No abstract available.

Universal media access (UMA) as proposed almost two decades ago is now reality. We can generate, distribute, share, and consume any media content, anywhere, anytime, and with/on any device. A technical breakthrough was the adaptive streaming over HTTP resulting in the standardization of MPEG Dynamic Adaptive Streaming over HTTP (DASH), which is now successfully deployed in a plethora of environments. The next big thing in adaptive media streaming is virtual reality applications, and specifically, omnidirectional (360-degree) media streaming, which is currently built on top of the existing adaptive streaming ecosystems. This tutorial provides a detailed overview of adaptive streaming of both traditional and omnidirectional media. The tutorial focuses on the basic principles and paradigms for adaptive streaming as well as on already deployed content generation, distribution, and consumption workflows. Additionally, the tutorial provides insights into standards and emerging technologies in the adaptive streaming space. Finally, the tutorial includes the latest approaches for immersive media streaming enabling six Degrees of Freedom (6DoF) DASH through Point Cloud Compression (PCC) and concludes with open research issues and industry efforts in this domain.

HTTP adaptive streaming with chunked transfer encoding can be used to offer low-latency streaming without sacrificing the coding efficiency. While this allows a media segment to be generated and delivered at the same time, which is critical in reducing the latency, the conventional bitrate adaptation schemes make often grossly inaccurate bandwidth measurements due to the presence of idle periods between the chunks. These wrong measurements cause the streaming client to make bad adaptation decisions. To this end, we design ACTE, a new bitrate adaptation scheme that leverages the unique nature of chunk downloads. ACTE uses a sliding window to accurately measure the available bandwidth and an online linear adaptive filter to predict the bandwidth into the future. Results show that ACTE achieves 96% measurement accuracy, which translates to a 65% reduction in the number of stalls and a 49% increase in quality of experience on average compared to other schemes.

The papers in this special issue focus on trustworthiness in multimedia communications. Recently, social multimedia content is being delivered to users with a high quality of experience (QoE) with the advance of multimedia technologies and social networks. However, as a huge amount of social users have various demands to exchange and share multimedia content with each other, it becomes a new challenge for the current social multimedia analytics and delivery to deal with the various attacks perpetrated by malicious users or through spam contents. Therefore, the trust and risk management for social multimedia content based on the social tie of users become of prime importance to face the unpredicted threats and subsequent damage. This Special Section aims to provide a premier forum for researchers working on the trust-based social multimedia analytics and delivery. It also provides the opportunity for both academic and industrial researchers to discuss recent results and provide solutions to the above-mentioned challenges.

The dramatic growth of HTTP adaptive streaming (HAS) traffic represents a practical challenge for service providers in satisfying the demand from their customers. Achieving this in a network where multiple players share the network capacity has so far proved hard because of the bandwidth competition among the HAS players. This competition is exacerbated by the bandwidth overestimation that is introduced due to the isolated and selfish behavior of the HAS players. Each player strives individually to select the maximum bitrate without considering the co-existing players or network resource dynamics. As a result, the HAS players suffer from video quality instability, quality unfairness, and network underutilization or oversubscription, and the players observe a poor quality of experience (QoE). To address this issue, we propose a fully distributed game theory and consensus-based collaborative adaptive bitrate solution for shared network environments, termed Game Theory and consensus-based Approach for Cooperative HAS delivery systems (GTAC). Our solution consists of two-stage games that run in parallel during a streaming session. We extensively evaluate GTAC on a broad set of trace-driven and real-world experiments. Results show that GTAC enhances the viewer QoE by up to 22%, presentation quality stability by up to 24%, fairness by at least 31%, and network utilization by 28% compared to the well-known schemes.

Video resolution changes in an HTTP adaptive streaming session may negatively affect the viewer's quality of experience. Our goal is, through encoding, to make such resolution changes less noticeable for the viewers. This can be achieved by taking video complexity features into account during the encoding process. In this paper, we compare Constant Bitrate (CBR) versus Constrained Constant Rate Factor (CRF) coding approaches and their effects on the noticeability of video resolution changes. To this end, we conducted a dedicated subjective study with 20 subjects in a quasi -lab environment. Our results suggest that choices for fixed-bitrate encoding have to be improved by deeper analysis of video complexity and resolution change patterns.

Past research has shown that concurrent HTTP adaptive streaming (HAS) players behave selfishly and the resulting competition for shared resources leads to underutilization or oversubscription of the network, presentation quality instability and unfairness among the players, all of which adversely impact the viewer experience. While coordination among the players, as opposed to all being selfish, has its merits and may alleviate some of these issues. A fully distributed architecture is still desirable in many deployments and better reflects the design spirit of HAS. In this study, we focus on and propose a distributed bitrate adaptation scheme for HAS that borrows ideas from consensus and game theory frameworks. Experimental results show that the proposed distributed approach provides significant improvements in terms of viewer experience, presentation quality stability, fairness and network utilization, without using any explicit communication between the players.

In streaming media, it is imperative to deliver a good viewer experience to preserve customer loyalty. Prior research has shown that this is rather difficult when shared Internet resources struggle to meet the demand from streaming clients that are largely designed to behave in their own self-interest. To date, several schemes for adaptive streaming have been proposed to address this challenge with varying success. In this paper, we take a different approach and develop a game theoretic approach. We present a practical implementation integrated in the dash.js reference player and provide substantial comparisons against the state-of-the-art methods using trace-driven and real-world experiments. Our approach outperforms its competitors in the average viewer experience by 38.5% and in video stability by 62%.

In designing an HTTP adaptive streaming (HAS) system, the bitrate adaptation scheme in the player is a key component to ensure a good quality of experience (QoE) for viewers. We propose a new online reinforcement learning optimization framework, called ORL-SDN, targeting HAS players running in a software-defined networking (SDN) environment. We leverage SDN to facilitate the orchestration of the adaptation schemes for a set of HAS players. To reach a good level of QoE fairness in a large population of players, we cluster them based on a perceptual quality index. We formulate the adaptation process as a Partially Observable Markov Decision Process and solve the per-cluster optimization problem using an online Q-learning technique that leverages model predictive control and parallelism via aggregation to avoid a per-cluster suboptimal selection and to accelerate the convergence to an optimum. This framework achieves maximum long-term revenue by selecting the optimal representation for each cluster under time-varying network conditions. The results show that ORL-SDN delivers substantial improvements in viewer QoE, presentation quality stability, fairness, and bandwidth utilization over well-known adaptation schemes.

HTTP adaptive streaming (HAS) is receiving much attention from both industry and academia as it has become the de facto approach to stream media content over the Internet. Recently, we proposed a streaming architecture called SDNDASH [1] to address HAS scalability issues including video instability, quality of experience (QoE) unfairness, and network resource underutilization, while maximizing per player QoE. While SDNDASH was a significant step forward, there were three unresolved limitations: 1) it did not scale well when the number of HAS players increased; 2) it generated communication overhead; and 3) it did not address client heterogeneity. These limitations could result in suboptimal decisions that led to viewer dissatisfaction. To that effect, we propose an enhanced intelligent streaming architecture, called SDNHAS, which leverages software defined networking (SDN) capabilities of assisting HAS players in making better adaptation decisions. This architecture accommodates large-scale deployments through a cluster-based mechanism, reduces communication overhead between the HAS players and SDN core, and allocates the network resources effectively in the presence of short- and long-term changes in the network.

No abstract available.

HTTP adaptive streaming (HAS) is being adopted with increasing frequency and becoming the de-facto standard for video streaming. However, the client-driven, on-off adaptation behavior of HAS results in uneven bandwidth competition and this is exacerbated when a large number of clients share the same bottleneck network link and compete for the available bandwidth. With HAS each client independently strives to maximize its individual share of the available bandwidth, which leads to bandwidth competition and a decrease in end-user quality of experience (QoE). The competition causes scalability issues, which are quality instability, unfair bandwidth sharing and network resource underutilization. We propose a new software defined networking (SDN) based dynamic resource allocation and management architecture for HAS systems, which aims to alleviate these scalability issues and improve the per-client QoE. Our architecture manages and allocates the network resources dynamically for each client based on its expected QoE. Experimental results show that the proposed architecture significantly enhances scalability by improving per-client QoE by at least 30% and supporting up to 80% more clients with the same QoE compared to the conventional schemes.

In this tutorial we present state of the art and challenges ahead in over-the-top content delivery. It particular, the goal of this tutorial is to provide an overview of adaptive media delivery, specifically in the context of HTTP adaptive streaming (HAS) including the recently ratified MPEG-DASH standard. The main focus of the tutorial will be on the common problems in HAS deployments such as client design, QoE optimization, multi-screen and hybrid delivery scenarios, and synchronization issues. For each problem, we will examine proposed solutions along with their pros and cons. In the last part of the tutorial, we will look into the open issues and review the work-in-progress and future research directions.

In conventional HTTP-based adaptive streaming (HAS), a video source is encoded at multiple levels of constant bitrate representations, and a client makes its representation selections according to the measured network bandwidth. While greatly simplifying adaptation to the varying network conditions, this strategy is not the best for optimizing the video quality experienced by end users. Quality fluctuation can be reduced if the natural variability of video content is taken into consideration. In this work, we study the design of a client rate adaptation algorithm to yield consistent video quality. We assume that clients have visibility into incoming video within a finite horizon. We also take advantage of the client-side video buffer, by using it as a breathing room for not only network bandwidth variability, but also video bitrate variability. The challenge, however, lies in how to balance these two variabilities to yield consistent video quality without risking a buffer underrun. We propose an optimization solution that uses an online algorithm to adapt the video bitrate step-by-step, while applying dynamic programming at each step. We incorporate our solution into PANDA -- a practical rate adaptation algorithm designed for HAS deployment at scale.

Video streaming is a major source of Internet traffic today and usage continues to grow at a rapid rate. To cope with this new and massive source of traffic, ISPs use methods such as caching to reduce the amount of traffic traversing their networks and serve customers better. However, the presence of a standard cache server in the video transfer path may result in bitrate oscillations and sudden rate changes for Dynamic Adaptive Streaming over HTTP (DASH) clients. In this paper, we investigate the interactions between a client and a cache that result in these problems, and propose an approach to solve it. By adaptively controlling the rate at which the client downloads video segments from the cache, we can ensure that clients will get smooth video. We verify our results using simulation and show that compared to a standard cache our approach (1) can reduce bitrate oscillations (2) prevents sudden rate changes, and compared to a no-cache scenario (3) provides traffic savings, and (4) improves the quality of experience of clients.

Today, the technology for video streaming over the Internet is converging towards a paradigm named HTTP-based adaptive streaming (HAS), which brings two new features. First, by using HTTP/TCP, it leverages network-friendly TCP to achieve both firewall/NAT traversal and bandwidth sharing. Second, by pre-encoding and storing the video in a number of discrete rate levels, it introduces video bitrate adaptivity in a scalable way so that the video encoding is excluded from the closed-loop adaptation. A conventional wisdom in HAS design is that since the TCP throughput observed by a client would indicate the available network bandwidth, it could be used as a reliable reference for video bitrate selection. We argue that this is no longer true when HAS becomes a substantial fraction of the total network traffic. We show that when multiple HAS clients compete at a network bottleneck, the discrete nature of the video bitrates results in difficulty for a client to correctly perceive its fair-share bandwidth. Through analysis and test bed experiments, we demonstrate that this fundamental limitation leads to video bitrate oscillation and other undesirable behaviors that negatively impact the video viewing experience. We therefore argue that it is necessary to design at the application layer using a "probe and adapt" principle for video bitrate adaptation (where "probe" refers to trial increment of the data rate, instead of sending auxiliary piggybacking traffic), which is akin, but also orthogonal to the transport-layer TCP congestion control. We present PANDA - a client-side rate adaptation algorithm for HAS - as a practical embodiment of this principle. Our test bed results show that compared to conventional algorithms, PANDA is able to reduce the instability of video bitrate selection by over 75% without increasing the risk of buffer underrun.

This special issue is concerned with the latest developments in state-of-the-art adaptive media streaming technologies and applications.

Prior work has shown that two or more adaptive streaming players can be unstable when they compete for bandwidth. The root cause of the instability problem is that, in Steady-State, a player goes through an ON-OFF activity pattern in which it overestimates the available bandwidth. We propose a server-based traffic shaping method that can significantly reduce such oscillations without significant (or any) loss in bandwidth utilization. The shaper is only activated when oscillations are detected, and it dynamically adjusts the shaping rate so that the player should ideally receive the highest available video profile while being stable. We evaluate the proposed method experimentally in terms of instability and utilization comparing with the unshaped case, under several scenarios.

With an increasing demand for high-quality video content over the Internet, it is becoming more likely that two or more adaptive streaming players share the same network bottleneck and compete for available bandwidth. This competition can lead to three performance problems: player instability, unfairness between players, and bandwidth underutilization. However, the dynamics of such competition and the root cause for the previous three problems are not yet well understood. In this paper, we focus on the problem of competing video players and describe how the typical behavior of an adaptive streaming player in its Steady-State, which includes periods of activity followed by periods of inactivity (ON-OFF periods), is the main root cause behind the problems listed above. We use two adaptive players to experimentally showcase these issues. Then, focusing on the issue of player instability, we test how several factors (the ON-OFF durations, the available bandwidth and its relation to available bitrates, and the number of competing players) affect stability.

As more and more PC and handheld like devices get connected, consumers are migrating to the Web to watch their favorite shows and movies. Increasingly, the Web is coming to digital TV, which incorporates movie downloads and streaming. Similarly, consumers also want their TV content on alternative devices. What does this mean for service and content providers? What do they have to do to not lose their subscribers and revenue streams? This talk overviews the TV Everywhere technologies available for integrating the emerging over-the-top content into a managed network and making premium content accessible for unmanaged devices. The talk also provides a few real-world use cases.

Internet protocol television (IPTV) systems employ IP multicast to deliver television programs to end-users. To provide reliable IPTV services over the error-prone digital subscriber line (DSL) access networks, a combination of multicast forward error correction and unicast retransmissions is employed to mitigate the impulse noise in DSL links. In current systems, the error control function is provided by special retransmission servers. In this paper, we propose an alternative distributed solution where the burden of packet loss repair is partially shifted to end-user set-top boxes. Using a peer-assisted repair (PAR) protocol, we demonstrate how packet repairs can be delivered in a timely, reliable, and decentralized manner using the combination of server-peer coordination and redundant repairs. We also show that this distributed protocol can be seamlessly integrated with an application-layer source-aware error protection mechanism called forward and retransmitted systematic lossy error protection (SLEP/SLEPr). Analysis and simulations show that this joint PAR-SLEP/SLEPr framework not only efficiently improves the resistance to the impulse noise but also effectively mitigates the bottleneck experienced by the retransmission servers, thus greatly enhancing system scalability.

Adaptive (video) streaming over HTTP is gradually being adopted, as it offers significant advantages in terms of both user-perceived quality and resource utilization for content and network service providers. In this paper, we focus on the rate-adaptation mechanisms of adaptive streaming and experimentally evaluate two major commercial players (Smooth Streaming, Netflix) and one open source player (OSMF). Our experiments cover three important operating conditions. First, how does an adaptive video player react to either persistent or short-term changes in the underlying network available bandwidth. Can the player quickly converge to the maximum sustainable bitrate? Second, what happens when two adaptive video players compete for available bandwidth in the bottleneck link? Can they share the resources in a stable and fair manner? And third, how does adaptive streaming perform with live content? Is the player able to sustain a short playback delay? We identify major differences between the three players, and significant inefficiencies in each of them.

Quality of experience (QoE) is an important, and admittedly overloaded, concept for the emerging IPTV services. Service providers are continuously working towards delivering a better TV experience. To this effect, cost-effective and scalable tools are highly desirable for QoE monitoring, diagnostics and reporting. In this paper, we demonstrate that the RTP control protocol and its network tomography extensions can satisfy the needs of the providers in collecting and reporting both detailed and summarized information using several numerical examples based on real-life scenarios.

In peer-to-peer (P2P) live streaming using unstructured mesh, packet scheduling is an important factor on overall playback delay. The hybrid pull-push approach has been recently proposed to reduce delay compared to classical pulling method. In this approach, video are divided into substreams and packets are pushed with low delay. There has been little work addressing the scheduling problem on substream assignment. In this paper, we study the scheduling problem on assigning substreams to minimize packet delay. Given heterogeneous contents, delays and bandwidths of parents, we formulate the substream assignment (SA) problem to assign substreams to parents with minimum delay. The SA problem can be optimally solved in polynomial time by transforming it into a max-weighted bipartite matching problem. Simulation results show that our distributed algorithm achieves substantially lower delay as compared with traditional pull and current hybrid pull-push approaches based on greedy algorithm.

We study video transcoding for accelerated channel changes in IPTV systems. Video transcoding at the Retransmission Server not only reduces the channel change latency, but also reduces the duration and data size of the unicast burst stream used for rapid acquisition. We develop an analytical model to capture the fundamental trade-offs in this system. This model is then used to characterize the potential savings from transcoding the unicast stream. Analysis and simulation results show that the stream compression factor affects linearly the saving in the channel change latency, and superlinearly the saving in unicast burst duration (or data size).

Emerging IPTV technology uses source-specific IP multicast to deliver television programs to end-users. To provide reliable IPTV services over the error-prone DSL access networks, a combination of multicast forward error correction (FEC) and unicast retransmissions is employed to mitigate the impulse noises in DSL links. In existing systems, the retransmission function is provided by the Retransmission Servers sitting at the edge of the core network. In this work, we propose an alternative distributed solution where the burden of packet loss repair is partially shifted to the peer IP set-top boxes. Through Peer-Assisted Repair (PAR) protocol, we demonstrate how the packet repairs can be delivered in a timely, reliable and decentralized manner using the combination of server-peer coordination and redundancy of repairs. We also show that this distributed protocol can be seamlessly integrated with an application-layer source-aware error protection mechanism called forward and retransmitted Systematic Lossy Error Protection (SLEP/SLEPr). Simulations show that this joint PARSLEP/ SLEPr framework not only effectively mitigates the bottleneck experienced by the Retransmission Servers, thus greatly enhancing the scalability of the system, but also efficiently improves the resistance to the impulse noise.

In peer-to-peer (P2P) live streaming using unstructured mesh, packet scheduling is an important factor in overall playback delay. In this paper, we propose a scheduling algorithm to minimize scheduling delay. To achieve low delay, our scheduling is predominantly push in nature, and the schedule needs to be changed only upon significant change in network states (due to, for examples, bandwidth change or parent churns). Our scheme, termed SPANC (Substream Pushing and Network Coding), pushes video packets in substreams and recovers packet loss using network coding. Given heterogeneous contents, delays, and bandwidths of parents of a peer, we formulate the substream assignment (SA) problem to assign substreams to parents with minimum delay. The SA problem can be optimally solved in polynomial time by transforming it to a max-weighted bipartite matching problem. We then formulate the fast recovery with network coding (FRNC) problem, which is to assign network coded packets to each parent to achieve minimum recovery delay. The FRNC problem can also be solved exactly in polynomial time with dynamic programming. Simulation results show that SPANC achieves substantially lower delay with little cost in bandwidth, as compared with recent approaches based on pull, network coding and hybrid pull-push.

Peer-to-peer (P2P) streaming has been widely deployed over the Internet. A streaming system usually has multiple channels, and peers may form multiple groups for content distribution. In this paper, we propose a distributed overlay framework (called SMesh) for dynamic groups where users may frequently hop from one group to another while the total pool of users remain stable. SMesh first builds a relatively stable mesh consisting of all hosts for control messaging. The mesh supports dynamic host joining and leaving, and will guide the construction of delivery trees. Using the Delaunay Triangulation (DT) protocol as an example, we show how to construct an efficient mesh with low maintenance cost. We further study various tree construction mechanisms based on the mesh, including embedded, bypass, and intermediate trees. Through simulations on Internet-like topologies, we show that SMesh achieves low delay and low link stress.

In multicast-based IPTV distribution networks, when an IPTV viewer tunes to a new channel, the IP set-top box (STB) joins a new multicast session. Upon join, the IP STB needs to acquire and parse certain key information before it can process any data sent in the multicast session. Depending on the join time, length of the key information repetition interval, size of the key information as well as the application and transport properties, the time lag before the IP STB can usefully consume the multicast data, which we refer to as the synchronization delay, varies and may be large. This is an undesirable phenomenon and degrades the quality of experience perceived by the IPTV viewers. In this study, we describe a unified standards-based approach that can be used both to repair lost packets in real time and reduce the synchronization delay.

In this study, we discuss how to provide low-delay peer-to-peer streaming with high video quality by considering mesh design with backup parents. To achieve robustness against peer churns, each peer has a certain number of streaming parents and backup parents. We have designed a distributed algorithm that constructed a low-delay mesh and at the same time achieved a certain stream continuity for the peers. We have conducted extensive simulations to study the performance of our algorithms. The results show that our distributed algorithm achieves a lower source-to-peer delay as compared with a traditional scheme. Our results have shown that peer-to-peer live streaming can be delivered in short delays while providing a high level of quality, despite peer churns and the lack of a centralized planner.

In live peer-to-peer (P2P) streaming, each peer (child) has a number of supplying parents whose packets have to be scheduled and delivered in time for continuous playback at the child. It is challenging to develop a scheduling algorithm that achieves low delay given heterogeneous bandwidth, propagation delays and available content in all the parents. This paper proposes a novel, simple and effective scheduling scheme called pattern-push. As compared to the traditional mesh-pull, pattern-push does not require continuous buffermap advertisements from the parents, and operates on the packet level instead of the larger segment level. In pattern-push, each parent pushes its packets according to a pattern as indicated by a starting packet ID and a cycle bitmap. Pattern-push requires only minimal feedback from the child, as the pattern only needs to be changed when the child detects a marked change in network conditions or its parents. Simulation results show that pattern-push achieves a significantly lower delay and overhead as compared with both traditional and recent scheduling algorithms proposed in the literature.

Emerging IPTV technology uses source-specific IP multicast to deliver TV programs to the end-users. To provide timely and reliable services over the error-prone DSL access networks, a combination of multicast forward error correction and unicast retransmissions is employed to mitigate the impact of impulse noise. In current systems, the retransmission function is provided by the Retransmission Servers.

We discuss the necessity of error control for supporting IPTV over imperfect access networks. In particular, we consider typical DSL environments, and examine the physical-layer impairments and error-mitigation techniques. For these networks, we evaluate the performance of two different application-layer Forward Error Correction (FEC) methods. An overview of hybrid error-control methods and recent developments in standardization is also presented.

Developing error-control and error-resiliency methods for transmitting delay-sensitive media content over the best-effort networks poses several challenges. Due to the lack of QoS guarantees in the conventional Internet as well as in emerging wireless networks, these methods must continuously monitor the characteristics of the underlying network and try to infer the incipient network conditions so that they can take the necessary actions on time. This is utmost important for enhancing the end-user quality, particularly in low-delay multimedia applications. In this study, we tackle this problem from an error-control method perspective and develop an innovative framework that optimizes the retransmission decisions based on the urgency and importance of the media packets.

Time-constrained error recovery is an integral component of reliable low-delay video applications. Regardless of the error-control method adopted by the application, unacknowledged or missing packets must be quickly identified as lost or delayed, so that necessary actions can be taken by the server/client on time. Historically, this problem has been referred to as retransmission timeout (RTO) estimation. Earlier studies show that existing RTO estimators suffer from either long loss detection times or a large number of spurious timeouts. The goal of this study is to address these problems by developing an RTO estimation method specifically tailored for low-delay video applications. In the media-unaware mode, this method exploits the temporal dependence in packet delay to optimally manage the tradeoff between the amount of overwaiting and redundant retransmission rate. As opposed to existing methods, our approach is completely adaptive to the source video characteristics and time-varying network conditions, and does not use any preset parameters. In the media-aware mode, on the other hand, the timeout estimates are jointly optimized based on the importance and urgency of the video packets such that the rendering quality is maximized under the given rate constraints. With a comprehensive set of simulation and experimental results, we show that both the media-unaware and media-aware RTO estimators detect lost packets faster and more accurately than their rivals. Furthermore, our results also substantiate the fact that the media-aware RTO estimator outperforms all other RTO estimators in terms of video quality

Transmission of time-critical video traffic over the networks with large end-to-end delays poses two main difficulties: slow adaptation to changing network conditions and inability of timely recovery from lost packets. Previously, we proposed a proxy-based solution that enabled us to cope with these problems. Our Internet experiments with a single proxy system showed that the QoS delivered by interactive video applications could be greatly improved by the Intermediate-Proxy approach. In multi-proxy systems, however, the availability of a large number of proxies throughout the Internet will bring the proxy selection problem to the fore. This is an important problem since the benefits of the intermediate-proxy approach can be best exploited with a proper proxy selection. In this study, our goal is to model the dynamics involved in the networks with proxies, and investigate mathematical and practical proxy selection methods.

This paper studies linear prediction algorithms for packet-delay modeling. A detailed examination of the delay traces collected from video streams encoded at different bitrates, suggests that autoregressive (AR) models can exploit the correlation among the delay samples and produce the best estimates in terms of the mean-squared error criterion. Simulation results show that AR prediction can reduce the average prediction-error power significantly as compared to the exponentially-weighted moving average prediction as well as the recursive weighted median filtering. This is a promising result since many layers in the multimedia communication protocol stack, e.g., rate control, error control and network adaptation, can greatly benefit from accurate packet-delay prediction.

Time-constrained error recovery is an integral component of reliable low-delay video applications. Regardless of the error-control method adopted by the application, unacknowledged or missing packets must be quickly identified as lost or delayed, so that necessary timely actions can be taken by the server/client. Historically, this problem has been referred to as the retransmission timeout (RTO) estimation. Earlier studies show that existing RTO estimators suffer from either long loss detection times or a large number of pre-mature timeouts. The goal of this study is to address these problems by developing an adaptive RTO estimator for high-bitrate low-delay video applications. By exploiting the temporal dependence between consecutive delay samples, we propose an adaptive linear delay predictor. This way, our RTO estimator configures itself based on the video characteristics and varying network conditions. Our approach also features a controller that optimally manages the trade-off between the amount of overwaiting and redundant retransmission rate. The skeleton implementation shows that the proposed RTO estimator discriminates lost packets from excessively-delayed packets faster and more accurately than its rivals, which consequently enables the applications to recover more packets under stringent delay requirements.

Wireless mesh systems offer several advantages for emerging high-bandwidth networks because of their cooperative routing capabilities. In this study, we consider the potential benefits of using wireless meshes within residential networks for applications that require high bandwidth and low latency. In particular, we consider high-bitrate video transmission inside a mesh-networked home. To quantify our findings, we present experimental results obtained from a high-resolution video streaming application. Our experiments involve single-hop, multi-hop, single-path and multi-path transmission methods, and two types of video coding techniques, namely single description coding and multiple description coding. In the light of our results, we discuss the pros and cons of each streaming method, and motivate a set of research problems.

In retransmission-based error-control methods, the most fundamental yet the paramount problem is to determine how long the sender (or the receiver) should wait before deciding that an unacknowledged (or a missing) packet is lost. This waiting time is generally referred to as retransmission timeout (RTO). An accurate RTO estimation has two main advantages: First, the lost packets can be identified earlier, and hence, can be recovered faster. Second, redundant retransmissions can be avoided, which subsequently not only saves the network resources, but also helps existing network congestion alleviate sooner. Although it is statistically possible to prevent any unnecessary retransmission at the expense of long error-recovery times, such an approach can only be justified for data applications; it is not well-suited for delay-sensitive applications, for which the agility in recovering the lost packets is as important. With this motivation, we recently introduced an RTO estimation algorithm for delay-sensitive applications (A. C. Begen et al., 2004). Provided that the packets are transmitted at equal intervals, this technique successfully estimates the arrival times based on the interarrival-time observations. In this study, we relax the requirement of equal transmission intervals and generalize our technique to handle bursty video applications

This paper presents a client-driven rate-distortion optimized packet scheduling algorithm for streaming multiple description (MD) encoded video, where the descriptions are distributed among different video-on-demand servers. Previously, multiple description coding (MDC) has been proposed to provide reliable video communication within a content delivery network, where the main goal was to imitate path diversity by using multiple servers concurrently. Although this approach can alleviate the severe effects of bursty packet losses, transmitting video packets in a time-sensitive and network-adaptive manner is fundamental to the success of MD streaming. Hence, in this paper we propose a packet scheduling algorithm that maximizes the quality of the video rendered at the client under the given rate constraints. This algorithm jointly considers the timeliness requirements of the streaming application, dependency structure of the streamed video, network conditions as well as the error-resiliency features of MDC.

This paper presents a new framework for multimedia streaming that integrates the application and network layer functionalities to meet such stringent application requirements as delay and loss. The coordination between these two layers provides more robust media transmission even under severe network conditions. In this framework, a multiple description source coder is used to produce multiple independently-decodable streams that are routed over partially link-disjoint (non-shared) path to combat bursty packet losses. We model multi-path streaming and propose a multi-path streaming and propose a multi-path selection method that chooses a set of paths maximizing the overall quality at the client. Overlay infrastructure is then used to achieve multi-path routing over these selected paths. The simulation results show that the average peak signal-to-noise ratio (PSNR) improves by up to 8.1 dB, if the same source video is routed over intelligently selected multiple paths instead of the shortest path or maximally link-disjoint paths. In addition to PSNR improvement in quality, the end-user experiences a more continual steaming quality.

This paper studies the streaming of packetized media from multiple servers to a client over a lossy network. In particular, we propose a client-driven rate-distortion optimal packet scheduling algorithm that decides which packet(s) to be requested from which server(s) at a given request opportunity. In doing so, the proposed scheduling algorithm not only attains the maximal presentation quality but also conforms to the rate constraints dictated by the flow, window and congestion control mechanisms. The simulation results clearly demonstrate the efficacy of the proposed algorithm over the single-server rate-distortion optimized streaming.

In a previous work [A. C. Begen, et al., 2003], we proposed an optimal multi-path selection method for multiple description (MD) encoded video streaming. To do so, we first modelled multi-path streaming and then developed an expression, i.e., an objective (cost) function, that estimated average streaming distortion in terms of network statistics, media characteristics and application requirements. Naturally, the ultimate goal was to find the set of paths that minimized this cost function. However, finding such sets of paths turned out to be intractable in large topologies. Hence, in this paper, we provide a fast heuristics-based solution by exploiting the infrastructure features of the Internet. The simulations run over various random Internet topologies show that the proposed heuristic is able to find a good solution in a much shorter time than the brute-force approach. Particularly, this heuristic is best suited to such interactive multimedia applications as video-conferencing and VoIP, where multi-path computation is a time-critical process. In addition, it is also suitable for the clients whose processing power capabilities are limited.

Multiple description (MD) and layered coding (LC) are two source-coding approaches proposed for adaptive and robust video streaming over lossy networks. Both streaming methods aim to improve the streaming quality by transmitting the generated multiple sub-bitstreams over partially link-disjoint paths. However, the achieved qualities heavily depend on the media characteristics, path conditions and application requirements. In order to attain the highest quality, we study optimal multi-path selection methods for both MD and LC streaming. The simulations run over an overlay infrastructure show 9.0 - 12.5 dB peak signal-to-noise ratio (PSNR) improvement when the video is streamed over intelligently selected multiple paths instead of the shortest path or maximally link-disjoint paths. By the help of the proposed path selection methods, the end users estimate the expected qualities of MD and LC streaming for the given network conditions and application requirements prior to the streaming, which allows the users to make a choice accordingly.