This special issue presents recent advances in multimedia systems research showcased at ACM Multimedia Systems 2024 and its co-located workshops. The selected papers span adaptive and immersive video streaming, low-latency and scalable delivery architectures, and innovations in video coding and processing. Together, they illustrate the rapid progress and broad impact of emerging techniques across the multimedia stack.

Dynamic Adaptive Streaming over HTTP (DASH) is the standard for video streaming applications such as YouTube and Netflix. According to DASH, the video player must include an Adaptive Bit-Rate (ABR) controller designed to maximize users' Quality of Experience (QoE). The controller must avoid playback interruptions, due to buffer underflow, while at the same time selecting the maximum video encoding quality compatible with the available time-varying bandwidth. This paper proposes a controller designed using a nicely constrained Model Predictive Control (MPC) which employs Bellman Dynamic Programming (DP) to reduce the computational cost of the algorithm from exponential to polynomial. Compared with state-of-the-art ABR algorithms, the proposed Real-Time MPC (RT-MPC) improves QoE while remarkably reducing the computational time, so that the algorithm can be used for both live-video streaming and real-time conferencing.

Autonomous mobile robots (AMRs) are increasingly deployed for remote inspection in various scenarios. While AMRs can perform missions autonomously, human intervention becomes necessary in cases where safety risks or potential damage to the robot arise. Since these platforms are often deployed in remote or potentially dangerous areas, physical access is not always available. In such situations, teleoperation provides an effective solution to resolve issues. The navigation stack of AMRs include obstacle avoidance and mapping functionalities that rely on the environmental feedback, typically acquired with stereo cameras which represent depth and color information using Point Clouds (PCs). In addition to obstacles mapping, PCs provide operators and algorithms with real-time environmental data improving situational awareness and supporting AI-based downstream tasks. However, the large volumes of data inherent to PCs pose challenges for real-time transmission. For this reason, dedicated transmission pipelines and data compression algorithms compatible with the limited computational capabilities of the on-board computer are required. This paper demonstrates such a scenario, equipping a differential drive ground robot with a stereo camera that produces PCs that are streamed to a remote operator wearing a standalone VR Headset. To accomodate the limited computational capabilities of the on-board computer and reduce bandwidth requirements, a distance-based filtering and quantization encoding is applied. Users will step in to assist the robot, with remote real-time teleoperation, when the AMR issues a "cry for help" signal during an autonomous mission.

Adaptive video streaming systems are expected to provide the best user experience to improve service engagement. To the purpose, video players host a controller that dynamically chooses the most suitable video representation to be downloaded. It is well-known that finding one tuning of the controller’s parameters which performs satisfactorily in a wide range of scenarios is very challenging. This paper studies the problem of providing users with (near) optimal Quality of Experience (QoE) for Dynamic Adaptive Streaming over HTTP (DASH) systems. We present ERUDITE, a closed-loop system to optimally tune – at run-time – the adaptive streaming controller’s parameters to adapt to changing scenario’s parameters. ERUDITE employs a Deep Neural Network (DNN) which continuously provides the streaming controller with estimates of optimal parameters based on measured metrics such as bandwidth samples and overall obtained QoE. The DNN is trained using a dataset that we have built by finding, for thousands of realistic scenarios, robust optimal adaptive streaming controller’s parameters using a Bayesian optimization algorithm. Results, gathered considering a large number of diverse scenarios, show that ERUDITE is able to provide near optimal performances by reducing impairments due to rebuffering and video level switching.

Unmanned Aerial Vehicles (UAVs), commonly denoted as drones, are being increasingly adopted as platforms to enable applications such as surveillance, disaster response, environmental monitoring, live drone broadcasting, and Internet-of-Drones (IoD). In this context, drone systems are required to carry out tasks autonomously in potentially unknown and challenging environments. As such, deep learning algorithms are widely adopted to implement efficient perception from sensors, making the availability of comprehensive datasets capturing real-world environments important. In this work, we introduce APEIRON, a rich multimodal aerial dataset that simultaneously collects perception data from a stereocamera and an event based camera sensor, along with measurements of wireless network links obtained using an LTE module. The assembled dataset consists of both perception and network data, making it suitable for typical perception or communication applications, as well as cross-disciplinary applications that require both types of data. We believe that this dataset will help promoting multi-disciplinary research at the intersection of multimedia systems, computer networks, and robotics fields. APEIRON is available at https://c3lab.github.io/Apeiron/.

Millimeter-wave (mmWave) radar sensors produce Point Clouds (PCs) that are much sparser and noisier than other PC data (e.g., Li-DAR), yet they are more robust in challenging conditions such as in the presence of fog, dust, smoke, or rain. This paper presents MilliNoise, a point cloud dataset captured in indoor scenarios through a mmWave radar sensor installed on a wheeled mobile robot. Each of the 12M points in the MilliNoise dataset is accurately labeled as true/noise point by leveraging known information of the scenes and a motion capture system to obtain the ground truth position of the moving robot. Each frame is carefully pre-processed to produce a fixed number of points for each cloud, enabling classification tools which require data with a fixed shape. Moreover, MilliNoise has been post-processed by labeling each point with the distance to its closest obstacle in the scene, which allows casting the denoising task into the regression framework. Along with the dataset, we provide researchers with the tools to visualize the data and prepare it for statistical and machine learning analysis. MilliNoise is available at: https://github.com/c3lab/MilliNoise

A steadily increasing number of users consume videos over the Internet. In current video platforms, players run a control algorithm that dynamically chooses the video bitrate to match the time-varying network bandwidth. Such an algorithm strives to improve the quality individually perceived by users. Consequently, this control architecture leads, in the optimal case, to maximize the average quality perceived collectively by all users rather than to a quality-fair distribution of resources, possibly leading to user abandonment for those users receiving a lower quality. Therefore, we argue that well-designed video delivery networks should gracefully degrade the perceived quality equally for all users when resources become scarce. In this paper, we propose the Multi-Commodity Flow Problem (MCFP) optimization framework to address the issue of designing a QoE-fair optimal allocation strategy. We show how to make the resulting problem tractable for video platforms serving massive audiences. The performance of the proposed optimal fair resource allocation strategy is tested through realistic simulations involving thousands of concurrent users on two real networks by varying both the total load on the network and the system parameters.

Video streaming platforms are required to innovate their delivery pipeline to allow new and more immersive video content to be supported. In particular, Omnidirectional videos enable the user to explore a 360° scene by moving their heads using Head Mounted Display devices. Viewport adaptive streaming allows changing dynamically the quality of the video falling in the user's field of view. In this paper, we present TAPAS-360°, an open-source tool that enables designing and experimenting all the components required to build omnidirectional video streaming systems. The tool can be used by researchers focusing on the design of viewport-adaptive algorithms and also to produce video streams to be employed for subjective and objective Quality of Experience evaluations.

Adaptive video streaming systems are expected to provide the best user experience to improve service engagement. To this purpose, the video player implements a controller to dynamically choose the most suitable video representation to be downloaded. It is well-known that finding one tuning of the controller's parameters which performs satisfactorily in a wide range of scenarios is very challenging. This paper studies the problem of providing users with (near) optimal Quality of Experience (QoE) for Dynamic Adaptive Streaming over HTTP (DASH) systems. We present ERUDITE, a closed-loop system to optimally tune - at run-time - the adaptive streaming controller's parameters to adapt to changing scenario's parameters. The proposed system is based on a Deep Neural Network (DNN) which continuously provides the streaming controller with estimates of optimal parameters based on measured metrics such as bandwidth samples and overall obtained QoE. The DNN is trained using a dataset that we have built by finding, for thousands of scenarios, the optimal adaptive streaming controller's parameters using a Bayesian optimization algorithm. Results, gathered considering a large number of diverse scenarios, show that ERUDITE is able to provide near optimal performances by reducing impairments due to rebuffering and video level switching.

Virtual Reality/Augmented Reality applications require streaming 360° videos to implement new services in a diverse set of fields such as entertainment, art, e-health, e-learning, and smart factories. Providing a high Quality of Experience when streaming 360° videos is particularly challenging due to the very high required network bandwidth. In this paper, we showcase a proof-of-concept implementation of a complete DASH-compliant delivery system for 360° videos that: 1) allows reducing the required bitrate, 2) is independent of the employed encoder, 3) leverages technologies that are already available in the vast majority of mobile platforms and devices. The demo platform allows the user to directly experiment with various parameters, such as the duration of segments, the compression scheme, and the adaptive streaming algorithm parameters.

We address security issues in cyber-physical systems (CPSs). We focus on the detection of attacks against cyber-physical systems. Attacks against these systems shall be handled both in terms of safety and security. Networked-control technologies imposed by industrial standards already cover the safety dimension. However, from a security standpoint, using only cyber information to analyze the security of a cyber-physical system is not enough, since the physical malicious actions that can threaten the correct behavior of the systems are ignored. For this reason, the systems have to be protected from threats to their cyber and physical layers. Some authors have handled replay and integrity attacks using, for example, physical attestation to validate the cyber process and to detect the attacks, or watermark-based detectors which uses also physical parameters to ensure the cyber layers. We reexamine the effectiveness of a stationary watermark-based detector. We show that this approach only detects adversaries that do not attempt to get any knowledge about the system dynamics. We analyze the detection ratio of the original design under the presence of new adversaries that are able to infer the system dynamics and are able to evade the detector with high frequency. We propose a new detection scheme which employs several non-stationary watermarks. We validate the detection efficiency of the new strategy via numeric simulations and via running experiments on a laboratory testbed. Results show that the proposed strategy is able to detect adversaries using non-parametric methods, but it is not equally effective against adversaries using parametric identification methods.

This article investigates several network-assisted streaming approaches that rely on active cooperation between video streaming applications and the network. We build a Video Control Plane that enforces Video Quality Fairness among concurrent video flows generated by heterogeneous client devices. For this purpose, a max-min fairness optimization problem is solved at runtime. We compare two approaches to actuate the optimal solution in an Software Defined Networking network: The first one allocates network bandwidth slices to video flows, and the second one guides video players in the video bitrate selection. We assess performance through several QoE-related metrics, such as Video Quality Fairness, video quality, and switching frequency. The impact of client-side adaptation algorithms is also investigated.

Applications requiring real-time communication (RTC) between Internet peers are ever increasing. RTC requires not only congestion control but also minimization of queuing delays to provide interactivity. It is known that the well-established transmission control protocol congestion control is not suitable for RTC due to its retransmissions and in-order delivery mechanisms, which induce significant latency. In this paper, we propose a novel congestion control algorithm for RTC, which is based on the main idea of estimating-using a Kalman Filter-the end-to-end one-way delay variation which is experienced by packets traveling from a sender to a destination. This estimate is compared with a dynamic threshold and drives the dynamics of a controller located at the receiver, which aims at maintaining queuing delays low, while a loss-based controller located at the sender acts when losses are detected. The proposed congestion control algorithm has been adopted by Google Chrome. Extensive experimental evaluations have shown that the algorithm contains queuing delays while providing intra and inter protocol fairness along with full link utilization.

We address detection of attacks against cyber-physical systems. Cyber-physical systems are industrial control systems upgraded with novel computing, communication and interconnection capabilities. In this paper we reexamine the security of a detection scheme proposed by Mo and Sinopoli (2009) and Mo et al. (2015). The approach complements the use of Kalman filters and linear quadratic regulators, by adding an authentication watermark signal for the detection of integrity attacks. We show that the approach only detects cyber adversaries, i.e., attackers with the ability to eavesdrop information from the system, but that do not attempt to acquire any knowledge about the system model itself. The detector fails at covering cyber-physical adversaries, i.e., attackers that, in addition to the capabilities of the cyber adversary, are also able to infer the system model to evade the detection. We discuss an enhanced scheme, based on a multi-watermark authentication signal, that properly detects the two adversary models.

Video conferencing applications require low latency and high bandwidth. Standard TCP is not suitable for video conferencing since its reliability and in order delivery mechanisms induce large latency. Recently the idea of using the delay gradient to infer congestion is appearing again and is gaining momentum. In this paper we present an algorithm that is based on estimating through a Kalman filter the end-to-end one way delay variation which is experienced by packets traveling from a sender to a destination. This estimate is compared to an adaptive threshold to dynamically throttle the sending rate. The control algorithm has been implemented over the RTP/RTCP protocol and is currently used in Google Hangouts and in the Chrome WebRTC stack. Experiments have been carried out to evaluate the algorithm performance in the case of variable link capacity, presence of heterogeneous or homogeneous concurrent traffic, and backward path traffic.

In this paper we investigate several network-assisted streaming approaches which rely on active cooperation between video streaming applications and the network. We build a Video Control Plane which enforces Video Quality Fairness among concurrent video flows generated by heterogeneous client devices. To the purpose, a max-min fairness optimization problem is solved at run-time. We compare two approaches to actuate the optimal solution in an SDN network: the first one allocating network bandwidth slices to video flows, the second one guiding video players in the video bitrate selection. Performance is assessed through several QoE-related metrics, such as Video Quality Fairness, video quality, and switching frequency. The impact of client-side adaptation algorithms is also investigated.

Adaptive video streaming is a relevant advancement with respect to classic progressive download streaming a la YouTube. Among the different approaches, the video stream-switching technique is getting wide acceptance, being adopted by Microsoft, Apple, and popular video streaming services such as Akamai, Netflix, Hulu, Vudu, and Livestream. In this paper, we present a model of the automatic video stream-switching employed by one of these leading video streaming services along with a description of the client-side communication and control protocol. From the control architecture point of view, the automatic adaptation is achieved by means of two interacting control loops having the controllers at the client and the actuators at the server: One loop is the buffer controller, which aims at steering the client playout buffer to a target length by regulating the server sending rate; the other one implements the stream-switching controller and aims at selecting the video level. A detailed validation of the proposed model has been carried out through experimental measurements in an emulated scenario.

The existence of excessively large and too filled network buffers, known as bufferbloat, has recently gained attention as a major performance problem for delay-sensitive applications. One important network scenario where bufferbloat may occur is cellular networks. This paper investigates the interaction between TCP congestion control and buffering in cellular networks. Extensive measurements have been performed in commercial 3G, 3.5G and 4G cellular networks, with a mix of long and short TCP flows using the CUBIC, NewReno and Westwood+ congestion control algorithms. The results show that the completion times of short flows increase significantly when concurrent long flow traffic is introduced. This is caused by increased buffer occupancy from the long flows. In addition, for 3G and 3.5G the completion times are shown to depend significantly on the congestion control algorithms used for the background flows, with CUBIC leading to significantly larger completion times.

Multimedia content feeds an ever increasing fraction of the Internet traffic. Video streaming is one of the most important applications driving this trend. Adaptive video streaming is a relevant advancement with respect to classic progressive download streaming such as the one employed by YouTube. It consists in dynamically adapting the content bitrate in order to provide the maximum Quality of Experience, given the current available bandwidth, while ensuring a continuous reproduction. In this paper we propose a Quality Adaptation Controller (QAC) for live adaptive video streaming designed by employing feedback control theory. An experimental comparison with Akamai adaptive video streaming has been carried out. We have found the following main results: 1) QAC is able to throttle the video quality to match the available bandwidth with a transient of less than 30s while ensuring a continuous video reproduction; 2) QAC fairly shares the available bandwidth both in the cases of a concurrent TCP greedy connection or a concurrent video streaming flow; 3) Akamai underutilizes the available bandwidth due to the conservativeness of its heuristic algorithm; moreover, when abrupt available bandwidth reductions occur, the video reproduction is affected by interruptions.

End-to-end rate-based congestion control algorithms are advocated for audio/video transport over the Internet instead of window-based protocols. Once the congestion controller has computed the sending rate, all rate-based algorithms proposed in the literature schedule packets to be sent spaced at intervals that are equal to the inverse of the desired sending rate. In this paper we show that such an implementation exhibits a fundamental flaw. In fact, scheduling the sending time of a packet is affected by significant uncertainty due to the fact that it is handled by the Operating System, which manages a CPU shared by other processes. To overcome this problem, the Rate Mismatch Controller (RMC) is designed aiming at counteracting the disturbance on the effective sending time due to the CPU time-varying load. Experimental results using Linux OS highlight the effectiveness of the proposed controller.

The TCP/IP stack has been extremely successful for reliable delivery of best-effort, time insensitive elastic type data traffic. Nowadays, the Internet is rapidly evolving to become an equally efficient platform for multimedia content delivery. Key examples of this evolution are, to name few, YouTube, Skype Audio/Video, IPTV, P2P video distribution such as Coolstreaming or Joost. While YouTube streams videos using the Transmission Control Protocol (TCP), applications that are time-sensitive such as Skype VoIP or Video Conferencing employ the UDP because they can tolerate small loss percentages but not delays due to TCP recovery of losses via retransmissions. Since the UDP does not implement congestion control, these applications must implement those functionalities at the application layer in order to avoid congestion and preserve network stability. In this paper we investigate Skype Video in order to discover to what extent this application is able to throttle its sending rate to match the unpredictable Internet bandwidth while preserving resource for co-existing best-effort TCP traffic.