Many Internet of Things (IoT) applications require a distributed architecture for decision making either because of a lack of a centralized system, failure-prone connectivity to a centralized system or because the imposed latency to contact such a system is too high for real-time applications. Often, these IoT applications fall in the domain of reinforcement learning (RL), e.g., autonomous robot navigation in smart factories and traffic signal control in smart cities. However, RL-based applications require a long learning time. To overcome this limitation and scale with the number of agents, parallel RL (PRL) algorithms run multiple RL agents in parallel and on distributed environments. However, deploying PRL algorithms in such environments entails a communication overhead that increases the (actual) execution time. The state-of-the-art PRL algorithms are designed for reducing the learning time while assuming no (or limited) communication overhead. In this article, we present a novel partitioning algorithm that minimizes the communication overhead in PRL running on IoT environments. To the best of our knowledge, this is the first work that focuses on solving the communication overhead of distributing PRL algorithms without requiring any a priori knowledge about the structure of the problem. The proposed algorithm intelligently combines a dynamic state partitioning strategy, which exploits the agent's exploration capabilities to build partition knowledge while learning, with an efficient mapping of agents to partitions, which reduces the communication among agents. Performance evaluations show that the proposed algorithm can achieve almost no communication among PRL agents at the converged state.

In the unlicensed spectrum, many wireless technologies (e.g Wi-Fi, Bluetooth) use the same spectrum for wireless transmission. This often results in cross-technology interference effects, which are hard to address. Without new methods to manage this shared spectrum, wireless communication is increasingly challenging as too many nodes attempt at accessing the same spectrum. Collaboration between different wireless networks that use the same spectrum will be required to handle this massive amount of devices. In this paper, we present two algorithms based on Neural Networks (NNs) to demonstrate that a function approximation can accurately predict free slots in a Multiple Frequencies Time Division Multiple Access (MF-TDMA) network. By observing the spectrum, we are able to do online learning and let the corresponding NN predict the behavior of the spectrum a second in advance using our approach. We are able to reduce the number of collisions by half if the nodes from other networks are sending data following a Poisson distribution. When the nodes of the other network follow a more periodic traffic pattern, a collision reduction of factor 15 could be achieved.

HTTP Adaptive Streaming (HAS) has become the de facto standard technology for the delivery of video streaming services. Current adaptation heuristics for HAS focus on the selection of the optimal quality representation to be delivered from a single server. However, many content providers use multiple content servers storing replicas of the segmented video or are deployed over Content Delivery Networks (CDNs). Hence, the problem is not limited to selecting the optimal quality but also consists in requesting the segments from the best performing video server. In this paper a dynamic server selection strategy is proposed that enables the streaming client to select the optimal video delivery server. The proposed mechanism allows any quality adaptation algorithm to be plugged into it. The selection algorithm uses probability-based search strategies to explore the search space of available servers and to gain insights in their characteristics. This prevents the selection strategy to end up in a local optimum. To avoid buffer starvations, the exploration behavior is dependent on the current buffer filling. The proposed approach allows to achieve a Quality of Experience (QoE) that is within 25% of the optimum for which the client has a priori knowledge of the server characteristics.

Video streaming services have continuously gained popularity over the last decades, accounting for about 70% of all consumer Internet traffic in 2016. All of these video streaming sessions have strict delivery deadlines in order to avoid playout interruptions, detrimentally impacting the Quality of Experience (QoE). However, the vast majority of this traffic uses TCP at the transport layer, which is known to be far from minimizing the number of deadline-missing streams. By introducing deadline-awareness at the transport layer, video delivery can be optimized by prioritizing specific flows. This paper proposes a deadline-aware congestion control mechanism, based on a parametrization of the traditional TCP New Reno congestion control strategy. By taking into account the available deadline information, the modulation of the congestion window is dynamically adapted to steer the aggressiveness of a considered stream. The proposed approach has been thoroughly evaluated in both a video-on-demand (VoD)-only scenario and a scenario where VoD streams co-exist with live streaming sessions and non-deadline-aware traffic. It was shown that in a video streaming scenario the minimal bottleneck bandwidth can be reduced by 16% on average when using deadline-aware congestion control. In coexistence with other TCP traffic, a bottleneck reduction of 11% could be achieved.

HTTP Adaptive Streaming (HAS) is becoming the de-facto standard for video streaming services over the Internet. In HAS, each video is segmented and stored in different qualities. Rate adaptation heuristics, deployed at the client, allow the most appropriate quality level to be dynamically requested based on the current network conditions, in order to achieve a continuous playout. Due to the ability of HAS protocols to dynamically adapt to bandwidth fluctuations, they are especially suited for the delivery of multimedia content in mobile environments. However, current HAS solutions do not take the battery lifetime into account, which is a typical issue for mobile devices. In this paper, we therefore propose an energy-aware heuristic for HAS. We first present a measurement study to identify and quantify the main factors influencing the battery lifetime on mobile devices. We then develop a heuristic based on these findings, which optimizes both the quality of experience and the battery consumption of a video streaming session. Particularly, we found that the video resolution and display size have the highest impact on the battery lifetime and that our energy-aware heuristic can prolong a streaming session with up to 13%, compared to a standard HAS heuristic. This result represents a consistent improvement for the overall user experience on battery-constrained devices.

Content delivery networks (CDNs) have been the prevalent method for the efficient delivery of content across the Internet. Management operations performed by CDNs are usually applied only based on limited information about Internet Service Provider (ISP) networks, which can have a negative impact on the utilization of ISP resources. To overcome these issues, previous research efforts have been investigating ISP-operated content delivery services, by which an ISP can deploy its own in-network caching infrastructure and implement its own cache management strategies. In this paper, we extend our previous work on ISP-operated content distribution and develop a novel scalable and efficient distributed approach to control the placement of content in the available caching points. The proposed approach relies on parallelizing the decision-making process and the use of network partitioning to cluster the distributed decision-making points, which enables fast reconfiguration and limits the volume of information required to take reconfiguration decisions. We evaluate the performance of our approach based on a wide range of parameters. The results demonstrate that the proposed solution can outperform previous approaches in terms of management overhead and complexity while offering similar network and caching performance.

Recently, video-on-demand (VoD) streaming services like Netflix and Hulu have gained a lot of popularity. This has led to a strong increase in bandwidth capacity requirements in the network. To reduce this network load, the design of appropriate caching strategies is of utmost importance. Based on the fact that, typically, a video stream is temporally segmented into smaller chunks that can be accessed and decoded independently, cache replacement strategies have been developed that take advantage of this temporal structure in the video. In this paper, two caching strategies are proposed that additionally take advantage of the phenomenon of binge watching, where users stream multiple consecutive episodes of the same series, reported by recent user behavior studies to become the everyday behavior. Taking into account this information allows us to predict future segment requests, even before the video playout has started. Two strategies are proposed, both with a different level of coordination between the caches in the network. Using a VoD request trace based on binge watching user characteristics, the presented algorithms have been thoroughly evaluated in multiple network topologies with different characteristics, showing their general applicability. It was shown that in a realistic scenario, the proposed election-based caching strategy can outperform the state-of-the-art by 20% in terms of cache hit ratio while using 4% less network bandwidth.

HTTP Adaptive Streaming (HAS) is quickly becoming the de facto standard for video streaming services. In HAS, each video is temporally segmented and stored in different quality levels. Rate adaptation heuristics, deployed at the video player, allow the most appropriate level to be dynamically requested, based on the current network conditions. It has been shown that today’s heuristics underperform when multiple clients consume video at the same time, due to fairness issues among clients. Concretely, this means that different clients negatively influence each other as they compete for shared network resources. In this article, we propose a novel rate adaptation algorithm called FINEAS (Fair In-Network Enhanced Adaptive Streaming), capable of increasing clients’ Quality of Experience (QoE) and achieving fairness in a multiclient setting. A key element of this approach is an in-network system of coordination proxies in charge of facilitating fair resource sharing among clients. The strength of this approach is threefold. First, fairness is achieved without explicit communication among clients and thus no significant overhead is introduced into the network. Second, the system of coordination proxies is transparent to the clients, that is, the clients do not need to be aware of its presence. Third, the HAS principle is maintained, as the in-network components only provide the clients with new information and suggestions, while the rate adaptation decision remains the sole responsibility of the clients themselves. We evaluate this novel approach through simulations, under highly variable bandwidth conditions and in several multiclient scenarios. We show how the proposed approach can improve fairness up to 80% compared to state-of-the-art HAS heuristics in a scenario with three networks, each containing 30 clients streaming video at the same time.

The growing popularity of over the top (OTT) video streaming services has led to a strong increase in bandwidth capacity requirements in the network. By deploying intermediary caches, closer to the end-users, popular content can be served faster and without increasing backbone traffic. Designing an appropriate replacement strategy for such caching networks is of utmost importance to achieve high caching efficiency and reduce the network load. Typically, a video stream is temporally segmented into smaller chunks that can be accessed and decoded independently. This temporal segmentation leads to a strong relationship between consecutive segments of the same video. Therefore, caching strategies have been developed, taking into account the temporal structure of the video. In this paper, we propose a novel caching strategy that takes advantage of clients announcing which videos will be watched in the near future, e.g., based on predicted requests for subsequent episodes of the same TV show. Based on a Video-on-Demand (VoD) production request trace, the presented algorithm is evaluated for a wide range of user behavior and request announcement models. In a realistic scenario, a performance increase of 11% can be achieved in terms of hit ratio, compared to the state-of-the-art.

HTTP Adaptive Streaming (HAS) is becoming the de-facto standard for Over-The-Top video streaming. A HAS video consists of multiple segments, encoded at multiple quality levels. Allowing the client to select the quality level for every segment, a smoother playback and a higher Quality of Experience (QoE) can be perceived. Although results are promising, current quality selection heuristics are generally hard coded. Fixed parameter values are used to provide an acceptable QoE under all circumstances, resulting in suboptimal solutions. Furthermore, many commercial HAS implementations focus on a video-on-demand scenario, where a large buffer size is used to avoid play-out freezes. When the focus is on a live TV scenario however, a low buffer size is typically preferred, as the video play-out delay should be as low as possible. Hard coded implementations using a fixed buffer size are not capable of dealing with both scenarios. In this paper, the concept of reinforcement learning is introduced at client side, allowing to adaptively change the parameter configuration for existing rate adaptation heuristics. Bandwidth characteristics are taken into account in the decision process, thus allowing to improve the client's bandwidth-awareness. Focus in this paper is on actively reducing the average buffer filling, evaluating results for two heuristics: the Microsoft IIS Smooth Streaming heuristic and the QoE-driven Rate Adaptation Heuristic for Adaptive video Streaming by Petrangeli et al. We show that using the proposed learning-based approach, the average buffer filling can be reduced by 8.3% compared to state of the art, while achieving a comparable level of QoE.

Over the past decades, the importance of multimedia services such as video streaming has increased considerably. Since streaming protocols such as Real Time Streaming Protocol (RTSP) and Real Time Transport Protocol (RTP) require server-side bit-rate adaptation schemes, they are not ideally suited to deal with highly heterogeneous and dynamically changing network conditions. Therefore, research shifted towards client-side adaptation schemes, requiring significantly less investments in server-side infrastructure. HTTP Adaptive Streaming (HAS) is now becoming omnipresent in video streaming services due to many advantages offered by HTTP-based streaming: reliable transmission over TCP, reuse of existing caching infrastructure and compatibility with NATs and firewalls. In HAS, the video content is split temporally into segments which are encoded at different quality rates. The client side heuristic decides at which quality rate each segment should be downloaded, based on measured network statistics, buffer filling level and device characteristics. Traditionally, Advanced Video Coding (AVC) is used to encode the different segments, introducing a significant amount of redundancy across quality representations. Scalable Video Coding (SVC) can cope with these issues of content redundancy by creating dependencies between the base and enhancement layers. Adopting SVC in HAS significantly improves caching and bandwidth efficiency at the server side. Another advantage of SVC, is the ability to gradually upgrade the quality of the video by downloading additional video layers.

HTTP Adaptive Streaming (HAS) has several advantages compared to traditional streaming protocols, such as easy traversal of firewalls and reuse of widely deployed HTTP infrastructure. HAS content is temporally segmented, and encoded at different quality representations, allowing the video player to autonomously adapt to network conditions by adapting play-out quality between subsequent segment downloads. However, to guarantee continuous playback, current-generation HAS protocols require a large play-out buffer. This makes them ill-suited for live television, as it significantly increases the live signal delay. This paper proposes a novel HAS solution for live streaming services. A HAS video player was designed that can cope with buffers as small as 2 seconds. This obviously requires the player to more rapidly react to bandwidth changes, which was achieved by using the Scalable Video Coding (SVC) extension of the H.264 Advanced Video Coding (AVC) video codec. Moreover, an intelligent network proxy was developed that guarantees the delivery of the SVC base quality layer using Differentiated Services (DiffServ). Furthermore, a more dynamic deadline-based approach is proposed which allows the client itself to decide which segments should be prioritized based on the risk of running into a buffer starvation. This enables more efficient use of the prioritized channel, leading to less freezes and increased quality and stability. The combination of these technologies allows the video player to align its quality adaptation decisions to the available bandwidth more efficiently and completely avoid buffer starvations. The small buffer size also reduces the total live signal delay from multiple dozens to only a few seconds.

In recent years, HTTP Adaptive Streaming (HAS) is becoming the de facto standard for video delivery over the best effort Internet. In HAS, the video consists out of multiple temporal segments encoded at different quality rates. In this way, HAS allows to dynamically adapt the quality level to the perceived network conditions. Using Scalable Video Coding (SVC), the redundancy between these representations can be eliminated, increasing the efficiency of server and caching infrastructure. Software Defined Networking (SDN) allows the dynamic adjustment of forwarding tables to reroute different flows. Using a combination of the layered characteristics of SVC and the dynamic routing of flows, the delivery of video can be optimized. In this paper, an algorithm is presented to dynamically calculate the optimal delivery paths for the different video layers. This enables guaranteeing a reliable and continuous video playout. Using this approach the number of video freezes can be reduced with 72% compared to shortest path routing.

Network virtualisation is considerably gaining attention as a solution to ossification of the Internet. However, the success of network virtualisation will depend in part on how efficiently the virtual networks utilise substrate network resources. In this paper, we propose a machine learning-based approach to virtual network resource management. We propose to model the substrate network as a decentralised system and introduce a learning algorithm in each substrate node and substrate link, providing self-organization capabilities. We propose a multiagent learning algorithm that carries out the substrate network resource management in a coordinated and decentralised way. The task of these agents is to use evaluative feedback to learn an optimal policy so as to dynamically allocate network resources to virtual nodes and links. The agents ensure that while the virtual networks have the resources they need at any given time, only the required resources are reserved for this purpose. Simulations show that our dynamic approach significantly improves the virtual network acceptance ratio and the maximum number of accepted virtual network requests at any time while ensuring that virtual network quality of service requirements such as packet drop rate and virtual link delay are not affected.

HTTP Adaptive Streaming (HAS) is quickly becoming the de facto standard for Over-The-Top video streaming. In HAS, each video is temporally segmented and stored in different quality levels. Quality selection heuristics, deployed at the video player, allow dynamically requesting the most appropriate quality level based on the current network conditions. Today's heuristics are deterministic and static, and thus not able to perform well under highly dynamic network conditions. Moreover, in a multi-client scenario, issues concerning fairness among clients arise, meaning that different clients negatively influence each other as they compete for the same bandwidth. In this article, we propose a Reinforcement Learning-based quality selection algorithm able to achieve fairness in a multi-client setting. A key element of this approach is a coordination proxy in charge of facilitating the coordination among clients. The strength of this approach is three-fold. First, the algorithm is able to learn and adapt its policy depending on network conditions, unlike current HAS heuristics. Second, fairness is achieved without explicit communication among agents and thus no significant overhead is introduced into the network. Third, no modifications to the standard HAS architecture are required. By evaluating this novel approach through simulations, under mutable network conditions and in several multi-client scenarios, we are able to show how the proposed approach can improve system fairness up to 60% compared to current HAS heuristics.

In recent years, HTTP (Hypertext Transfer Protocol) adaptive streaming (HAS) has become the de facto standard for adaptive video streaming services. A HAS video consists of multiple segments, encoded at multiple quality levels. State-of-the-art HAS clients employ deterministic heuristics to dynamically adapt the requested quality level based on the perceived network conditions. Current HAS client heuristics are, however, hardwired to fit specific network configurations, making them less flexible to fit a vast range of settings. In this article, a (frequency adjusted) Q-learning HAS client is proposed. In contrast to existing heuristics, the proposed HAS client dynamically learns the optimal behaviour corresponding to the current network environment in order to optimise the quality of experience. Furthermore, the client has been optimised both in terms of global performance and convergence speed. Thorough evaluations show that the proposed client can outperform deterministic algorithms by 11–18% in terms of mean opinion score in a wide range of network configurations.