Reinforcement learning (RL) has been increasingly applied to network control problems, such as load balancing. However, existing RL approaches often suffer from lack of interpretability and difficulty in extracting controller equations. In this paper, we propose the use of Kolmogorov-Arnold Networks (KAN) for interpretable RL in network control. We employ a PPO agent with a 1-layer actor KAN model and an MLP Critic network to learn load balancing policies that maximise throughput utility, minimize loss as well as delay. Our approach allows us to extract controller equations from the learned neural networks, providing insights into the decision-making process. We evaluate our approach using different reward functions demonstrating its capability in improving network performance while providing interpretable policies.

Deep Reinforcement Learning (DRL) algorithms have recently made significant strides in improving network performance. Nonetheless, their practical use is still limited in the absence of safe exploration and safe decision-making. In the context of commercial solutions, reliable and safe-to-operate systems are of paramount importance. Taking this problem into account, we propose a safe learning-based load balancing algorithm for Software Defined-Wide Area Network (SD-WAN), which is empowered by Deep Reinforcement Learning (DRL) combined with a Control Barrier Function (CBF). It safely projects unsafe actions into feasible ones during both training and testing, and it guides learning towards safe policies. We successfully implemented the solution on GPU to accelerate training by approximately 110x times and achieve model updates for on-policy methods within a few seconds, making the solution practical. We show that our approach delivers near-optimal Quality-of-Service (QoS) performance in terms of end-to-end delay while respecting safety requirements related to link capacity constraints. We also demonstrated that on-policy learning based on Proximal Policy Optimization (PPO) performs better than off-policy learning with Deep Deterministic Policy Gradient (DDPG) when both are combined with a CBF for safe load balancing.

In SD-WAN networks, the traffic issued from multiple high capacity hubs towards low capacity spokes can create congestions in the underlay and degrade unnecessarily the Quality of Service (QoS). To mitigate this issue, shaping policies can be dynamically controlled to adapt to WAN performance and traffic. While existing solutions work for a single hub, at tunnel level, and in a reactive manner, we propose a more advanced solution for multiple hubs at application level. Our Dynamic QoS solution also takes proactive actions to prevent congestions and protect high priority traffic. It ensures a fast convergence towards an optimal rate allocation. This demonstration presents the design and implementation of this feature in AR8140 devices. It presents a performance evaluation in a testbed and simulation.

In modern SD-WAN networks, a global controller is able to steer traffic on different paths based on application requirements and global intents. However, existing solutions cannot dynamically tune the way bandwidth is shared between flows inside each network, in particular when the available capacity is uncertain due to cross traffic. In this context, we propose a global QoS (Quality of Service) policy optimization model that dynamically adjusts rate limits of applications based on their requirements to follow the evolution of network conditions. It relies on a novel cross-traffic estimator for the available bandwidth of overlay links that only exploits already available measurements. We propose a centralized local search algorithm with cross-traffic estimation and show in packet-level simulations a significant performance improvement in terms of SLA (Service Level Agreement) satisfaction. The adaptive tuning of load balancing and QoS policies based on cross-traffic estimation can improve SLA satisfaction by 40% compared to static policies.

This paper proposes an Attention-based Multi-Agent Cooperation (AMAC) approach to reduce message exchange overhead in Multi-Agent Reinforcement Learning-based smart load balancing. AMAC shares only most relevant messages across agents to coordinate decision-making without degrading original performance. Experiments show that AMAC significantly lowers inter-agent communications overhead and learning complexity and outperforms multiple MARL benchmarks in Key Performance Indicators (KPIs) and Key Quality Indicators (KQIs).

This paper explores the use of multi-agent deep learning as well as learning to cooperate principles to meet strict service level agreements, in terms of throughput and end-to-end delay, for a set of classified network flows. We consider agents built on top of a weighted fair queuing algorithm that continuously set weights for three flow groups: gold, silver, and bronze. We rely on a novel graph-convolution based, multi-agent reinforcement learning approach known as DGN. As benchmarks, we propose centralized and distributed deep Q-network algorithms and evaluate their performances in different network, traffic, and routing scenarios, highlighting both the effectiveness of our proposals and the importance of agent cooperation. We show that our DGN-based approach meets stringent throughput and delay requirements across different scenarios, decreasing silver and bronze flow median waiting delays by more than 50 % and reducing the SLA violations of the latter by nearly 60 %, with respect to a classic priority queuing approach.

In this paper, we explore the use of an actor-critic architecture for Deep Reinforcement Learning (DRL) to improve load balancing beyond traditional algorithms. Some centralized Reinforcement Learning (RL) algorithms have targeted in the reward function expression the Quality of Experience (QoE) for video flows, but this requires access to clients, or the Maximum Link Utilization (MLU) for other types of flows. In our approach, we tune the actor-critic algorithm to only leverage on QoS parameters in order to load balance traffic in the network and maximize the QoE experienced by the users. This avoids having to collect observations and performance measurements from client applications, as it only focuses on network metrics that can be easily measured. We explore both centralized and distributed solutions to assess the feasibility of the proposed smart load balancing solutions. We compare them to ECMP, QoE-based reward methods, and RILNET that uses an underlying DDPG optimization approach. The proposed algorithms are shown to outperform previous approaches.

To optimize bandwidth utilization in wide area networks, a centralized controller typically maintains routing policies at edge routers. In this context, we propose a versatile intent-based policy optimization model that carefully selects the set of overlay links which are allowed for applications based on their requirements and the overall intents of the operator. The optimization model embeds QoS and traffic predictions to anticipate the impact of routing decisions. To address large scale scenarios where the behavior of the network and devices is not known exactly, we integrate data-driven predictions into a local search algorithm to optimize routing policies. The algorithm supports several intents such as the minimization of the congestion or the maximization of the network quality. Thanks to packet-level simulations on an SD-WAN scenario, we show that our intent-based policy optimization system improves significantly performances. For instance, the latency is improved by 40% when the high-quality intent is selected. In addition, the percentage of time SLAs are met is improved by 10% compared to legacy load balancing mechanisms.

To improve bandwidth utilization in IP networks, flow aggregates are typically split over multiple paths. In this context, we propose a fully distributed load balancing mechanism that operates only from the edge. Each source is able to determine the split ratios based on already available link state information so as to minimize the maximum link utilization in the network. Without extra signaling, our solution provides a feasible load balancing at each iteration and diminishing returns until convergence to a stable state. Through numerical results on a wide variety of instances, we show that it converges to a near-optimal solution in a few iterations. Thanks to packet-level simulations on an SD-WAN scenario, we also compare its performance in a dynamic environment over centralized and legacy load balancing solutions.

Fifth generation and beyond networks are envisioned to support multi industrial Internet of Things (IIoT) applications with a diverse quality-of-service (QoS) requirements. Network slicing is recognized as a flagship technology that enables IIoT networks with multiservices and resource requirements by allowing the network-as-infrastructure transition to the network-as-service. Motivated by the increasing IIoT computational capacity, and taking into consideration the QoS satisfaction and private data sharing challenges, federated reinforcement learning (RL) has become a promising approach that distributes data acquisition and computation tasks over distributed network agents, exploiting local computation capacities and agent's self-learning experiences. This article proposes a novel deep RL scheme to provide a federated and dynamic network management and resource allocation for differentiated QoS services in future IIoT networks. This involves IIoT slices resource allocation in terms of transmission power (TP) and spreading factor (SF) according to the slices QoS requirements. Toward this goal, the proposed deep federated Q-learning (DFQL) is reached into two main steps. First, we propose a multiagent deep Q-learning-based dynamic slices TP and SF adjustment process that aims at maximizing self-QoS requirements in term of throughput and delay. Second, the deep federated learning is proposed to learn multiagent self-model and enable them to find an optimal action decision on the TP and the SF that satisfy IIoT virtual network slice QoS reward, exploiting the shared experiences between agents. Simulation results show that the proposed DFQL framework achieves efficient performance compared to the traditional approaches.

The placement of Virtual Network Function - Forwarding Graphs (VNF-FGs) is one of the basic operations in the networks of the future. Being NP-hard, several heuristics and metaheuristics have been proposed. However, these approaches are inefficient due to the need to recalculate the solution at each service placement. In this paper, we adapt one of the most advanced approaches in Deep Reinforcement Learning (DRL), in order to improve exploration by generalizing the neural network calculating action values. We also propose an evolutionary algorithm to evolve these neural networks in order to discover better ones, which also avoids getting stuck in local minima. In order to avoid going through the almost innumerable number of infeasible solutions, we propose a heuristic, which combined with our DRL, makes it possible to guarantee the feasibility of the solutions and therefore to make the placement much more efficient. The simulation results we obtained confirm the quality of the solutions obtained as well as the superiority of the proposed solution over the existing one.

Resource allocation of 5G network slices is one of the most important challenges for network operators. It can be formulated using the Virtual Network Embedding (VNE) problem, which was and remains an active field of studies, also known because of its NP-hardness. Owing to its complexity, several heuristics, meta-heuristics and Deep Learning-based solutions have been proposed. However, these solutions are inefficient either due to their slowness or to not taking into account the structure of data which results in an inefficient exploration of the solutions space. To overcome these issues, in this work we unveil the potential of Graph Convolutional Neural (GCN) networks and Deep Reinforcement Learning techniques in solving the VNE problem. The key point of our approach is modeling of the VNE problem as an episodic Markov Decision Process which is solved in a Reinforcement Learning fashion using a GCN-based neural architecture. The simulation results highlight the efficiency of our approach through an increased performance over time, while outperforming state-of-art solutions in terms of the services' acceptance ratio.

Network Function Virtualization (NFV) and service orchestration simplify the deployment and management of network and telecommunication services. The deployment of these services requires, typically, the allocation of Virtual Network Function - Forwarding Graph (VNF-FG), which implies not only the fulfillment of the service's requirements in terms of Quality of Service (QoS), but also considering the constraints of the underlying infrastructure. This topic has been well-studied in existing literature, however, its complexity and uncertainty of available information unveil challenges for researchers and engineers. In this paper, we explore the potential of reinforcement learning techniques for the placement of VNF-FGs. However, it turns out that even the most well-known learning technique is ineffective in the context of a large-scale action space. In this respect, we propose approaches to find out feasible solutions while improving significantly the exploration of the action space. The simulation results clearly show the effectiveness of the proposed learning approach for this category of problems. Moreover, thanks to the deep learning process, the performance of the proposed approach is improved over time.

Network function virtualization (NFV) will simplify deployment and management of network and telecommunication services. NFV provides flexibility by virtualizing the network functions and moving them to a virtualization platform. In order to achieve its full potential, NFV is being extended to mobile or wireless networks by considering virtualization of radio functions. A typical network service setup requires the allocation of a virtual network function-forwarding graph (VNF-FG). A VNF-FG is allocated considering the resource constraints of the lower infrastructure. This topic has been well-studied in existing literature, however, the effects of variations of networks over time have not been addressed yet. In this paper, we provide a model of the adaptive and dynamic VNF allocation problem considering also VNF migration. Then we formulate the optimization problem as an integer linear programming (ILP) and provide a heuristic algorithm for allocating multiple VNF-FGs. The idea is that VNF-FGs can be reallocated dynamically to obtain the optimal solution over time. First, a centralized optimization approach is proposed to cope with the ILP-resource allocation problem. Next, a decentralized optimization approach is proposed to deal with cooperative multi-operator scenarios. We adopt AD3, an alternating direction method of multipliers-based algorithm, to solve this problem in a distributed way. The results confirm that the proposed algorithms are able to optimize the network utilization, while limiting the number of reallocations of VNFs which could interrupt network services.

Network Function Virtualization (NFV) is emerging as an efficient mean to deploy and manage network and telecommunication services. With wireless access networks, NFV has to take into account the radio resources at wireless nodes in order to provide an end-to-end optimal virtual network function (VNF) allocation. This topic has been well-studied in existing literature, however, the effects of variations of networks over time have not been addressed yet. In this paper, we provide a model of the adaptive and dynamic VNF allocation problem considering VNF migration. Moreover, we also consider service function chains (SFCs) with QoS constraints. Then we formulate the optimisation problem as an Integer Linear Programming (ILP) and provide a heuristic algorithm for allocating multiple SFCs. The proposed approach allows SFCs to be reallocated so as to obtain the optimal solution over time. The results confirm that the proposed algorithm is able to optimize the network utilization while limiting the reallocation of VNFs which could interrupt services.

Network Function Virtualization (NFV) is deemed as a mean to simplify deployment and management of network and telecommunication services. With wireless access networks, NFV has to take into account the radio resources at wireless nodes in order to provide an end-to-end optimal virtual network function (VNF) allocation. This topic has been well-studied in existing literature, however, the effects of variations of networks over time have not been addressed yet. In this paper, we provide a model of the adaptive and dynamic VNF allocation problem considering VNF migration. Then we formulate the optimisation problem as an Integer Linear Programming (ILP) and provide a heuristic algorithm for allocating multiple service function chains (SFCs). The proposed approach allows SFCs to be reallocated so as to obtain the optimal solution over time. The results confirm that the proposed algorithm is able to optimize the network utilization while limiting the reallocation of VNFs which could interrupt services.

Relaying packets through multi-hop ad-hoc networks and the scarcity of wireless resources can severely deteriorate the quality of service. As a result, one of the major challenges in video streaming over ad-hoc networks is enhancing users experience and network utilization. An extension of H.264 standard, named scalable video coding (SVC), enables smooth adaptation of video quality to networks' status. In this paper, we formulate the problem of optimizing quality of experience (QoE) for SVC under time-constraints. The problem itself is a mixed integer linear programming which is NP-hard, and therefore we propose a heuristic algorithm to solve it. Simulation results show that the proposed algorithm can provide the similar video quality as optimal solutions with shorter calculation time.

This paper presents a QoE-based multipath routing algorithm for scalable video coding (SVC) over multichannel wireless mesh networks (WMN). The availability and quality of links in WMN significantly depends on not only the positions of neighbors but also the interference between links. In video streaming, the quality perceived by the users is the key to success. Consequently, enhancing experience of users and network utilization in these networks is an interesting challenge. In this paper, we utilize multicommodity model to formulate the problem of streaming SVC video over multichannel WMN. Then, we propose a heuristic algorithm to speed up searching solution procedure. Simulation results demonstrate that the proposed algorithm can obtain a near-optimal solution within a short calculation time.

The major challenge in ad-hoc network is multi-hop paradigm. Moreover, multimedia streaming over ad-hoc networks increasingly emerges since it can be useful in various practical cases. In video streaming, previous studies provided quality of service (QoS)-based methods that focused on the value of technical parameters such as bandwidth, jitter, delay, etc. It is however not perfectly correlated to users' experience. In this paper, we propose a routing mechanism based on optimized link state routing (OLSR) to guarantee a good quality of experience (QoE) of users. The pseudo-subjective quality assessment (PSQA) is adopted to estimate mean opinion scores (MOS), then this MOS value will be exploited by the source for selecting the appropriate path. In addition, an event-triggered based on the MOS value is exploited to provide more relevant information. The results show that the proposed scheme outperforms other OLSR-based routing protocols particularly in heavy load and moderate mobility scenario.

In-vehicle multimedia applications are gaining interest since recent years. However, the high loss rate caused by high mobility in vehicular ad-hoc networks (VANETs) imposes several challenges in multimedia transmission. Moreover, in the context of multimedia, the quality of service (QoS)-based approaches assess the quality of streaming services through network-oriented metrics while the concept of quality of experience (QoE) is built upon the perception of users. Consequently, adopting QoE metric to control multimedia streams in VANETs is interesting and relevant. In this paper, a QoE-based routing protocol for video streaming over VANETs is proposed. By taking the mean opinion score (MOS) into account for path selection, a better performance can be achieved. Simulation results demonstrate how the proposed scheme improves the performance of video streaming applications in VANETs.

This paper proposes a routing algorithm that enhances throughput and decreases end-to-end delay in industrial cognitive radio sensor networks (ICRSNs) based on ISA100.11a. In ICRSNs, the throughput is downgraded by interference from primary networks. The proposed routing algorithm is targeted at large-scale networks where data are forwarded through different clusters on their way to the sink. By estimating the maximum throughput for each path, the data can be forwarded through the most optimal path. Simulation results show that our scheme can enhance throughput and decrease end-to-end delay.

This paper proposes gradient routing with two-hop information for industrial wireless sensor networks to enhance real-time performance with energy efficiency. Two-hop information routing is adopted from the two-hop velocity-based routing, and the proposed routing algorithm is based on the number of hops to the sink instead of distance. Additionally, an acknowledgment control scheme reduces energy consumption and computational complexity. The simulation results show a reduction in end-to-end delay and enhanced energy efficiency.