Multipath QUIC (MPQUIC) is an emerging multi-path transport protocol that lets a mobile client simultaneously use several wireless networks (e.g., Wi-Fi and cellular) in 5G and beyond. MPQUIC's performance heavily relies on its scheduler, which determines a path or several ones for sending packets in the upcoming time slot. Despite numerous efforts, the traditional design of MPQUIC schedulers can not handle wireless networks' dynamicity. Recently, a learning-based approach has shown the potential to bypass such limitations of the MPQUIC scheduler with various learning-based schedulers proposed in the literature. However, the existing works only consider the scheduling task in a single client context. When applying such a scheduler to multiple client scenarios (likely to occur in practice), they suffer from a so-called rush scheduling phenomenon. More specifically, the packet forwarding decisions made by a scheduler are only accountable to one client, resulting in conflicts of interest with other clients' schedulers. Consequently, it may harm the network performance. This paper addresses the issue and designs a learning-based MPQUIC scheduler considering the existence of multiple clients. To the best of our knowledge, this is the first work to do so. We propose MuLeS, a learning-based scheduler for MPQUIC in the multi-client scenario. MuLeS uses a central controller, which allows it to observe the state of all flows in the network. Our evaluation results show that MuLeS outperforms contemporary schedulers in terms of various metrics, including download time and loss rate. Notably, MuLeS reduces the average download time by 7%-16% compared to the other schedulers.

With the increasing demand for video streaming services, efficient video streaming with MPQUIC in mobile wireless networks is gaining interest. Although MPQUIC can leverage multiple network connections (e.g., Wi-Fi, 5G) for simultaneous transfer, improving bandwidth and resilience, its performance is heavily dependent on the MPQUIC scheduler. In mobile scenarios, network fluctuations challenge the scheduler to adapt to dynamic conditions while maintaining good video streaming quality. Addressing this issue, this paper proposes a novel MPQUIC scheduler named LoGra (i.e., LSTM-DDPG integrated MPQUIC Scheduler) with two advanced features. First, LoGra utilizes Long Short-Term Memory (LSTM) to model the temporal correlation of network conditions over time and handle the challenges posed by mobility patterns. Second, it leverages the Deep Deterministic Policy Gradient (DDPG), with its self-learning capabilities and the strength of deep neural networks, to analyze the informative temporal feature vector to influence scheduling decisions. We have implemented the proposed scheduler and compared it to existing ones. The results show that the LoGra scheduler effectively manages multipath communication in heterogeneous wireless networks under mobility scenarios. Moreover, compared to existing schedulers, LoGra achieves significant improvements in data transmission, both in general metrics (loss, goodput) and streaming-related metrics (bitrate, freezing time).

Realizing extreme reliability for Internet of Things (IoT) communications is one of the major milestones paving the way towards Beyond 5G (B5G) and 6G. In this work, we investigate the issue of improving the reliability of packet transmissions in the absence of prior knowledge of network statistics, nor of instantaneous Channel State Information (CSI), for B5G Sub-6GHz/mmWave integrated networks. Specifically, the aim is to maximize the global successful packet reception at devices, while guaranteeing their individual Packet Loss Rate (PLR) requirements. The proposed method exploits a newly developed approach of Risk-Averse Reinforcement Learning (RARL), for exploiting multi-connectivity over Sub-6Hz and mmWave interfaces. Namely, the Access Point (AP) is able to optimize its interface selection decisions despite the unknown dynamics of the wireless environment based on limited feedback from its associated devices, so as to increase reliability under low delay and resource consumption. Numerical results show that, the proposed method significantly improves the global reliability performance by rapidly learning and adapting its decisions as compared to baseline methods.

The joint provision of higher data rates and massive Internet of Things (IoT) connectivity has been identified as one of the key milestones toward beyond 5G (B5G). To this end, we investigate the issue of device selection and beamforming (BF) optimization assuming a large-scale IoT network using mmWaves. We formulate the considered problem as a network sum-rate maximization problem under Access Points’ load constraints, and where the BF parameters belong to discrete sets, as in practical cases. First, we mathematically prove the submodularity of the objective function, under specific yet reasonable assumptions. Based on the identified features of the problem at hand, we propose three different approaches to tackle this intricate optimization problem: 1) a Branch-and-Bound-based; 2) a Lagrangian Relaxation-based; and 3) a Greedy-based approach inspired by the submodular objective. The numerical results validate the three approaches, as they achieve a near-optimal sum rate in small network cases, and largely outperform benchmark schemes in terms of sum rate and individual rates. Among them, the proposed Greedy-based approach achieves the best sum rate with very low complexity, thereby providing excellent scalability.

In this work, we investigate the problem of joint user-to-access points (AP) association and beamforming in an integrated sub-6GHz/mmWave system. The goal is to maximize the long-term throughput of the system, while satisfying a large number of heterogeneous user QoS requirements in a distributed manner. We propose a method based on Deep Q-Networks (DQN), where each user self-optimizes its AP association and interface requests, and can be served by several APs simultaneously for supporting multiple applications. Based on these requests, each AP selects its associated users and applications served on each interface, while optimizing its mm Wave beamforming parameters. Simulation results show that, compared to baseline DQN schemes among which the Action Elimination (AE)-DQN, the proposed method enables to fine-tune the selection of APs and interfaces to the specific level of each required QoS, thereby achieving a high global throughput while notably reducing user outage probabilities1.1.This collaborative research project is funded by NTT Corporation, Japan.

This work investigates the design of a sustainable Deep Q-Network (DQN) implemented at the user device, whose purpose is to optimize the user’s association to multiple access points (AP) in a Beyond 5G (B5G) Sub-6GHz and mmWave integrated network. To better cope with dynamic mobile environments, we first propose an adaptive $\varepsilon$-greedy policy at each user’s DQN in order to maximize the long-term sum-rate while simultaneously satisfying the Quality of Service (QoS) constraints of different applications. We then provide the detailed analysis of the energy consumed by each user device, in particular the power for DQN processing and for data movement. The trade-off between network performance in terms of sum-rate and QoS outage probability, and energy consumption at the user side is evaluated. Numerical results not only show the effectiveness of the proposed method compared to baseline, but also reveal the tremendous energy costs required by the default user DQN, underscoring the paramount importance of the proposed trade-off aware user DQN design1.

Future wireless networks will be facing unprecedented difficulties arising from mobile traffic growth, network densification, as well as diversification of applications and services. Indeed, future user devices are expected to integrate diverse radio interfaces such as 5G, WBAN or IoT, enabling each user to be served a wide range of applications at any time. This poses significant challenges in terms of wireless resource sharing and interference management, as more and more stringent Quality of Service (QoS) constraints should be jointly satisfied in dense interfering environments. Furthermore, future networks are expected to be highly autonomous and decentralized. To meet these challenges, this work proposes distributed user-to-multiple Access Points (AP) association methods, where the objective is to maximize the long-term sum-rate subject to application QoS constraints, as well as to AP load constraints. Our distributed methods enable each user to leverage their Deep Reinforcement Learning (DRL) capabilities, in particular Deep Q-Learning (DQL), to self-optimize their APs’ selection solely based on their local network state knowledge, so as to best satisfy their diverse requirements. Numerical results show that, compared to baseline schemes, the proposed methods enable global throughput enhancements while reducing user QoS outage probabilities, even in large and dense networks.

Recently, Fog-RANs have been introduced as the evolution of Cloud Radio Access Networks (CRAN) for enabling edge computing in 5G systems. By alleviating the fronthaul burden for data transfer, transport delays are expected to be greatly reduced. However, in order to support envisioned 5G real-time and delay-sensitive applications, tailored radio resource and interference management schemes become necessary. Therefore, this paper investigates the issues of user scheduling and beamforming for energy efficient Fog-RAN. We formulate the energy efficiency maximization problem, taking into account the local user clustering constraint specific to Fog-RANs. Given the difficulty of this non-convex optimization problem, we propose a strategy where the energy efficient user scheduling is split in two parts: first, we solve an equivalent sum-rate maximization problem, then, the most energy-efficient FogAPs are activated in a greedy manner. To meet the requirement of low computational complexity of FogAPs, local beamforming is performed given fixed user scheduling. Simulation results show that the proposed scheme not only provides similar levels of user rates and fairness, but also largely outperforms the system energy efficiency in comparison with the baseline scheme1.

In future wireless networks, more and more users will be requiring various applications provided by multiple wireless interfaces simultaneously. This poses significant challenges for enabling efficient wireless resource sharing while satisfying the diverse and stringent Quality of Service (QoS) constraints, especially in dense interfering networks. In such a context, this work proposes distributed user-to-multiple Access Points (AP) association methods, where a user requiring several applications may be served by several APs simultaneously. The problem is formulated as a network sum-rate maximization subject to the required QoS constraints for each user and application, and AP load constraints. In the proposed distributed association methods, each user can decide to associate to multiple APs simultaneously using its locally available network information, leveraging reinforcement learning techniques. Simulation results show that, compared to a baseline scheme, the proposed methods enable large throughput enhancements while satisfying the QoS constraints and AP load limitations, thereby reducing user outage probabilities.

We investigate the user pre-scheduling and beamforming design for 5G Fog Radio Access Networks (FogRANs). Conventional Cloud Radio Access Networks (CRANs) enabled centralized resource and power allocation optimization over all the small cells served by multiple Access Points (APs). However, the fronthaul links connecting each AP to the cloud introduce delays and cause outdated Channel State Information (CSI). By contrast, FogRAN enables lower latencies and better CSI qualities, at the cost of local optimization. To alleviate these issues, we propose a hybrid algorithm exploiting both the centralized feature of the cloud for globally-optimized pre-scheduling using outdated global CSIs, and the distributed nature of FogRAN for accurate beamforming with high quality local CSIs. The centralized phase enables to consider the interference patterns over the global network, while the distributed phase allows for latency reduction, in line with the requirements of FogRAN applications. Simulation results show that our hybrid algorithm for FogRAN outperforms the centralized algorithm under outdated CSI, both in terms of throughput and delays.