The rapid evolution of telecommunication networks is leading to increasingly complex systems, requiring adaptive, flexible, and intelligent mechanisms for resource management, orchestration, and access control. In this context, the Network Digital Twin (NDT) paradigm emerges as a powerful tool to model the behavior of devices, communication links, operating environments, and applications in complex networks. This paper introduces FALCON, a Digital-Twin-based orchestration framework designed to optimize horizontal offloading in UAV-based Flying Ad Hoc Networks (FANETs) providing edge computing services to ground devices in remote areas. FALCON integrates multiple Smart Agents (DQN, A2C, PPO) running concurrently on the Digital Twin to dynamically determine the optimal offloading probabilities. A proof-of-concept demonstrates how the framework performs real-time What-if Scenario analyses and adapts to varying workload and channel conditions. Numerical results highlight the gains achieved through coordinated model selection and reuse, showing reduced end-to-end delay and faster convergence compared to standalone DRL-based controllers. • Digital Twin framework for real-time FANET orchestration. • Parallel Smart Agents enable fast What-if scenario evaluation. • Dynamic model selection adapts to FANET state and intents. • Reduced service delay compared to standalone DRL methods. • Model reuse ensures rapid reaction to changing UAV conditions.

Acoustic communications are the most exploited technology in the so-called Internet of Underwater Things (IoUT). UnderWater (UW) environments are often characterized by harsh propagation features, limited bandwidth, fast-varying channel conditions, and long propagation delay. On the other hand, IoUT nodes are usually battery-powered devices with limited processing capabilities. Accordingly, it is necessary to design optimization algorithms to address the challenging propagation features while balancing them with the limited device capabilities. To address the constraints of the nodes in energy and processing resources, it is crucial to adjust the transmission parameters based on the channel conditions while also developing communication procedures that are both lightweight and energy-efficient. In this work, we introduce a novel Multi-Player Multi-Armed Bandit (MP-MAB) framework for modulation adaptation in Multi-Hop IoUT Acoustic Networks. As opposed to widely used, computation-demanding Deep Reinforcement Learning (DRL) techniques, MP-MAB algorithms are simple and lightweight and allow to iteratively make decisions by selecting one among multiple choices, or arms. The framework is fully-distributed and is able to dynamically select the best modulation technique at each IoUT node by leveraging on high-level statistics (e.g., network throughput), without the need to exploit hard-to-extract channel features (e.g., channel state). We evaluate the performance of the proposed framework using the DESERT UW simulator and compare it with state-of-the-art centralized solutions based on Deep Reinforcement Learning (DRL) for cognitive and heterogeneous networks, namely DRL-MCS, DRL-AM, PPO, SAC, as well as with a multiple-agent, distributed version of the PPO. The results highlight that, despite its simplicity and fully-distributed nature, the proposed framework achieves superior performance in UW networks in terms of throughput, convergence speed, and energy efficiency. Compared to DRL-MCS and DRL-AM, our approach improves network throughput by up to 33% and 20%, respectively, and reduces energy consumption by up to 18% and 16%. When compared to PPO, SAC, and Multi-PPO, the proposed solution achieves up to 11%, 34%, and 38% higher throughput, and up to 7%, 17%, and 33% lower energy consumption, respectively.

Reliable communication in UnderWater (UW) environments is challenged by limited bandwidth, high temporal variability, and significant propagation delays. Orthogonal Frequency Division Multiplexing (OFDM) has emerged as a promising solution for UW communication; however, its performance heavily depends on dynamic subcarrier selection to adapt to fluctuating channel conditions. This paper presents MERMAID, a novel framework based on a Combinatorial Multi-Armed Bandit (CMAB) approach for adaptive subcarrier selection in Underwater Acoustic Networks (UANs). MERMAID dynamically activates subcarriers based on real-time Packet Delivery Ratio (PDR) feedback to maximize reliability while minimizing energy consumption. Unlike Deep Reinforcement Learning (DRL) techniques, which require extensive training and high computational overhead, MERMAID is specifically designed for resource-constrained UW nodes, enabling fast convergence and lightweight operation. We implement MERMAID in the DESERT simulator and benchmark its performance against state-of-the-art DRL baselines. Simulation results show that MERMAID improves PDR by up to 21.15% and reduces energy consumption by up to 45.40%. These findings highlight MERMAID as a scalable, efficient, and high-performance solution for real-world Internet of Underwater Things (IoUT) applications. The framework aligns with the 6G vision by advancing intelligent and energy-efficient connectivity in harsh environments, highlighting its potential to extend communication paradigms to UW networks.

UnderWater (UW) communication channels pose unique challenges due to their limited bandwidth, significant temporal variability, and long transmission delays. Addressing these challenges calls for the implementation of robust protocols capable of dynamically adjusting transmission parameters in response to channel conditions. In such a perspective, the development of intelligent algorithms capable of quickly adapting to current channel conditions based on measurements is of crucial importance. Indeed, this allows to adapt the signal transmission characteristics to the channel conditions, and to accordingly ensure optimal performance. In this perspective, this paper introduces a Multi-Player Multi-Armed Bandit (MP-MAB) framework for smart modulation adaptation in UnderWater Acoustic (UWA) Networks. Our solution is specifically tailored to run on resource-constrained UW nodes, thanks to the simplicity and low-complexity of MAB. Our framework can leverage real-time throughput statistics to dynamically select the optimal modulation technique for multi-hop signal transmission in UW scenarios. Notably, this happens in a fully-distributed way on a per-node basis, as each UW node runs a local MAB agent to autonomously select the best modulation to use according to its own channel conditions. Using the DESERT UW simulator, we evaluate the performance of our proposed framework and compare it with alternative state-of-the-art learning approaches. Results demonstrate the higher efficiency and responsiveness of our algorithm compared to the alternatives, despite its simplicity and fully-decentralized nature.

Deep reinforcement learning (DRL) techniques have the potential to significantly improve the ability of Unmanned Aerial Vehicles (UAVs) for mobile device localization in disaster scenarios by optimizing flight paths and enhancing signal detection accuracy using Reference Signal Received Power (RSRP) measurements. DRL allows UAVs to learn optimal navigation strategies autonomously in dynamic and complex environments, leading to more efficient and accurate localization of mobile devices. The integration between UAVs and 4G/5G technology allows for more accurate and timely localization of mobile devices under the rubble, thereby improving the overall effectiveness of the system. Smallcells, low-power cellular base stations, are used to enhance coverage and capacity. In this study, we propose a DRL-based UAV-Smallcell system that can quickly and efficiently localize devices in large disaster areas. The performance of the proposed system is evaluated through an extensive simulation campaign to demonstrate that our approach significantly improves the effectiveness of mobile device localization compared to other state-of-the-art approaches.

Job offloading represents one powerful key-enabler for delay-sensitive applications in Intelligent Transport Systems (ITS). Vehicles can rely on job offloading to move their jobs to the Road Side Units (RSUs) devices, which are typically equipped with better computing power. Ultimately, this procedure can help relieve the computational burden on the vehicles, to the benefit of the overall processing latency. However, fixed vehicular infrastructure cannot guarantee ubiquitous, ever-present support for job offloading, especially in the case of hard-to-reach, remote areas where the power grid and/or connectivity are not present. One way to solve this issue is to resort to portable, batterypowered RSUs, which can be installed potentially everywhere and allow for greater flexibility. However, this calls for a management framework that strikes the trade-off between RSU battery-life on one hand, and processing latency on the other. Moreover, this framework should be able to work in a fully-distributed way, without the need of an existing infrastructure. With all of this in mind, our contribution is two-fold. First, we introduce the idea of MEC-in-a-box (M-Box) stations, batterypowered RSUs that can operate even in the absence of a fixed infrastructure or connectivity. Second, in order to support the operation of M-Box stations, we design MANTRA, an offloading framework for vehicular networks that balances energy consumption and processing latency in the M-Box stations. MANTRA allows the M-Box stations to autonomously i) turn on and off their processing units to alternatively increase their processing power or save energy, and ii) manage the amount of jobs offloaded towards the other M-Box stations to balance the network load. We evaluated the performance of MANTRA vs other approaches, including a centralized solution with a full knowledge of the system. First, the results highlight how MANTRA is able to match the performance of the centralized approach, and is also capable of surpassing all the other baselines in various experimental scenarios. Index Terms

Underwater (UW) communication channels experience limited bandwidth, high time variability and much longer delays as compared to traditional terrestrial channels. This makes communication in UW scenarios particularly challenging. One way to cope with this issue is to employ reliable smart protocols that can dynamically adjust transmission parameters based on channel conditions. The effectiveness of these solutions also relies on the design of intelligent algorithms that can forecast channel conditions based on measurements and adapt the transmission characteristics of signals according to the current state of the UW channel, in order to always guarantee the best performance. In this work we propose AMUSE, the first Multi-Armed Bandit-based algorithm for smart modulation adaptation in Underwater Acoustic Networks. Note that AMUSE is specifically designed to suit resource-constrained UW nodes thanks to its simplicity and low-complexity. In particular, AMUSE relies on the current Packet Delivery Ratio (PDR) statistics to select in real time the best modulation technique to use for multihop signal transmission in the aforementioned UW scenarios. By employing the DESERT simulator we compare the performance achieved using AMUSE to those obtained using alternative state-of-the-art learning approaches. The results show that, in spite of its simplicity, our algorithm is more efficient and responsive than the other considered approaches.

In this paper we propose MARBLE, a distributed framework based on multi-player multi-armed bandit algorithms that aims to support job offloading procedures in FANETs to comply with strict job latency requirements while also minimizing the energy consumption of the system, thus also increasing the Unmanned Aerial Vehicles (UAVs) flight duration. To demonstrate the effectiveness of the framework, we conduct an extensive evaluation campaign and compare MARBLE with several baselines, including a centralized, oracle-based approach. Our results show that MARBLE outperforms the baselines and quickly converges to the performance of the centralized approach in a fully-distributed manner, in compliance with the latency and energy efficiency requirements. Most notably, MARBLE also improves the FANETs flight duration of up to 69% as compared to the baselines.

In the last few years, Virtual Reality (VR) is assuming a prominent role and is recognized as a pivotal technology in various sectors. However, transmission of immersive videos produced by real-time 360° cameras or stored on remote servers to reproduce 3D environments, or streamed by video games accessible through headsets, would require a lot of network bandwidth that, in many cases, is not available or too expensive to be obtained. In this paper, we leverage on network softwarization provided by new 5G&B networks, and introduce an Adaptive Closed-loop Encoding VNF named 360-ST for adaptive compression of 360° video streaming. This VNF is able to apply a hierarchical compression that takes into account both the bandwidth currently available in the network, and the user viewport. The agent that is in charge of deciding the different compression ratio at runtime uses Deep Reinforcement Learning to optimize a reward function and adapt to the changes of the network bandwidth, the end-to-end latency, the user movements within the scene and the video content. The results indicate that our proposed method consistently outperforms state-of-the-art algorithms by an average of 8% to 46% in terms of achieved Peak Signal-to-Noise Ratio (PSNR).

The contention window (CW) has a significant impact on the efficiency of Wi-Fi networks. Unfortunately, the basic access method employed by 802.11 networks does not scale well for increasing number of stations. Therefore, in this paper we propose a new CW control method which leverages Deep Reinforcement Learning (DRL) to learn the optimal policies under different network conditions. For this reason, we propose the Online Smart Collision Avoidance Reinforcement learning (OSCAR) algorithm, a DRL-based algorithm that can be deployed online to quickly and efficiently find the best contention window that maximizes the throughput. We also demonstrate through a simulation campaign that it is able to learn the optimal policies way faster than the current state of art methods while also being able to keep the computational cost low.

The optimization of job offloading procedures in modern vehicular networks is a problem of utmost importance. In this regard, this paper proposes MANTRA, a distributed framework based on multi-player multi-armed bandit (MP-MAB) algorithms for latency- and energy-aware job offloading in vehicular networks. The main goal of MANTRA is to support procedures of job offloading in green vehicular networks to achieve a target tradeoff between energy consumption and job processing latency. In particular, MANTRA is intended to run on so-called MEC-in-a-box (M-Box) devices, portable battery-powered Road Side Units (RSUs) specifically designed to work without mobile connectivity and of a fixed power grid.To demonstrate MANTRA effectiveness, we model the vehicular network using the queueing theory for M/M/m/K systems. We run an extensive evaluation campaign and compare MANTRA with several baselines, including a centralized, oracle-based approach. In such a way, we demonstrate how MANTRA outperforms the baselines and quickly converges to the performance of the centralized approach in a fully-distributed way in terms of job processing latency and network outage probability.

The next generation of wireless communications networks, namely 6G, will be aimed at realizing a fully connected world, and at providing ubiquitous connectivity to people and objects even in remote areas that are very far from the structured Internet core network. These goals include the definition and the design of intelligent communications environments mainly characterized by pervasive artificial intelligence and large-scale automation. The target of this paper is the design of a management framework for edge networks realized with Flying Ad-Hoc Networks (FANET) consisting of a set of Unmanned Aerial Vehicles (UAVs) to provide a remote geographic area with computing and networking facilities for delay-sensitive applications. To this purpose, each UAV is equipped with a Computing Element (CE) to process jobs received through vertical offloading from ground devices. In addition, horizontal offload among UAVs of the FANET is introduced for load balancing purposes, to guarantee that the FANET computation delay for each received job is minimized and is almost independent of the activity state of the area covered by the UAV receiving that job. The proposed FANET management framework is based on Deep Reinforcement Learning (DRL) to allow zero-touch adaptation to the time-variant activity state of the area covered by each UAV. Numerical results demonstrate the power of the proposed framework and the enhancements achieved with respect to the current literature.

The fifth generation (5G) of mobile networks has the goal of providing ultra-high-speed access everywhere and enabling connectivity of massive number of devices in an ultra-reliable and affordable way. However, in many environments that are considered strategic for 5G applications, a structured network is not available. A solution to extend the features provided by Multi-access Edge Computing (MEC), one of the main enabler of 5G, in these contexts, is to use fleets of MEC UAVs, each equipped with a computing element (CE), and organized in Flying Ad-hoc Networks (FANET).In this paper, we propose a FANET platform with “horizontal” offload from the most overloaded UAVs to the least overloaded ones, aimed at balancing load among UAVs. A decision policy called UAV Smart Offloading (USO), based on Deep Reinforcement Learning, is also defined to optimize performance in terms of delay perceived by the ground devices connected to the FANET. A numerical analysis is introduced to evaluate performance achieved by the proposed platform.