To address the limitations of existing wireless networks for demanding applications like brain-computer interfaces and intelligent transportation systems, we propose an advanced framework for joint resource allocation and task offloading across integrated terrestrial and non-terrestrial networks (TN-NTN). This framework utilizes multiple layers, including ground users, UAVs, HAPs, and satellites, to improve service quality and immersive experiences, particularly in scenarios like Metaverse applications. Ground users request resources, while UAVs and HAPs serve as resource providers, and satellites ensure reliable communication during emergencies. A double auction-based incentive scheme is employed in which operators control UAV and HAP resources to maximize utility, and users aim to minimize computation costs and protect data privacy. To handle the complexity of the operator-user interaction, which results in an NP-hard optimization problem, we applied a hierarchical multi-agent federated deep reinforcement learning (FeDRL) approach. Our simulation results demonstrate that the FeDRL algorithm significantly improves social welfare by 6.38%, 17.43%, and 28.73% over modified MADDPG, FRL, and DDPG algorithms, respectively.

In the dynamic vehicular metaverse, delivering a seamless user experience (UX) and effective human-machine interaction (HMI) is challenging due to vehicle mobility and varying resource needs. This paper introduces an adaptive resource allocation and twin migration framework using Multi-Agent Deep Reinforcement Learning (MADRL) for a multi-tier vehicular metaverse. The framework enables cooperative agents to dynamically allocate resources and migrate vehicle twins across vehicle, edge, and cloud layers, ensuring seamless UX and efficient HMI. The joint resource allocation and twin migration optimization problem is modeled as MDP and a hierarchical multi-agent deep deterministic policy gradient-with QMIX (MADDPG-Q) strategy is adopted to solve it, reducing latency and optimizing resource use. Moreover, the proposed framework is designed to be context-aware, adjusting HMI based on real-time conditions, and enhancing interaction quality. Simulation results show significant improvements in UX, latency reduction, and resource efficiency.

The integration of terrestrial and non-terrestrial networks (TN-NTN) in 6 G is essential to support real-time applications like the Metaverse and intelligent edge services, which demand ultra-reliable low-latency communications (xURLLC). Managing these networks and maintaining robust security presents significant challenges due to their complexity and high-dimensional environments. Quantum communication, particularly quantum key distribution (QKD), offers a promising solution by providing unbreakable encryption and enhancing security across TN-NTN architectures. In this paper, we propose a novel deep reinforcement learning approach for QKD-enabled resource allocation in 6 G TN-NTN Metaverse service and transform the joint resource allocation and QKD deployment cost optimization problem into a stochastic game model to ensure secure and efficient resource distribution across TN-NTN environment. We introduce a novel hierarchical multi-agent proximal policy optimization (MAPPO) framework to address the formulated optimization problem. This framework enables dynamic and secure allocation of Metaverse resources and services from multiple providers to users while minimizing QKD deployment costs. Our simulations demonstrate that the proposed framework significantly enhances network performance, reduces key generation costs, and optimizes resource utilization and service quality.

The integration of IoT devices with smart home energy management systems (SHEMS) presents a significant advancement in energy demand response (DR) optimization. However, due to the rapid proliferation of home appliances with varying operating characteristics as well as the variable comfort level demands of users, making effective DR decisions becomes more challenging. In this paper, we propose a hierarchical Stackelberg game-based incentive mechanism with multi-agent deep reinforcement learning (MADRL) to optimize DR in IoT-based SHEMS. We formulate the hierarchical decision-making problem as a Markov decision process (MDP) and then adopt the multi-agent deep deterministic policy gradient (MADDPG) algorithm to solve it by finding an equilibrium solution. Through extensive simulations, we demonstrate that our proposed DR optimization approach can effectively reduce overall energy consumption and peak load by 30.41% and 28.57% from the benchmark approaches, respectively. In addition, the proposed approach maintains user comfort and increases system utility by 13.11% and 15.74% than the benchmark schemes, respectively, resulting in improved energy efficiency.

Sixth generation (6G) networks deploy unmanned aerial vehicles and mobile edge computing to provide collaborative computing and reliable connectivity for resource-limited mobile devices (MDs). However, due to the untrusted and broadcast nature of wireless transmission among communicating MDs and computing resource providers, ensuring the security of resource transactions will be challenging. Blockchain-based resource-sharing systems have been proposed to address security issues. However, these systems use existing consensus mechanisms like Proof-of-Work that consume massive amounts of system resources. In addressing this, some studies attempted to use single-agent deep reinforcement learning (DRL) in leader selection. Nevertheless, these solutions overlooked the intelligence and flexibility of blockchain configuration, and a single-point of failure can cause the system to fail. We propose a multiagent distributed deep deterministic policy gradient (MAD3PG)-assisted consensus mechanism for blockchain-based collaborative resource sharing to address these issues. First, we propose a stochastic game-based incentive-mechanism to encourage consensus nodes to participate in transaction validation. Then, we formulate the optimization problem of node selection and blockchain configuration as a Markov decision process and solve it with the MAD3PG algorithm. With MAD3PG, the agents select consensus nodes based on their experience and available resources and dynamically adjust blockchain settings. The simulation results show that MAD3PG outperforms the benchmarks in maximizing throughput and incentive while minimizing block production latency.

The increasing demand for high-quality digital content has driven the growth of content exchange among mobile users (MUs) via device-to-device (D2D) communication. However, MUs often face challenges such as limited storage, low computational power, and short battery life, making it very difficult to meet the rising demands for content sharing. UAV-assisted D2D communication has emerged as a promising solution, integrating aerial and ground networks to enable efficient content caching and distribution while reducing latency and communication costs. However, the high mobility of MUs and increasing content size make it challenging to maintain stable communication links between MUs. This increases the complexity of content distribution, caching, and resource allocation in D2D content-sharing frameworks, resulting in higher latency, fluctuating resource demands, and lower QoS, ultimately affecting system efficiency and reliability. To address these challenges, we propose an adaptive and collaborative content-sharing and resource allocation framework integrating multi-agent twin delayed deep deterministic policy gradient (MATD3), blockchain, and a multiple-round distributed double auction (MDDA). MATD3 enables dynamic decision-making for content caching and resource allocation based on user behavior and mobility, while blockchain ensures secure, transparent, and tamper-proof content-sharing transactions. Furthermore, we propose the MDDA-based incentive scheme that allows content sellers, buyers, and the auctioneer to interact and establish optimal pricing strategies. This optimizes the content-sharing capability of MUs and edge devices, enhancing the cache hit rate and average system utility. Finally, the extensive simulation results demonstrate that our proposed scheme outperforms the benchmarks in enhancing cache hit rates, communication latency, and average system utility.

Unmanned Aerial Vehicle (UAV)-powered 5G/6G networks integrated with rechargeable wireless sensor networks (RWSNs) offer promising solutions for extending system lifetime, collecting data, and providing computing services and power to sensor nodes (SNs). UAVs offer significant advantages, including exceptional mobility, cost-effective deployment, and the ability to be easily reprogrammed for a wide range of missions. However, the limited onboard power capacity of UAVs, coupled with the lack of dynamic and intelligent charging station (CS) management and inefficient path planning, can lead to SN failure in dynamic mobile environments. To address these challenges, we propose an energy-efficient laser-charged UAV (LCU)-enabled RWSN environment, wherein UAVs, powered by laser beams from ground-based stations, provide services, collect data, and transfer energy to SNs. We formulate a joint optimization problem involving power allocation, dynamic charging strategy (DCS), and path planning to minimize task completion time and sensor node death time. Given the NP-hard nature of the problem, we employ a stochastic game model based on a Markov decision process (MDP) for its solution. To solve this problem, we propose a deep reinforcement learning (DRL) based algorithm that enables real-time charging scheduling decisions while optimizing network performance. We introduce a multi-agent double deep Q-network (MA-DDQN) model to determine the optimal trajectories for all UAVs in large and complex environments. Simulation results demonstrate that the MA-DDQN approach outperforms state-of-the-art techniques, showing significant improvements in terms of average delay, energy consumption, and task completion time.

The integration of terrestrial and non-terrestrial networks (TN-NTN) with open radio access network (O-RAN) technology presents a significant advancement for facilitating scalable and immersive Metaverse services within 6G networks. Seamless virtual experiences necessitate highly reliable, low-latency communication, effective resource management, and adaptive decision-making to satisfy the varied and rigorous requirements of Metaverse applications, including gaming, healthcare, and autonomous systems. The inherent heterogeneity, dynamic nature, and substantial resource requirements of TN-NTN present significant challenges for effective resource allocation and optimizing quality of experience (QoE). Then, we formulate a multi-objective optimization problem for joint resource allocation and spectrum sharing in O-RAN-enabled TN-NTN Metaverse environments. This problem is inherently NP-hard due to the intricate coupling between continuous action spaces and discrete decision variables. Solving such a complex problem using traditional optimization approaches is complex. To overcome this, we transform the problem into a decentralized partially observable Markov decision process (Dec-POMDP) and address it using a hierarchical multi-agent deep reinforcement learning (MADRL) approach. This study presents a hierarchical multi-agent proximal policy optimization (MAPPO) framework, a new MADRL solution for dynamic resource allocation and spectrum sharing in O-RAN-enabled TN-NTN Metaverse environments. MAPPO facilitates collaborative learning among intelligent agents to optimize resource management strategies in a decentralized manner, considering essential metrics, including energy consumption, latency, and meta-distance. The proposed framework enhances resource utilization efficiency, minimizes latency, and improves the QoE for Metaverse users through the seamless allocation and management of resources. Comprehensive simulations show that MAPPO outperforms baseline methods, such as conventional reinforcement learning and centralized optimization approaches, achieving better energy efficiency, lower latency, and improved QoE. This demonstrates its effectiveness in adapting to dynamic 6G-enabled Metaverse requirements, enabling intelligent and scalable TN-NTN networks.

The integration of Mobile Edge Computing (MEC) with aerial platforms introduces novel potential for the Metaverse world by providing low-latency and highly reliable computing and communication services at the network edge. Nevertheless, this integration presents critical challenges, such as low Quality of Experience (QoE) due to the dynamic nature of aerial platforms, high resource demands, and the requirements for real-time data processing in the Metaverse environment. To address these challenges, we propose a Combinatorial Client-Master Multiagent Deep Reinforcement Learning (CCM-MADRL) based joint resource allocation and QoE maximization framework to enable intelligent real-time decision-making in aerial MEC enabled Metaverse services. We form a collaborative ecosystem where agents are designed to represent both Metaverse service providers and aerial platforms to promote fairness and efficiency in resource allocation, as well as optimize service delivery. By incorporating CCM, our approach considers diverse metrics, such as latency, reliability, meta-distance, and energy efficiency, to ensure a holistic optimization of Metaverse services. The MADRL approach enables adaptive decision-making, allowing the system to respond to the dynamic and unpredictable nature of Metaverse applications. Results from simulations that mimic realistic Metaverse scenarios demonstrate the effectiveness of the proposed CCM-MADRL framework in terms of improved service performance, reduced latency, cost, and virtual meta-distance, maximized average QoE utility of Metaverse users, and enhanced resource utilization compared to baseline algorithms.

The integration of multi-agent deep reinforcement learning (MADRL) in adaptive and intelligent home energy management systems (AI-HEMS) enhances real-time energy management by enabling intelligent decision-making among multiple agents to optimize various problems. This approach allows smart homes to dynamically respond to changes in energy demand, pricing, and user preferences. The integration of Internet of Things (IoT) devices with AI-HEMS has been promoted to efficiently manage energy resources and maintain occupants’ comfort, where IoT devices collect data on energy consumption, usage patterns, and environmental conditions. However, ensuring trade-offs between conflicting optimization objectives, such as reducing energy consumption and electricity prices, and maximizing users’ comfort levels is challenging. In this paper, we propose a MADRL-based multi-objective demand response (MODR) optimization framework to efficiently manage and control the energy consumption of smart homes. The proposed approach aims to simultaneously reduce energy costs and maximize users’ comfort, improving the overall reliability of energy systems. We first formulate the MODR optimization problem as MDP and then adopt the MADRL algorithm to solve it. The simulation results demonstrate that our proposed DR optimization approach can effectively balance the trade-off between energy cost and user comfort levels, resulting in improved energy efficiency compared to benchmark approaches.

Demand response (DR) plays a significant role in modern energy management systems, particularly within the context of adaptive and intelligent building energy management systems (AI-BEMS). In the AI-BEMS context, DR focuses on dynamically adjusting energy usage in response to external factors, such as electricity prices, grid conditions, and environmental considerations. This survey paper explores the evolving landscape of DR within the framework of AI-BEMS, focusing on the integration of advanced technologies and adaptive strategies to optimize energy consumption and enhance grid reliability. This article reviews state-of-the-art research addressing the key concepts associated with integrating DR and AI-BEMS, including an overview of DR techniques in AI-BEMS, and an artificial intelligence and machine learning applications for the development of adaptive control strategies and DR optimization. Then, insights are provided on the future directions and the challenges in this field regarding the implementation of DR within AI-BEMS.

In sixth-generation (6G) networks, unmanned aerial vehicles (UAVs) are expected to be widely used as aerial base stations (ABS) due to their adaptability, low deployment costs, and ultra-low latency responses. However, UAVs consume large amounts of power to collect data from multiple sensor nodes (SNs). This can limit their flight time and transmission efficiency, resulting in delays and low information freshness. In this paper, we present a multi-access edge computing (MEC)-integrated UAV-assisted wireless sensor network (WSN) with a laser technology-based energy harvesting (EH) system that makes the UAV act as a flying energy charger to address these issues. This work aims to minimize the age of information (AoI) and improve energy efficiency by jointly optimizing the UAV trajectories, EH, task scheduling, and data offloading. The joint optimization problem is formulated as a Markov decision process (MDP) and then transformed into a stochastic game model to handle the complexity and dynamics of the environment. We adopt a multi-agent deep Q-network (MADQN) algorithm to solve the formulated optimization problem. With the MADQN algorithm, UAVs can determine the best data collection and EH decisions to minimize their energy consumption and efficiently collect data from multiple SNs, leading to reduced AoI and improved energy efficiency. Compared to the benchmark algorithms such as deep deterministic policy gradient (DDPG), Dueling DQN, asynchronous advantage actor-critic (A3C) and Greedy, the MADQN algorithm has a lower average AoI and improves energy efficiency by 95.5%, 89.9%, 78.02% and 65.52% respectively.

The space-air-ground integrated network (SAGIN) is an emerging architecture that has the potential to provide seamless, high data rates, and reliable transmission with a vastly increased coverage for intelligent edge devices (iEDs). However, the SAGIN infrastructure is quite complex consisting of multiple network segments; it is thus critical to efficiently manage the network segments' resources to ensure QoS satisfaction (e.g., delay and rate) for the various services provided to the iEDs. In this regard, network slicing (NS) and overall network softwarization technologies can play an essential role in addressing iEDs QoS and utility needs. In this work, we propose an optimal intelligent end-to-end resource allocation with network slicing in multi-tier SAGIN to maximize the network performance. We model the network depending on its service requirements. As the above optimization problem turns out to be NP-hard, we transform it into a stochastic game model and efficiently solve it through hierarchical multi-agent deep reinforcement learning (HMADRL). In particular, we decompose it into two parts, i.e., optimizing the mapping combined with slice adjustment and the resource allocation with association problem. Both problems are then solved using multi-agent DRL. The simulation results demonstrate that our proposed HMADRL algorithm outperforms the baseline algorithms in terms of maximizing the utility and QoS satisfaction of iEDs.

A cyber-physical system (CPS) is a promising paradigm in 5G and future 6G networks that controls physical components through computing and communication while ensuring efficacy, intelligence, and security. The number of smart mobile devices or sensors in smart cities is growing very fast. These devices can process applications in real-time only for a short time due to limited resource capacity. The Mobile Edge Computing (MEC) paradigm is a prominent solution that allows devices to offload intensive tasks and allocate resources. However, the terrestrial MEC servers will be overwhelmed and unable to meet the requirements of 6G technologies for ultra-low-latency applications and mobile devices. Aerial-borne MEC servers have recently supported ultra-reliable, low-latency communication applications and mobile devices in an emergency scenario by providing resources and relaying them to a cloud server. In the intelligent aerial-enabled smart city CPS (S2CPS), decisionmaking tasks such as resource allocation, association, and ensuring trust between links are challenging, and the optimization problem is multi-objective. Therefore, we proposed a hierarchical, deep federated learning-empowered, energy-efficient resource allocation for aerial-enabled S2CPS to minimize the overall energy consumption while considering the quality of service of user devices and the privacy of task offloading in a dynamic environment. We validated the proposed framework through extensive simulations, proving it outperformed the baseline algorithms.

In the 5G/B5G network paradigms, intelligent medical devices known as the Internet of Medical Things (IoMT) have been used in the healthcare industry to monitor remote users’ health status, such as elderly monitoring, injuries, stress, and patients with chronic diseases. Since IoMT devices have limited resources, mobile edge computing (MEC) has been deployed in 5G networks to enable them to offload their tasks to the nearest computational servers for processing. However, when IoMTs are far from network coverage or the computational servers at the terrestrial MEC are overloaded/emergencies occur, these devices cannot access computing services, potentially risking the lives of patients. In this context, unmanned aerial vehicles (UAVs) are considered a prominent aerial connectivity solution for healthcare systems. In this article, we propose a multiagent federated reinforcement learning (MAFRL)-based resource allocation framework for a multi-UAV-enabled healthcare system. We formulate the computation offloading and resource allocation problems as a Markov decision process game in federated learning with multiple participants. Then, we propose an MAFRL algorithm to solve the formulated problem, minimize latency and energy consumption, and ensure the quality of service. Finally, extensive simulation results on a real-world heartbeat data set prove that the proposed MAFRL algorithm significantly minimizes the cost, preserves privacy, and improves accuracy compared to the baseline learning algorithms.

With the Industrial Internet of Things (IIoT), mobile devices (MDs) and their demands for low-latency data communication are increasing. Due to the limited resources of MDs, such as energy, computation, storage, and bandwidth, IIoT systems cannot meet MDs’ quality of service (QoS) and security requirements. Recently, UAVs have been deployed as aerial base stations in the IIoT network to provide connectivity and share resources with MDs. We consider a resource trading environment where multiple resource providers compete to sell their resources to MDs and maximize their profit by continually adjusting their pricing strategies. Multiple MDs, on the other hand, interact with the environment to make purchasing decisions based on the prices set by resource providers to reduce costs and improve QoS. We propose a novel intelligent resource trading framework that integrates multi-agent deep reinforcement Learning (MADRL), blockchain, and game theory to manage dynamic resource trading environments. A consortium blockchain with a smart contract is deployed to ensure the security and privacy of the resource transactions. We formulated the optimization problem using a Stackelberg game. However, the formulated optimization problem in the multi-agent IIoT environment is complex and dynamic, making it difficult to solve directly. Thus, we transform it into a stochastic game to solve the dynamics of the optimization problem. We propose a dynamic pricing algorithm that combines the Stackelberg game with the MADRL algorithm to solve the formulated stochastic game. The simulation results show that our proposed scheme outperforms others to improve resource trading in UAV-assisted IIoT networks.

In 6G networks, unmanned aerial vehicles (UAVs) can serve as aerial flying base stations (AFBS) with aerial mobile edge computing (AMEC) server capabilities. AFBS is an increasingly popular solution for delivering time-sensitive applications, extending network coverage, and assisting ground base stations in the healthcare systems for remote areas with limited infrastructure. Furthermore, the UAVs are deployed in the healthcare system to support the Internet of medical things (IoMT) devices in data collection, medical equipment distribution, and providing smart services. However, ensuring the privacy and security of patients' data with the limited UAV resources is a major challenge. In this paper, we present a federated deep reinforcement learning framework for resource allocation in UAV-enabled healthcare systems, where IoMT devices send their trained model parameters without transmitting sensitive raw data to the AMEC server. In the proposed framework, the IoMT device is associated with AFBS based on the quality of the data and its demand in order to maximize learning efficiency and accuracy. This work aims to minimize the computation costs of the IoMT devices while considering UAV resources and the fairness of UAV coverage. Simulation results prove that our proposed algorithm outperforms other baseline algorithms in learning accuracy and computational cost.

Unmanned Aerial Vehicle (UAV) is a promising technology that can serve as aerial base stations to assist Internet of Things (IoT) networks, solving various problems such as extending network coverage, enhancing network performance, transferring energy to IoT devices (IoTDs), and perform computationally-intensive tasks of IoTDs. Heterogeneous IoTDs connected to IoT networks have limited processing capability, so they cannot perform resource-intensive activities for extended periods. Additionally, IoT network is vulnerable to security threats and natural calamities, limiting the execution of real-time applications. Although there have been many attempts to solve resource scarcity through computational offloading with Energy Harvesting (EH), the emergency and vulnerability issues have still been under-explored so far. This paper proposes a blockchain and multi-agent deep reinforcement learning (MADRL) integrated framework for computation offloading with EH in a multi-UAV-assisted IoT network, where IoTDs obtain computing and energy resources from UAVs. We first formulate the optimization problem as the joint optimization problem of computation offloading and EH problems while considering the optimal resource price. And then, we model the optimization problem as a Stackelberg game to investigate the interaction between IoTDs and UAVs by allowing them to continuously adjust their resource demands and pricing strategies. In particular, the formulated problem can be addressed indirectly by a stochastic game model to minimize computation costs for IoTDs while maximizing the utility of UAVs. The MADRL algorithm solves the defined problem due to its dynamic and large-dimensional properties. Finally, extensive simulation results demonstrate the superiority of our proposed framework compared to the state-of-the-art.