In 5G/6G wireless networks, enhanced mobile broadband (eMBB) services provide gigabit-per-second high data rates, while ultra-reliable and low-latency communication (URLLC) services offer extremely low latency and high reliability. Simultaneously supporting these two types of services in the same wireless environment presents a significant challenge due to their fundamentally different service requirements. In this paper, we formulate the bandwidth and power resource allocation problem of multiplexing eMBB and URLLC services in a wireless non-orthogonal multiple access (NOMA) network as a nonlinear programming problem. The objective is to maximize the average supply-demand ratio (SDR) of eMBB users while minimizing the SDR variance among eMBB users, subject to the reliability constraint of URLLC users. We propose a deep reinforcement learning (DRL)-based NOMA resource allocation scheme (DRL-NRAS), which employs two DRL agents at the base station to jointly optimize bandwidth and power allocation for eMBB services, as well as the puncturing and superposition ratios for URLLC transmissions. The proposed DRL-NRAS fully utilizes a DRL technique, twin delayed deep deterministic policy gradient, to effectively handle the time-varying data rate requirements of eMBB users and the stochastic request arrivals of URLLC users. Simulation results demonstrate that the proposed DRL-NRAS reduces the SDR variance by 78.4%, 87.2%, and 72.7% compared to the CVX, B-greedy, and random schemes, respectively, while achieving a 0% delay violation rate under the considered simulation settings. These results indicate that the proposed DRL-NRAS can effectively improve the fairness among eMBB users while ensuring the reliability of URLLC services.

IN the above article [1], in Table II (Continued.), the text is written as “ $R_{m,n}^{e,pun}\left ({ i,j }\right)$ and $R_{m,n}^{e,sup}\left ({ i,j }\right)$ , respectively, in the definition of $R_{m,n}^{e,aff} ({i,j})$ in Table II (Continued.)”.

Space-air-ground-integrated networks (SAGINs) can meet the goal of creating a seamless global coverage for 6G. With software-defined networking and network function virtualization technologies, a service can be represented by a service function chain (SFC), which is a sequenced set of multiple virtual network functions (VNFs). Nevertheless, the embedding of VNFs of an SFC, referred to as the SFC embedding problem, including SFC placement and SFC migration, remarkably impacts the quality of experience (QoE) of user equipments (UEs) and the energy efficiency of an SAGIN. In this paper, we jointly investigate the SFC placement and migration in the SFC embedding problem in an SAGIN. We formulate this problem as a non-linear programming problem seeking to minimize SAGIN energy consumption while meeting the service downtime (SD) and traverse delay (TD) restrictions of each UE. To tackle this problem, we propose a deep reinforcement learning-based energy-efficient and mobility-aware SFC embedding strategy (DRL-EMSES). This strategy employs an SFC deployment controller to keep track of the conditions of network nodes in the SAGIN and make intelligent embedding decisions in response to placement and migration requests from network nodes. The proposed DRL-EMSES leverages a DRL technique, Deep Deterministic Policy Gradient (DDPG), to handle the stochastic arrivals of placement and migration requests in the SFC deployment controller, thereby optimizing long-term network performance. The simulation results show that the proposed DRL-EMSES can successfully lower energy consumption of the SAGIN while meeting SD and TD restrictions of UEs.

The immersive experiences offered by the metaverse have been gradually integrated into people’s lives. However, managing the bandwidth and power resources of the base station (BS) in wireless metaverse networks to meet the high data rate requirements of metaverse users has become an important issue. In this paper, we formulate the non-orthogonal multiple access (NOMA) bandwidth and power resources allocation problem in a wireless metaverse network as a nonlinear programming problem, aiming to maximize the average supply-demand ratio (SDR) of NOMA groups while minimizing the SDR variance of NOMA groups and the average delay difference among high-interaction metaverse users. To this end, we propose a deep reinforcement learning-based resource allocation mechanism (DRL-RAM), which utilizes the resource manager within the BS to monitor the bandwidth and power resources usage and intelligently allocate these resources in each time slot. The proposed DRL-RAM fully utilizes a DRL technique, twin delayed deep deterministic policy gradient, to efficiently handle the dynamic nature of time-varying data rate requirements of metaverse users thereby achieving long-term network performance optimization. Simulation results show that the proposed DRL-RAM achieves competitive average SDR performance while significantly reducing SDR variance of NOMA groups and delay differences among high-interaction metaverse users, compared to the P-Greedy and B-Greedy allocation schemes.

With the rapid growth of emerging delay-sensitive and computation-intensive vehicular applications, the integration of vehicle-to-everything (V2X) networks and multi-access edge computing (MEC) has emerged as a promising paradigm. However, how to make appropriate offloading decisions to reduce the overall processing time has become a critical issue. In this paper, we propose a novel V2X collaborative MEC network (VCMN) architecture that enables inter-roadside unit (RSU) collaboration for distributed task offloading. Based on this architecture, we formulate the task offloading problem as a nonlinear programming optimization problem, aiming to minimize the average total processing time of all offloading tasks while satisfying the processing time constraint of each offloading tasks. We propose an event-driven deep reinforcement learning-based task offloading policy (EDRL-TOP), which utilizes a centralized RSU manager to monitor the load conditions of all RSUs and RSU-to-RSU links, and intelligently makes offloading decisions for each event. The proposed EDRL-TOP fully utilizes a DRL technique, deep deterministic policy gradient (DDPG), to effectively handle the stochastic request arrivals thereby achieving long-term network performance optimization. Simulation results show that the proposed EDRL-TOP can decrease the average total processing time of tasks and achieves a higher task success rate compared to the greedy algorithm.

In spectrum pooling, which is a well-known technique of spectrum sharing, the initial licensed spectrum of each Mobile Network Operator (MNO) is partitioned into reserved and shared spectrum. The reserved spectrum is for the personal use of an MNO, and the shared spectrum of all MNOs constitutes a spectrum pool that can be flexibly utilized by MNOs that require extra spectrum. Nevertheless, the spectrum pool management problem substantially impacts the spectrum efficiency among these MNOs. In this paper, we formulate this problem as a non-linear programming problem that strives to maximize the average binary scale satisfaction (BSS) of MNOs. To achieve this objective, we introduce an event-driven deep reinforcement learning-based spectrum management scheme, termed EDRL-SMS. This approach adopts a spectrum pool manager (SPM) to efficiently supervise the spectrum pool to reach long-term optimization of network performance. The SPM smartly allocates spectrum resources by fully utilizing a DRL approach, Deep Deterministic Policy Gradient, for each stochastic arrival spectrum request event. The simulation results show that the average BSS of MNOs of the proposed EDRL-SMS significantly outperform our previous work, Bankruptcy Game-based Resource Allocation (BGRA), greedy, random, and without sharing schemes.

In this paper, we formulate the interference problem among satellites as a nonlinear programming problem, which aims to maximize the average binary scale satisfaction (BSS) of low-earth-orbit (LEO) satellites while satisfying the data rate requirement of the geostationary orbit (GEO) satellite in a GEO-LEO co-existing satellite networks (GLCSNs) through the minimization of frequency band interference among these satellites. We propose an event-driven deep reinforcement learning based frequency bands allocation mechanism (EDRL-FBAM) that utilizes a frequency bands manager in a GEO satellite to monitor the spectrum usage of all LEO satellites and intelligently allocate the frequency bands for each frequency band request from LEO satellites. The proposed EDRL-FBAM fully utilizes a DRL technique, Proximal Policy Optimization, to deal with stochastic arrivals of frequency band requests in the frequency bands manager so as to achieve long-term optimization of the network performance. The simulation results show that the proposed EDRL-FBAM can significantly improve the average data rate of LEO satellites, the average BSS of LEO satellites, and the average data rate of GEO satellite when compared to greedy and random approaches.

Satellite networks provide higher coverage and provide ubiquitous mobile services. However, how to allocate precious satellite spectrum resources to improve better network performance has become an important issue in satellite networks. In this paper, we study the power allocation problem of the Low Earth Orbit (LEO) satellite to maximize the Supply-Demand Ratio (SDR) of the LEO satellite users while minimize the standard deviation (SD) of LEO satellite users’ SDR. We propose an Event-Driven Deep Reinforcement Learning based Power Allocation Mechanism (EDRL-PAM), which utilizes the LEO satellite’s power manager to intelligently allocate the power request for each cell of the LEO satellite. Furthermore, we propose a NOMA-based power allocation algorithm for allocating power to all LEO satellite users within the cells. The proposed EDRL-PAM fully utilizes a Deep Reinforcement Learning (DRL) technique, Deep Deterministic Policy Gradient (DDPG), to deal with stochastic arrivals of power requests in the power manager to achieve long-term optimization of the network performance. The simulation results show that our proposed EDRL-PAM can significantly improve the average data rate, and achieve the long-term optimization and fairness of network performance for satellite users.

With the increasing demands of global communication, integrating the low-earth-orbit (LEO) satellite networks (LSNs) with the existing terrestrial networks can significantly extend the wireless coverage in 6G networks. Since the deployment cost of LEO satellites is extremely high, an energy-efficient routing scheme should be designed to prolong the lifetime of LEO satellites while satisfying the end-to-end delay constraint of each data flow. Thus, the routing problem significantly affects the performance of an LSN. In this paper, we formulate this problem as a nonlinear programming problem, which aims to minimize the energy consumption of an LSN caused by data transmission among LEO satellites while satisfying the end-to-end delay constraint of each data flow. We propose a centralized LSN (CLSN) architecture that utilizes a routing manager in a medium-earth-orbit (MEO) satellite to monitor the condition of LEO satellites and intelligently decide the routing path for each routing request event from LEO satellites. We further propose an energy-efficient event-driven deep reinforcement learning (DRL), Deep Deterministic Policy Gradient (DDPG), enhanced Dijkstra’s algorithm (EEDRL-Dijkstra) in the routing manager to deal with the stochastic routing request event arrivals so as to achieve long-term optimization of the energy consumption performance of an LSN while satisfying the end-to-end delay constraint of each data flow. The simulation results show that in the proposed CLSN architecture, the proposed EEDRL-Dijkstra significantly improves the energy consumption performance as compared to Dijkstra’s algorithm and random routing. Both the proposed EEDRL-Dijkstra and Dijkstra’s algorithm can satisfy the end-to-end delay constraint of each data flow.

Service function chain (SFC), which consists of an ordered combination of virtual network functions (VNFs), has been incorporated with multi-access edge computing (MEC) to provide a variety of low latency services in a more flexible manner in 5G MEC networks (MNs). However, embedding of the VNFs of an SFC significantly affects the quality of experience (QoE) of user equipments (UEs). In this paper, we formulate the SFC embedding problem as a nonlinear programming problem, aiming to maximize the average acceptance rate (AR) of an MN while satisfying the service downtime and end-to-end delay constraints of each UE. We propose a centralized MN architecture that utilizes an SFC embedding manager to monitor the condition of MEC servers and intelligently make the embedding decision for each request event. We further propose an event-driven deep reinforcement learning (DRL) based mobility-aware SFC embedding scheme (EDRL-MSES) which includes placement and migration agents, to achieve long-term optimization of the QoE of UEs. The placement agent intelligently deploys the VNFs of an SFC to MEC servers for each placement request event, and the migration agent intelligently migrates the VNFs’ states of an SFC and redeploys those VNFs to new MEC servers if necessary for each migration request event. We utilize a DRL technique, Deep Deterministic Policy Gradient, in placement and migration agents to handle the stochastic request event arrivals and the network condition. The simulation results show that the proposed EDRL-MSES significantly improves the AR performance as compared to FMC, which is the best available related work.

Spectrum sharing has been proposed to effectively utilize the limited licensed spectrum resource and provide extra capacity for mobile network operator (MNO) who requires more spectrum in 5G network. In spectrum sharing, the initialized licensed spectrum of each MN O is divided into a reserved and shared spectrum. The reserved spectrum is for the private use of an MNO, while the shared spectrum of all MNOs forms a spectrum pool which will be dynamically used by MNO who requires an additional spectrum. However, the spectrum pool management problem significantly affects the efficiency of spectrum sharing among multi-MNO. In this paper, we formulate this problem into a non-linear programming problem that aims to maximize the average data rate of UEs, which subscribe to an MNO sending the spectrum request event, of all events from all MNOs. We propose an event-driven Deep Reinforcement Learning based spectrum sharing mechanism (EDRL-SSM) that utilizes a spectrum pool manager to effectively manage the spectrum pool to achieve long-term optimization of data rate performance of UEs. The spectrum pool manager intelligently allocates spectrum pool resource for each incoming spectrum request event. The proposed EDRL-SSM fully utilizes a DRL technique, Deep Deterministic Policy Gradient (DDPG), to deal with stochastic spectrum request event arrivals in the spectrum pool manager. The simulation results show that the proposed EDRL-SSM can significantly improve the data rate performance of UEs as compared to greedy spectrum pool allocation and without spectrum sharing under both identical and different initialized allocation of licensed spectrum scenarios.

In 5G, eMBB services are defined to support high data rate, while URLLC services focus on low latency and high reliability. Multiplexing these two services on the same wireless radio frequency leads to a challenging radio resource allocation problem due to their heterogeneous requirements. In this paper, we formulate this problem as two non-linear programming non-convex optimization subproblems, aiming to maximize the average data rate of all eMBB services while satisfying the delay constraint of each URLLC service. We propose an event-driven deep reinforcement learning (DRL) based resource allocation mechanism (EDRL-RAM), which includes two schedulers: eMBB scheduler and URLLC scheduler to achieve long-term optimization of eMBB and URLLC performance. The eMBB scheduler will intelligently allocate resource for each incoming eMBB event, and the URLLC scheduler will intelligently distribute the incoming URLLC event during the ongoing transmissions of eMBB services. The proposed EDRL-RAM makes full use of four different DRL techniques to deal with stochastic event arrivals and network conditions, namely Policy Gradient (PG), Deep Q-learning Network (DQN), Advantage Actor Critic (A2C), and Deep Deterministic Policy Gradient (DDPG), in both eMBB and URLLC schedulers. The simulation results show that in our proposed EDRL-RAM, the order of data rate and delay performance is DDPG, A2C, DQN, and PG. The data rate and delay performance of the proposed EDRL-RAM utilizing any of the four DRL techniques are better than SAFE-TS, which is the best available related work.

In this paper, we study fine-grained offloading for multi-access edge computing (MEC) in 5G. Existing works for computation offloading is on a per-task basis and do not take into account the execution order among tasks in one application. Fine-grained offloading, on the other hand, considers the task structure of an application upon making offloading decision and may only offload computation-hungry tasks to the MEC, thus making better use of system resource. To solve the problem, we propose an online solution based on Actor-Critic Federated Learning, called AC-Federate. In AC-Federate, we consider a multi-MEC network in which each edge node trains a model-free advantage Actor-Critic (AC) model based on local data. The AC model of each edge node jointly optimizes the continuous actions (i.e., radio and computing resource allocations) and the discrete action (i.e., offloading decision), and trains the model with a weighted loss function. To further improve the inference accuracy of the AC model, each edge node uploads the gradients of its actor and critic neural networks to a central controller in an asynchronous manner. The central controller then ensembles the collected gradients from different edge nodes and updates all edge nodes with the integrated network parameters. Simulation results show that the proposed AC-Federate outperforms DDPG and others in terms of delay, energy consumption, and mixed consideration of delay and energy consumption performance even when the number of UEs is very large.