In this paper, a novel experienced deep reinforcement learning (deep-RL) framework is proposed to provide model-free resource allocation for ultra reliable low latency communication (URLLC-6G) in the downlink of a wireless network. The goal is to guarantee high end-to-end reliability and low end-to-end latency, under explicit data rate constraints, for each wireless user without any models of or assumptions on the users' traffic. In particular, in order to enable the deep-RL framework to account for extreme network conditions and operate in highly reliable systems, a new approach based on generative adversarial networks (GANs) is proposed. This GAN approach is used to pre-train the deep-RL framework using a mix of real and synthetic data, thus creating an experienced deep-RL framework that has been exposed to a broad range of network conditions. The proposed deep-RL framework is particularly applied to a multi-user orthogonal frequency division multiple access (OFDMA) resource allocation system. Formally, this URLLC-6G resource allocation problem in OFDMA systems is posed as a power minimization problem under reliability, latency, and rate constraints. To solve this problem using experienced deep-RL, first, the rate of each user is determined. Then, these rates are mapped to the resource block and power allocation vectors of the studied wireless system. Finally, the end-to-end reliability and latency of each user are used as feedback to the deep-RL framework. It is then shown that at the fixed-point of the deep-RL algorithm, the reliability and latency of the users are near-optimal. Moreover, for the proposed GAN approach, a theoretical limit for the generator output is analytically derived. Simulation results show how the proposed approach can achieve near-optimal performance within the rate-reliability-latency region, depending on the network and service requirements. The results also show that the proposed experienced deep-RL framework is able to remove the transient training time that makes conventional deep-RL methods unsuitable for URLLC-6G. Moreover, during extreme conditions, it is shown that the proposed, experienced deep-RL agent can recover instantly while a conventional deep-RL agent takes several epochs to adapt to new extreme conditions.

When deployed as reflectors for existing wireless base stations (BSs), reconfigurable intelligent surfaces (RISs) can be a promising approach to achieve high spectrum and energy efficiency. However, due to the large number of RIS elements, the joint optimization of the BS and reflector RIS configuration is challenging. In essence, the BS transmit power and RIS's reflecting configuration must be optimized so as to improve users' data rates and reduce the BS power consumption. In this paper, the problem of energy efficiency optimization is studied in an RIS-assisted cellular network endowed with an RIS reflector powered via energy harvesting technologies. The goal of this proposed framework is to maximize the average energy efficiency by enabling a BS to determine the transmit power and RIS configuration, under uncertainty on the wireless channel and harvested energy of the RIS system. To solve this problem, a novel approach based on deep reinforcement learning is proposed, in which the BS receives the state information, consisting of the users' channel state information feedback and the available energy reported by the RIS. Then, the BS optimizes its action composed of the BS transmit power allocation and RIS phase shift configuration using a neural network. Due to the intractability of the formulated problem under uncertainty, a case study is conducted to analyze the performance of the studied RIS-assisted downlink system by asymptotically deriving the upper bound of the energy efficiency. Simulation results show that the proposed framework improves energy efficiency up to 77.3% when the number of RIS elements increases from 9 to 25.

In this paper, a novel deep reinforcement learning (deep-RL) framework is proposed to provide model-free ultra reliable low latency communication (URLLC) in the downlink of an orthogonal frequency division multiple access (OFDMA) system. The proposed deep-RL framework can guarantee high end-to-end reliability and low end-to-end latency, under data rate constraints, for each user in the cellular system without any models of or assumptions on the users' traffic. Using the proposed model-free approach, the users' traffic is predicted by the deep-RL framework and subsequently used in the resource allocation, irrespective of the actual underlying model. The problem is posed as a power minimization problem under reliability, latency, and rate constraints. To solve this problem using deep-RL, first, the rate of each user is determined. Then, these rates are mapped to the resource block and power allocation vectors of the studied OFDMA system. Finally, the end-to-end reliability and latency of each user are used as a feedback to the deep-RL framework. It is shown that at the fixed-point of the deep-RL algorithm, the reliability and latency of the users are guaranteed. Simulation results show how the proposed approach can achieve any feasible point in the rate-reliability-latency region, depending on the network and service requirements. For example, for a 7 Mbps rate guarantee, the results show that the proposed algorithm can provide ultra-reliable low latency communication with a delay of 8 milliseconds and a reliability of 98%.

Human-centric applications such as virtual reality and immersive gaming are central to future wireless networks. Common features of such services include: 1) their dependence on the human user’s behavior and state and 2) their need for more network resources compared to conventional applications. To successfully deploy such applications over wireless networks, the network must be made cognizant of not only the quality-of-service (QoS) needs of the applications, but also of the perceptions of thehuman userson this QoS. In this paper, by explicitly modeling the limitations of the human brain, a concrete measure for the delay perception of human users is introduced. Then, a learning method, called probability distribution identification, is developed to find a probabilistic model for this delay perception based on the brain features of a human user. Given the learned model for the delay perception of the human brain, a brain-aware resource management algorithm based on Lyapunov optimization is proposed for allocating radio resources to human users while minimizing the transmit power and taking into account the reliability of both machine type devices and human users. Then, a closed-form relationship between the reliability measure and wireless physical layer metrics of the network is derived. Simulation results show that a brain-aware approach can yield savings of up to 78% in power compared to the system that only considers QoS metrics. The results also show that, compared with QoS-aware, brain-unaware systems, the brain-aware approach can save substantially more power in low-latency systems.

In this paper, a novel concept of three-dimensional (3D) cellular networks, that integrate drone base stations (drone-BS) and cellular-connected drone users (drone-UEs), is introduced. For this new 3D cellular architecture, a novel framework for network planning for drone-BSs and latency-minimal cell association for drone-UEs is proposed. For network planning, a tractable method for drone-BSs' deployment based on the notion of truncated octahedron shapes is proposed, which ensures full coverage for a given space with a minimum number of drone-BSs. In addition, to characterize frequency planning in such 3D wireless networks, an analytical expression for the feasible integer frequency reuse factors is derived. Subsequently, an optimal 3D cell association scheme is developed for which the drone-UEs' latency, considering transmission, computation, and backhaul delays, is minimized. To this end, first, the spatial distribution of the drone-UEs is estimated using a kernel density estimation method, and the parameters of the estimator are obtained using a cross-validation method. Then, according to the spatial distribution of drone-UEs and the locations of drone-BSs, the latency-minimal 3D cell association for drone-UEs is derived by exploiting tools from an optimal transport theory. The simulation results show that the proposed approach reduces the latency of drone-UEs compared with the classical cell association approach that uses a signal-to-interference-plus-noise ratio (SINR) criterion. In particular, the proposed approach yields a reduction of up to 46% in the average latency compared with the SINR-based association. The results also show that the proposed latency-optimal cell association improves the spectral efficiency of a 3D wireless cellular network of drones.

In this paper, a novel concept of three-dimensional (3D) cellular networks, that integrate drone base stations (drone-BS) and drone users (drone-UEs), is introduced. For this new 3D cellular network architecture, a novel framework for the deployment of drone-BSs and latency-minimal cell association for drone-UEs is proposed. For drone-BSs' deployment, a tractable method based on the notion of truncated octahedron shapes is proposed that ensures full coverage for a given space with minimum number of drone-BSs. Then, an optimal 3D cell association scheme is determined such that the drone-UEs' latency, considering transmission, computation, and backhaul latencies, is minimized. In particular, using optimal transport theory, the optimal 3D cell partitions are derived according to the spatial distribution of drone-UEs and the drone-BSs' locations. Simulation results show that the proposed approach reduces the latency of drone-UEs compared to the classical cell association approach that uses a signal-to-interference-plus-noise ratio (SINR) criterion. In particular, the proposed approach yields a reduction of up to 46% in average latency compared to the SINR-based association. Also, it is shown that the proposed latency-optimal cell association improves the spectral efficiency of a 3D wireless cellular network of drones.

In this paper, a distributed, minimum overhead power control algorithm for underlay cognitive radio networks (CRNs) having multiple primary and secondary users is proposed. The problem is formulated as a noncooperative game and a learning algorithm is proposed for optimizing the power allocation of secondary users. In the considered network, secondary users (SUs) do not have full information on the interference and power control strategies of other SUs. As a result, they update their strategy using a simple feedback from the primary user base station that provides the total interference. Although there is no cooperation among secondary users, it is shown that, under incomplete information, the proposed learning algorithm converges to the strategy of the players in the Nash equilibrium of the complete information case. The Nash equilibrium point is derived analytically, and then it is demonstrated that, although each user individually tries to maximize its own payoff, at the end, the proposed algorithm will converge to the complete information game Nash equilibrium point. It is also shown that the algorithm will be capable of adapting to a time-varying environment if some conditions on the SUs' processing power are satisfied. This is due to the slotted time assumption of the algorithm. Simulation results are then used to corroborate the analytical derivations.