This paper focuses on advancing reinforcement learning for challenging environments characterized by partial observability and non-stationarity, such as dynamic spectrum access (DSA). In the literature, the Deep Recurrent Q-Network was introduced to capitalize on the inherent temporal correlations present in DSA. Nevertheless, its practicality is still questionable due to sample inefficiency and slow convergence. We introduce Dyna-ESN, leveraging both model-based and model-free methods by employing Reservoir Computing for generative modeling. Specifically, we utilize Echo State Networks (ESNs) to synthesize samples for enhancing the sample efficiency of a model-free Deep Echo State Q-network, enabling effective operation of agent given limited genuine relevant samples obtained through interaction with environment. To mitigate potential adverse effects of synthetic samples, an evaluation algorithm guides the sample selection process, ensuring reliability. A sample augmentation technique is also introduced to allow agents to collect adequate samples despite controlling the sensing rate and duration of secondary transmissions. Our analysis explores trade-offs between data evaluation and sample efficiency, as well as the bias-variance trade-off of the model, identifying optimal design parameters. Evaluating the performance of Dyna-ESN in DSA scenarios demonstrates its performance benefits over existing methods, paving the way for more efficient and effective techniques in complex dynamic environments.

Human respiration rate (RR) estimation is essential for various health care applications, such as sleep apnea detection and chronic obstructive pulmonary disease early diagnose. Recently, radio frequency based RR estimation has achieved high accuracy for single-person RR detection. However, multi-person RR estimation is still the obstacle blocking the wide commercialization of RF sensing based RR solution. In this paper, a novel multi-person RR estimation algorithm that can overcome the frequency resolution limit is present. The proposed algorithm is not only analytically justified but also verified in a real test-bed involving commercial off-the-shelf WiFi devices. Extensive experiment results show a 98% accuracy in people-counting and a root mean square error (RMSE) of 0.13 breath per minute (bpm) on RR detection. To the best of our knowledge, this is the first WiFi sensing work that can detect different people who share the same RR by only using a single pair of transmit and receive antenna.

Due to its ubiquitous and contact-free nature, the use of WiFi infrastructure for performing sensing tasks has tremendous potential. However, the channel state information (CSI) measured by a WiFi receiver suffers from errors in both its gain and phase, which can significantly hinder sensing tasks. By analyzing these errors from different WiFi receivers, a mathematical model for these gain and phase errors is developed in this work. Based on these models, several theoretically justified preprocessing algorithms for correcting such errors at a receiver and, thus, obtaining clean CSI are presented. Simulation results show that at typical system parameters, the developed algorithms for cleaning CSI can reduce noise by 40% and 200%, respectively, compared to baseline methods for gain correction and phase correction, without significantly impacting computational cost. The superiority of the proposed methods is also validated in a real-world test bed for respiration rate monitoring (an example sensing task), where they improve the estimation signal-to-noise ratio by 20% compared to baseline methods.

In dynamic spectrum access (DSA) networks, secondary users (SUs) need to opportunistically access primary users’ (PUs) radio spectrum without causing significant interference. Since the SU-PU interaction is limited, deep reinforcement learning has been introduced to help SUs conduct spectrum access. Specifically, deep recurrent Q network (DRQN) has been utilized in DSA networks for SUs to aggregate information from recent experiences to make spectrum access decisions. DRQN is notorious for its sample efficiency since it needs a rather large number of training samples to tune its parameters which is a computationally demanding task. Deep echo state network (DEQN) has been introduced for DSA networks to address the sample efficiency issue of DRQN. In this work, we compare the convergence of DRQN and DEQN by comparing the upper bounds we obtain on their covering number, a notion of richness. Furthermore, we introduce a method to determine the right hyper-parameters for DEQN, providing system design guidance for DEQN-based DSA networks. Extensive performance evaluation confirms that DEQN-based DSA strategy is the superior choice with regard to computational power while outperforming DRQN-based ones.

Mobility load balancing (MLB) aims to solve the problem of uneven resource utilization in cellular networks. Since network dynamics are usually complicated and non-stationary, conventional model-based MLB methods fail to cover all scenarios of cellular networks. On the other hand, deep reinforcement learning (DRL) can provide a flexible framework to learn to distribute cell load evenly without explicit modeling of the underlying network dynamics. In this paper, we introduce a novel decentralized DRL-based MLB method where each cell has a DRL agent to learn its handover parameters and antenna tilt angle. As the number of cells increases, the decentralized framework is more computationally efficient than its centralized counterpart by dividing the action space. Furthermore, our designed decentralized DRL architecture only requires readily known information defined in existing cellular standards, and it can achieve a more balanced cell load distribution than the centralized DRL one by using individual reward functions. To provide realistic performance evaluation, a network simulator is introduced strictly following the Third Generation Partnership Project (3GPP) specifications. Furthermore, field data is used to construct the underlying cellular environment. Extensive evaluations have been conducted to demonstrate the fact that the introduced decentralized DRL-based MLB method can achieve a more balanced cell load distribution and a better performance of edge users than the state-of-the-art MLB methods.

Dynamic spectrum access (DSA) has been introduced as a promising technology that allows a secondary system to access the licensed spectrum of the primary system to improve spectrum utilization. In this paper, we introduce Fed-MADRL by incorporating federated learning (FL) and multi-agent deep reinforcement learning (MADRL) to design a collaborative DSA strategy. Our Fed-MADRL scheme employs FL to enable multiple users to collaboratively optimize the system goal without sharing their training data. By keeping all the training data at the user end, FL improves the communication efficiency and strengthens user data privacy. To further reduce the communication overheads, each user only shares quantized information. We provide the convergence analysis to characterize the trade-off between the communication efficiency and the system performance. In particular, we show that the introduced method converges at a rate$\mathcal {O}(1/K^{1/4})$, where$K$is the number of FL iterations. To the best of our knowledge, Fed-MADRL is the first work that utilizes FL in DSA networks under quantized communication. Performance evaluation results show that the introduced Fed-MADRL method outperforms the independent learning method and achieves comparable performance with the centralized MADRL method, which requires much higher communication overheads.

Dynamic spectrum access (DSA) has emerged as a promising solution for spectrum usage enhancement by allowing opportunistic access of secondary users to the licensed spectrum. In this paper, we introduce Fed-MADRL, a collaborative DSA technique that exploits both federated learning (FL) and multiagent deep reinforcement learning (MADRL). FL allows numerous users to collaborate on the system goal optimization without sharing their training data. By keeping all training data at the user's end, FL simultaneously enhances communication efficiency and protects data privacy. To further reduce communication costs, each user in Fed-MADRL only shares quantized data. To the best of our knowledge, Fed-MADRL is the first effort that employs FL in DSA networks with quantized communication. Simulation results show that the introduced Fed-MADRL approach beats the independent learning method and provides comparable results to the synchronous FL method, which involves significantly greater communication overheads.

Deep reinforcement learning (DRL) has been shown to be successful in many application domains. Combining recurrent neural networks (RNNs) and DRL further enables DRL to be applicable in non-Markovian environments by capturing temporal information. However, training of both DRL and RNNs is known to be challenging requiring a large amount of training data to achieve convergence. In many targeted applications, such as those used in the fifth-generation (5G) cellular communication, the environment is highly dynamic, while the available training data is very limited. Therefore, it is extremely important to develop DRL strategies that are capable of capturing the temporal correlation of the dynamic environment requiring limited training overhead. In this article, we introduce the deep echo state Q-network (DEQN) that can adapt to the highly dynamic environment in a short period of time with limited training data. We evaluate the performance of the introduced DEQN method under the dynamic spectrum sharing (DSS) scenario, which is a promising technology in 5G and future 6G networks to increase the spectrum utilization. Compared with conventional spectrum management policy that grants a fixed spectrum band to a single system for exclusive access, DSS allows the secondary system to share the spectrum with the primary system. Our work sheds light on the application of an efficient DRL framework in highly dynamic environments with limited available training data.

Current studies that apply reinforcement learning (RL) to dynamic spectrum access (DSA) problems in wireless communications systems mainly focus on model-free RL (MFRL). However, in practice, MFRL requires a large number of samples to achieve good performance making it impractical in real-time applications such as DSA. Combining model-free and model-based RL can potentially reduce the sample complexity while achieving a similar level of performance as MFRL as long as the learned model is accurate enough. However, in a complex environment, the learned model is never perfect. In this article, we combine model-free and model-based RL, and introduce an algorithm that can work with an imperfectly learned model to accelerate the MFRL. Results show our algorithm achieves higher sample efficiency than the standard MFRL algorithm and the Dyna algorithm (a standard algorithm integrating model-based RL and MFRL) with much lower computation complexity than the Dyna algorithm. For the extreme case where the learned model is highly inaccurate, the Dyna algorithm performs even worse than the MFRL algorithm while our algorithm can still outperform the MFRL algorithm.

In this paper, we introduce a reservoir computing (RC) structure, namely, windowed echo state network (WESN), for multiple-input-multiple-output orthogonal frequency-division multiplexing (MIMO-OFDM) symbol detection. We show that adding buffers in input layers is able to bring an enhanced short-term memory (STM) to the standard echo state network. A unified training framework is developed for the introduced WESN MIMO-OFDM symbol detector using both comb and scattered patterns, where the training set size is compatible with those adopted in 3GPP LTE/LTE-Advanced standards. Complexity analysis demonstrates the advantages of WESN based symbol detector over state-of-the-art symbol detectors when the number of OFDM sub-carriers is large, where the benchmark methods are chosen as linear minimum mean square error (LMMSE) detection and sphere decoder. Numerical evaluations suggest that WESN can significantly improve the symbol detection performance as well as effectively mitigate model mismatch effects using very limited training symbols.

Power control plays an important role in improving the system throughput in communication system since co-channel interference is a major limitation to the system throughput. The power control problem of maximizing the system throughput in the multiuser and multichannel communication system is a highly complicated nonconvex problem since user are interfered with one another if operating in the same wireless channel. We reformulate the nonconvex objective function of this problem as a difference of two convex functions, which is called DC (difference of convex function) programming. To reduce the computation complexity in the high dimensional space, we introduce an alternative power allocation scheme to search in the low dimensional space, where each user updates its power sequentially. A global optimal power allocation is found by utilizing the branch-and- bound algorithm for each user while taking other users' power allocation as constant value. Furthermore, we incorporate each user's maximum power and minimum data rate constraint into the optimization framework. We found that the minimum data rate constraint of each user can be turned into multiple linear inequalities and then be added to the DC programming optimization framework. The simulation results show that our introduced method achieves the highest sum data rate compared to the state-of-the-art methods, including iterative water filling and geometric programming.

Dynamic spectrum access (DSA) is regarded as an effective and efficient technology to share radio spectrum among different networks. As a secondary user (SU), a DSA device will face two critical problems: 1) avoiding causing harmful interference to primary users (PUs) and 2) conducting effective interference coordination with other SUs. These two problems become even more challenging for a distributed DSA network where there is no centralized controllers for SUs. In this paper, we investigate communication strategies of a distributive DSA network under the presence of spectrum sensing errors. To be specific, we apply the powerful machine learning tool, deep reinforcement learning (DRL), for SUs to learn “appropriate” spectrum access strategies in a distributed fashion assuming NO knowledge of the underlying system statistics. Furthermore, a special type of recurrent neural network, called the reservoir computing (RC), is utilized to realize DRL by taking advantage of the underlying temporal correlation of the DSA network. Using the introduced machine learning-based strategy, SUs could make spectrum access decisions distributedly relying only on their own current and past spectrum sensing outcomes. Through extensive experiments, our results suggest that the RC-based spectrum access strategy can help the SU to significantly reduce the chances of collision with PUs and other SUs. We also show that our scheme outperforms the myopic method which assumes the knowledge of system statistics, and converges faster than the Q-learning method when the number of channels is large.