Markov Decision Processes (MDPs) serve as the mathematical foundation of Reinforcement learning (RL), where a Markov process with defined states is used to model the system and the actions to be taken affect the state transitions and the corresponding rewards. The RL and deep RL (DRL) can produce the high-performing action policy to maximize the long-term reward. Although RL/DRL have been widely applied to communication and computer systems, a key limitation is that the system under consideration often does not satisfy the required mathematical properties, thus making the MDP inexact and the derived policy flawed. Therefore, we consider the periodic Markov Decision Process (pMDP), where the evolution of the underlying process and model parameters for the pMDP demonstrate some forms of periodic characteristics (e.g., periodic job arrivals and available resources) which violate the Markov property. To obtain the optimal policies for the pMDP, a policy gradient method with a multi-policy solution framework is proposed, and a deep-learning method is developed to improve the effectiveness and stability of the proposed solution. Furthermore, a layer-sharing strategy is proposed to reduce the storage complexity by reducing the number of parameters in the neural networks. The deep-learning method is applied to achieve the near-optimal allocation of resources to arriving computational tasks in a network setting corresponding to the software-defined network (SDN). Evaluation results reveal that the proposed technique is valid and capable of outperforming a baseline method that employs a single policy by 31% on average.

High-quality data collection is crucial for mobile crowd sensing (MCS) with various applications like smart cities and emergency rescues, where various unmanned mobile terminals (MTs), e.g., driverless cars and unmanned aerial vehicles (UAVs), are equipped with different sensors that aid to collect data. However, they are limited with fixed carrying capacity, and thus, MT's energy resource and sensing range are constrained. It is quite challenging to navigate a group of MTs to move around a target area to maximize their total amount of collected data with the limited energy reserve, while geographical fairness among those point-of-interests (PoIs) should also be maximized. It is even more challenging if fully distributed execution is enforced, where no central control is allowed at the backend. To this end, we propose to leverage emerging deep reinforcement learning (DRL) techniques for directing MT's sensing and movement and to present a novel and highly efficient control algorithm, called energy-efficient distributed MCS (Edics). The proposed neural network integrates convolutional neural network (CNN) for feature extraction and then makes decision under the guidance of multi-agent deep deterministic policy gradient (DDPG) method in a fully distributed manner. We also propose two enhancements into Edics with N-step return and prioritized experienced replay buffer. Finally, we evaluate Edics through extensive simulations and found the appropriate set of hyperparameters in terms of number of CNN hidden layers and neural units for all the fully connected layers. Compared with three commonly used baselines, results have shown its benefits.

Unmanned aerial vehicles (UAVs) can be used to serve as aerial base stations to enhance both the coverage and performance of communication networks in various scenarios, such as emergency communications and network access for remote areas. Mobile UAVs can establish communication links for ground users to deliver packets. However, UAVs have limited communication ranges and energy resources. Particularly, for a large region, they cannot cover the entire area all the time or keep flying for a long time. It is thus challenging to control a group of UAVs to achieve certain communication coverage in a long run, while preserving their connectivity and minimizing their energy consumption. Toward this end, we propose to leverage emerging deep reinforcement learning (DRL) for UAV control and present a novel and highly energy-efficient DRL-based method, which we call DRL-based energy-efficient control for coverage and connectivity (DRL-EC3). The proposed method 1) maximizes a novel energy efficiency function with joint consideration for communications coverage, fairness, energy consumption and connectivity; 2) learns the environment and its dynamics; and 3) makes decisions under the guidance of two powerful deep neural networks. We conduct extensive simulations for performance evaluation. Simulation results have shown that DRL-EC3significantly and consistently outperform two commonly used baseline methods in terms of coverage, fairness, and energy consumption.