With the global aim of reducing carbon emissions, energy saving for communication systems has gained tremendous attention. Efficient energy-saving solutions are not only required to accommodate the fast growth in communication demand but solutions are also challenged by the complex nature of the load dynamics. Recent reinforcement learning (RL)-based methods have shown promising performance for network optimization problems, such as base station energy saving. However, a major limitation of these methods is the requirement of online exploration of potential solutions using a high-fidelity simulator or the need to perform exploration in a real-world environment. We circumvent this issue by proposing an offline reinforcement learning energy saving (ORES) framework that allows us to learn an efficient control policy using previously collected data. We first deploy a behavior energy-saving policy on base stations and generate a set of interaction experiences. Then, using a robust deep offline reinforcement learning algorithm, we learn an energy-saving control policy based on the collected experiences. Results from experiments conducted on a diverse collection of communication scenarios with different behavior policies showcase the effectiveness of the proposed energy-saving algorithms.

With the increasing use of data-intensive mobile applications and the number of mobile users, the demand for wireless data services has been increasing exponentially in recent years. In order to address this demand, a large number of new cellular base stations are being deployed around the world, leading to a significant increase in energy consumption and greenhouse gas emission. Consequently, energy consumption has emerged as a key concern in the fifth-generation (5G) network era and beyond. Reinforcement learning (RL), which aims to learn a control policy via interacting with the environment, has been shown to be effective in addressing network optimization problems. However, for reinforcement learning, especially deep reinforcement learning, a large number of interactions with the environment are required. This often limits its applicability in the real world. In this work, to better deal with dynamic traffic scenarios and improve real-world applicability, we propose a transfer deep reinforcement learning framework for energy optimization in cellular communication networks. Specifically, we first pre-train a set of RL-based energy-saving policies on source base stations and then transfer the most suitable policy to the given target base station in an unsupervised learning manner. Experimental results demonstrate that base station energy consumption can be reduced significantly using this approach.

The association of mobile devices with network resources (e.g., base stations, frequency bands/channels), known as load balancing, is critical to reduce communication traffic congestion and network performance. Reinforcement learning (RL) has shown to be effective for communication load balancing and achieves better performance than currently used rule-based methods, especially when the traffic load changes quickly. However, RL-based methods usually need to interact with the environment for a large number of time steps to learn an effective policy and can be difficult to tune. In this work, we aim to improve the data efficiency of RL-based solutions to make them more suitable and applicable for real-world applications. Specifically, we propose a simple, yet efficient and effective deep RL-based wireless network load balancing framework. In this solution, a set of good initialization values for control actions are selected with some cost-efficient approach to center the training of the RL agent. Then, a deep RL-based agent is trained to find offsets from the initialization values that optimize the load balancing problem. Experimental evaluation on a set of dynamic traffic scenarios demonstrates the effectiveness and efficiency of the proposed method.

Communication load balancing aims to balance the load between different available resources, and thus improve the quality of service for network systems. After formulating the load balancing (LB) as a Markov decision process problem, reinforcement learning (RL) has recently proven effective in addressing the LB problem. To leverage the benefits of classical RL for load balancing, however, we need an explicit reward definition. Engineering this reward function is challenging, because it involves the need for expert knowledge and there lacks a general consensus on the form of an optimal reward function. In this work, we tackle the communication load balancing problem from an inverse reinforcement learning (IRL) approach. To the best of our knowledge, this is the first time IRL has been successfully applied in the field of communication load balancing. Specifically, first, we infer a reward function from a set of demonstrations, and then learn a reinforcement learning load balancing policy with the inferred reward function. Compared to classical RL-based solution, the proposed solution can be more general and more suitable for real-world scenarios. Experimental evaluations implemented on different simulated traffic scenarios have shown our method to be effective and better than other baselines by a considerable margin.

The anticipated huge number of devices and large traffic volumes impose new challenges on the communication system requirements and design. One of the main requirements of massive machine-type communication (mMTC) is to support network energy efficiency. Data compression is a widely adopted technique that enables higher energy efficiency, lower latency, and better bandwidth utilization. Unfortunately, the current compression techniques are mainly designed for human-type communications (HTC). Therefore, they consider the reconstruction fidelity, rather than the accuracy of inferred decisions, as the sole performance metric. In this work, we propose a novel encoder for data compression in mMTC communications, which is termed Distillation Encoder (DE). Unlike prior work, the design of the proposed DE aims to achieve high compression ratios while preserving the accuracy of the inferred decisions. DE inherits the knowledge of a large teacher model (trained on the raw data) through knowledge distillation. Evaluating the proposed framework on several public datasets shows a clear performance advantage compared with baseline models in terms of the inferred decision accuracy and generalizing to yet-unseen data. Moreover, the DE can be applied to learn efficient quantizers, as shown in the results.

The modern power system is transitioning towards increasing penetration of renewable energy generation and demand from different types of electrical appliances. With this transition, residential load forecasting, especially short-term load forecasting (STLF), is becoming more and more challenging and important. Accurate short-term load forecasting can help improve energy dispatching efficiency and, as a consequence, reduce overall power system operation cost. Most current load forecasting algorithms assume that there is a large amount of training data available upon which to learn a reliable load forecasting model. However, this assumption can be challenging for real-world applications. In this work, we first propose the use of transfer learning and an attention mechanism to improve short-term load forecasting for a target domain with only a limited amount of available data. Furthermore, we extend the proposed method to utilize heterogeneous features which enables the approach to deal with more complex scenarios in the real world. Experimental results using real-world data sets show that the proposed methods can improve forecasting accuracy by a large margin over several existing baselines.

Due to the rapid increase in wireless communication traffic in recent years, load balancing is becoming increasingly important for ensuring the quality of service. However, variations in traffic patterns near different serving base stations make this task challenging. On one hand, crafting a single control policy that performs well across all base station sectors is often difficult. On the other hand, maintaining separate controllers for every sector introduces overhead, and leads to redundancy if some of the sectors experience similar traffic patterns. In this paper, we propose to construct a concise set of controllers that cover a wide range of traffic scenarios, allowing the operator to select a suitable controller for each sector based on local traffic conditions. To construct these controllers, we present a method that clusters similar scenarios and learns a general control policy for each cluster. We use deep reinforcement learning (RL) to first train separate control policies on diverse traffic scenarios, and then incrementally merge together similar RL policies via knowledge distillation. Experimental results show that our concise policy set reduces redundancy with very minor performance degradation compared to policies trained separately on each traffic scenario. Our method also outperforms handcrafted control parameters, joint learning on all tasks, and two popular clustering methods.

Mobile edge computing (MEC) networks have been recently adopted to accommodate the fast-growing number of mobile devices performing complicated tasks with limited hardware capability. Recently, edge nodes with communication, computation, and caching capacities are starting to be deployed in MEC networks. Due to the physical separation of these resources, efficient coordination and scheduling are important for efficient resource utilization and optimal network performance. In this paper, we study mobility load balancing for communication, computation, and caching-enabled heterogeneous MEC networks. Specifically, we propose to tackle this problem via a multi-agent deep reinforcement learning-based framework. Users served by overloaded edge nodes are handed over to less loaded ones, to minimize the load in the most loaded base station in the network. In this framework, the handover decision for each user is made based on the user’s own observation which comprises the user’s task at hand and the load status of the MEC network. Simulation results show that our proposed multi-agent deep reinforcement learning-based approach can reduce the time-average maximum load by up to 30% and the end-to-end delay by 50% compared to baseline algorithms.

With the increasing popularity of electric vehicles and the growing trend of working from home, electricity consumption in the residential sector is expected to continue to grow rapidly over the next few years. As a consequence, short-term residential load forecasting is becoming even more vital for the reliability and sustainability of the smart grid. Although deep learning models have shown impressive success in different areas including short-term electric load forecasting, such models require a large amount of training data. For many real-world load forecasting cases, we may not have enough training data to learn a reliable forecasting model. In this paper, we address this challenge through the use of boosting-based transfer learning with multiple sources. We first train a set of deep regression models on source houses that can provide relatively abundant data. We then transfer these learned models via the boosting framework to support data-scarce target houses. The transfer process is selective and customized for each target house to minimize the potential for negative transfer. Experimental results, based on real-world residential data sets, show that the proposed method can significantly improve forecasting accuracy.

Load balancing has become a key technique to handle the increasing traffic demand and improve the user experience. It evenly distributes the traffic across network resources by offloading users from overloaded base stations or channels to less crowded ones. Load balancing is a multi-objective optimization problem involving the automatic adjustment of several parameters to simultaneously maximize multiple network performance indicators. However, the existing methods mostly rely on single-objective approaches which lead to sub-optimal solutions. In this paper, we introduce the first multi-objective reinforcement learning (MORL) framework for load balancing. Specifically, we propose a solution based on meta-reinforcement learning (meta-RL) to learn a general policy capable of quickly adapting to new trade-offs between the objectives. We further enhance the generalization of our proposed solution using policy distillation techniques. To showcase the effectiveness of our framework, experiments are conducted based on real-world traffic scenarios. Our results show that our load balancing framework can (i) significantly outperform the existing rule-based and single-objective solutions, (ii) compute better Pareto front approximations compared to MORL baselines, and (iii) quickly adapt to new objective trade-offs.

By placing the computing, storage and networking resources close to the end users, distributed edge computing greatly benefits the performance of 5G communication systems. However, as a tradeoff, resources on the edge are usually limited and imbalanced among the heterogeneous edge nodes. To overcome this drawback, this paper proposes a Transfer Learning based Prediction (TLP) framework that allows the edge nodes to share their resources and data in an efficient manner. In particular, the TLP framework focuses on the prediction of the future traffic load, which is a key reference for many automated network functions. To enhance the efficiency of data and bandwidth, TLP first learns a base model on a data-abundant edge node (the source), and then transfers this model (instead of data) to other data-limited nodes (the targets). To achieve a delicate balance between maintaining common features and learning target-specific features, we develop a new transfer learning technique named Similarity-based Elastic Weight Con-solidation (SEWC), and integrate it into TLP. Experiments on real-world data illustrate that, compared to the state-of-the-art methods, TLP-SEWC reduces the Mean Absolute Error (MAE) of traffic prediction by up to 57.9%.

Prediction of key system characteristics, such as the communication load, is required to overcome the delays in wireless communication systems. State-of-The-Art (SOTA) approaches mostly apply existing Neural Network (NN) structures, and extract latent features purely based on their sensitivity to the forecasting accuracy. This way of feature extraction may neglect some non-obvious yet informative dimensions in the model input, leading to inaccurate forecasting results. In this paper, we present an Adaptive Feature Boosting (AFB) approach, which integrates multiple AutoEncoders (AEs) to automatically extract robust and comprehensive latent features for communication load forecasting. The recurrent and residual connections among the AEs make sure that the extracted latent features are representative for all input dimensions. With more comprehensive information extracted from the history, the forecasting accuracy is thus improved. We evaluate AFB against existing approaches on a real-world dataset that contains Call Detail Records (CDRs) of the Milan city over a period of two months. The evaluation shows that our AFB-based approach achieves 35.2% more accurate load forecasting results than the SOTA deep approaches.

Within a cellular network, load balancing between different cells is of critical importance to network performance and quality of service. Most existing load balancing algorithms are manually designed and tuned rule-based methods where near-optimality is almost impossible to achieve. These rule-based meth-ods are difficult to adapt quickly to traffic changes in real-world environments. Given the success of Reinforcement Learning (RL) algorithms in many application domains, there have been a number of efforts to tackle load balancing for communication systems using RL-based methods. To our knowledge, none of these efforts have addressed the need for data efficiency within the RL framework, which is one of the main obstacles in applying RL to wireless network load balancing. In this paper, we formulate the communication load balancing problem as a Markov Decision Process and propose a data-efficient transfer deep reinforcement learning algorithm to address it. Experimental results show that the proposed method can significantly improve the system performance over other baselines and is more robust to environmental changes.

Due to the uneven demographic distribution and people’s daily activities, communication systems usually experience highly imbalanced load across different cells. This imbalance leads to unsatisfied users in the congested cells and under-utilized resources in the less-loaded cells. To deal with this issue, existing work migrates the load from heavily loaded cells to lightly loaded cells, by either handing over active mode User Equipment (UEs) to other serving cells, or re-selecting the camping cells for idle mode UEs. In this paper, we further advance the research on Load Balancing (LB) with a hybrid control of both active and idle UEs. This task is challenging, due to the conflicts between Active-UE LB (AULB) and Idle-UE LB (IULB) policies. To overcome this challenge, we propose a Hierarchical Policy Learning (HPL) framework, which coordinates the actions between LB policies with a two-level learning structure. In this way, HPL produces AULB and IULB policies that are better aligned with each other. Extensive simulation results illustrate the efficiency and efficacy of the proposed HPL.

Passive sensing with ambient WiFi signals is a promising technique that will enable new types of human-robot interactions while preserving users' privacy. Here, we present PresSense, a system for human respiration sensing in noisy environments. Unlike existing WiFi-based respiration sensors, we employ a human presence detector, improving the robustness in scenarios where no human is present in an Area Of Interest (AOI). We also integrate our novel feature, Peak Distance Histogram (PDH), with other classic WiFi features to achieve better accuracy when someone is present in the AOI. We tested our system using commodity WiFi devices in an office room. Our PresSense outperforms the state of the arts in both respiration rate estimation and presence detection.