Opponent modelling (OM) can enable agents to reason behaviors of others, and hence act accordingly and interact effectively in multi-agent reinforcement learning (MARL). Existing OM approaches commonly assume the availability of opponents' trajectory, which needs incessant message exchange during execution in the partially observable environment. As a result, they are not cost-effective due to communication overhead, and are also inapplicable to many practical tasks with constraints on communication. To handle this problem, we propose attentional opponent modelling (ATOM) to infer the beliefs of neighboring entities with only local observation. ATOM does not require communication during execution and can be easily combined with existing MARL methods. As examples, we propose ATOM Actor-Critic and ATOM Q-learning architectures to facilitate multi-agent cooperation. Extensive experiments on several challenging tasks (i.e., Cooperative Navigation, Predator-Prey and Starcraft Multi-Agent Challenge) show the superior performance of our methods compared with benchmark OM approaches.

Unmanned aerial vehicles (UAVs) are capable of enhancing the coverage of existing cellular networks by acting as aerial base stations (ABSs). Due to the limited on-board battery capacity and dynamic topology of UAV networks, trajectory planning and interference coordination are crucial for providing satisfactory service, especially in emergency scenarios, where it is unrealistic to control all UAVs in a centralized manner by gathering global user information. Hence, we solve the decentralized joint trajectory and transmit power control problem of multi-UAV ABS networks. Our goal is to maximize the number of satisfied users, while minimizing the overall energy consumption of UAVs. To allow each UAV to adjust its position and transmit power solely based on local- rather the global-observations, a multi-agent reinforcement learning (MARL) framework is conceived. In order to overcome the non-stationarity issue of MARL and to endow the UAVs with distributed decision making capability, we resort to the centralized training in conjunction with decentralized execution paradigm. By judiciously designing the reward, we propose a decentralized joint trajectory and power control (DTPC) algorithm with significantly reduced complexity. Our simulation results show that the proposed DTPC algorithm outperforms the state-of-the-art deep reinforcement learning based methods, despite its low complexity.

Proactive caching at base stations (BSs) has been shown promising in offloading traffic, where most priori works consider full frequency reuse among cells and assume known file popularity. To facilitate proactive caching, recent works either adopt linear models or shallow neural networks to predict popularity, without explaining the rationale. In this paper, we consider cache-enabled multi-tier heterogenous networks. To show if full frequency reuse is superior, we consider underlay and overlay modes with random bandwidth allocation for interference management. After optimizing the caching policy and bandwidth allocation to maximize successful offloading probability, we show that the overlay mode outperforms the underlay mode. To confirm that complex non-linear prediction models are unnecessary for proactive caching when using the historical numbers of requests, we employ several linear and non-linear models to predict content popularity and user density, using MovieLens and Youku datasets. To interpret why the caching performance achieved by the optimized policies with the information predicted by the linear models is close to that using non-linear models, we prove that deterministic popularity with typical profiles can be predicted with linear models. Then, we show that most popular files are with these profiles in the real dataset.

Prior works in a designing caching policy do not distinguish content popularity with user preference. In this paper, we illustrate the caching gain by exploiting individual user behavior in sending requests. After showing the connection between the two concepts, we provide a model for synthesizing user preference from content popularity. We then optimize the caching policy with the knowledge of user preference and activity level to maximize the offloading probability for cache-enabled device-to-device communications, and develop a low-complexity algorithm to find the solution. In order to learn user preference, we model the user request behavior resorting to probabilistic latent semantic analysis, and learn the model parameters by the expectation maximization algorithm. By analyzing a Movielens data set, we find that the user preferences are less similar, and the activity level and topic preference of each user change slowly over time. Based on this observation, we introduce a prior knowledge-based learning algorithm for user preference, which can shorten the learning time. Simulation results show a remarkable performance gain of the caching policy with user preference over existing policy with content popularity, both with realistic data set and synthetic data validated by the real data set.

Caching at wireless edge is a promising way to support the explosive growth of traffic load. In this paper, we jointly optimize the caching and bandwidth allocation policy in a cache-enabled heterogeneous network, where macro base stations (BSs) each with high capacity backhaul but not cache and helpers each with cache are random located. Probabilistic caching and random bandwidth allocation are considered. We derive the closed-form expression of successful offloading probability, which is defined as the probability that a user is associated with helpers with the date rate greater than a threshold. The joint optimal problem is not concave in general case. Nonetheless, when the density of helpers is large, the joint global optimum can be obtained, where the optimal bandwidth allocation needs onedimensional search and the optimal caching policy is with closedform expression. Numerical results show that the offloading gain can be significantly improved by the joint optimization.

Cache-enabled device-to-device (D2D) communications can boost network throughput. By pre-downloading contents to local caches of users, the content requested by a user can be transmitted via D2D links by other users in proximity. Prior works optimize the caching policy at users with the knowledge of content popularity, defined as the probability distribution that each file in a library is requested by all users. However, content popularity can not reflect the interest of each individual user and thus existing caching policy based on popularity may not fully capture the performance gain introduced by caching. In this paper, we optimize caching policy for cache-enabled D2D by learning user preference, which is defined as the conditional probability distribution of a user's request given that the user sends a request. We first formulate an optimization problem with given user preference to maximize the offloading probability, which is proved as NP-hard, and then provide a greedy algorithm to find the solution. In order to predict the preference of each individual user, we model the user request behavior by probabilistic latent semantic analysis (pLSA), and then apply expectation maximization (EM) algorithm to estimate the model parameters. Simulation results show that using pLSA can learn user preference quickly. Compared to existing caching policy exploiting content popularity, the offloading gain achieved by the proposed policy can be remarkably improved even with predicted user preference.

By caching files at users, content delivery traffic can be offloaded via device-to-device (D2D) links if a helper user is willing to transmit the cached file to the user who requests the file. In practice, the user device has limited battery capacity, and may terminate the D2D connection when its battery has little energy left. Thus, taking the battery consumption allowed by the helper users to support D2D into account introduces a reduction in the possible amount of offloading. In this paper, we investigate the relationship between offloading gain of the system and energy cost of each helper user. To this end, we introduce a user-centric protocol to control the energy cost for a helper user to transmit the file. Then, we optimize the proactive caching policy to maximize the offloading opportunity and the transmit power at each helper to maximize the offloading probability. Finally, we evaluate the overall amount of traffic offloaded to D2D links and the average energy consumption at each helper, with the optimized caching policy and transmit power. Simulations show that a significant amount of traffic can be offloaded even when the energy cost is kept low.

Device-to-Device (D2D) communications can increase the throughput of cellular networks significantly, where the interference among D2D links should be properly managed. In this paper, we propose an opportunistic cooperative D2D transmission strategy by exploiting the caching capability at the users to deal with the interference among D2D links. To increase the cooperative opportunity and improve spatial reuse gain, we divide the D2D users into clusters and cache different popular files at the users within a cluster, and then find the optimal cluster size. To maximize the network throughput, we assign different frequency bands to cooperative and non- cooperative D2D links and optimize the bandwidth partition. Simulation results demonstrate that the proposed strategy can provide 500%~600% throughput gain over existing cache-enabled D2D communications when the popularity distribution is skewed, and can provide 40%~80% gain even when the popularity distribution is uniform.

Pushing files to users based on predicting the personal interest of each user may provide higher throughput gain than broadcasting popular files to users based on their common interests. However, the energy consumed at base station for pushing files individually to each user is also higher than broadcast. In this paper, we propose an energy-saving transmission strategy for pre- downloading the files to each user by exploiting the excess resources in the network during off-peak time. Specifically, a power allocation and scheduling algorithm is designed aimed to minimize the extra energy consumed for pushing, where network and user level context information are exploited. Simulation results show that when the energy of both content placement and content delivery is taken into account, the proposed unicast strategy consumes less energy and achieves higher throughput than broadcasting when the files popularity is not uniform and the personal interest prediction is with less uncertainty.

Device-to-Device (D2D) communications can offload the traffic and boost the throughput of cellular networks. By caching files at users, content delivery traffic can be offloaded via D2D links, if a helper user are willing to send the cached file to the user who requests the file. Yet it is unclear how much energy needs to be consumed at a helper user to support the traffic offloading. In this paper, we strive to find the minimal energy consumption required at a helper user to maximize the amount of offloaded traffic. To this end, we introduce a user-centric proactive caching policy that can control the energy cost for a helper user to convey a file, and then optimize the caching policy to maximize the offloaded traffic. To reduce the energy during transmission, we optimize the transmit power to minimize the energy consumed by a helper to send a file. We analyze the relationship between traffic offloading and energy cost with the optimized caching policy and transmit power by numerical and simulation results, which demonstrate that a significant amount of traffic can be offloaded with affordable energy costs.

The throughput of small cell networks (SCNs) is limited by intercell interference. Except numerous techniques of interference coordination, simply caching several popular files at each user provides an alternative approach to reduce interference by pre-offloading. This becomes possible nowadays due to the facts that the cost of memory devices is reducing quickly and content delivery is gradually dominating the traffic load. In this paper, we quantify the performance gain introduced by precaching at users. We derive the average throughput for non-coordinated downlink SCN with precaching, and compare with the SCN without caching and with existing prefetching policy. We use simulation to validate the analytical results and further evaluate the average download time of different policies. Our results show that the caching gain is remarkable when the traffic load is high and the popularity distribution is far from uniform.