The role-oriented learning approach could improve the performance of multi-agent reinforcement learning by decomposing complex multi-agent tasks into different roles. However, due to the dynamic environment and interactions among agents, the role undertaken by an agent changes rapidly with time going on. Therefore, the roles of agents should be adapted to the varying situation during the learning process. In this paper, we propose a role-oriented graph convolution based multi-agent reinforcement learning framework (ROGC). Firstly, we design a role assigner based on samples generated from the environment to learn roles for classifying agents into different groups. To further enhance cooperation among agents in the same group for higher performance, we design a graph convolutional module to achieve intra-role communications based on discovered roles. With roles and extracted role features, we design a role-oriented policy learning module that embeds the role information into the algorithm and generates effective policies for individuals. Further, we introduce an auto-encoder to learn the intra-role cooperation knowledge in the graph convolutional module, which ensures our framework executes in a decentralized way. Extensive experiments show that our framework can learn dynamic roles and make full use of learned roles, which makes it outperform popular MARL methods.

The distributed system Ray has attracted much attention for many decision-making applications. It provides a flexible and powerful distributed running mechanism for the training of the learning algorithms, which could map the computation tasks to the resources automatically. Task scheduling is a critical component in Ray, adopting a two-layer structure. It uses a simple general scheduling principle, which leaves much space to optimize. In this paper, we will study the two-layer scheduling problem in Ray, setting it as an optimization problem. We firstly present a comprehensive formulation for the problem and point out that it is a NP-hard problem. Then we design a hierarchical reinforcement learning method, named HierRL, which consists of a high-level agent and a low-level agent. Sophisticated state space, action space, and reward function are designed for this method. In the high level, we devise a value-based reinforcement learning method, which allocates a task to an appropriate node of the low level. With tasks allocated from the high level and generated from applications, a low-level reinforcement learning method is constructed to select tasks from the queue to be executed. A hierarchical policy learning method is introduced for the training of the two-layer agents. Finally, we simulate the two-layer scheduling procedure in a public platform, Cloudsim, with tasks from a real Dataset generated by the Alibaba Cluster Trace Program. The results show that the proposed method performs much better than the original scheduling method of Ray.

Heterogeneous behaviours are widespread in many multi-agent tasks, which have not been paid much attention in the community of multi-agent reinforcement learning. It would be a key factor for improving the learning performance to efficiently characterize and automatically find heterogeneous behaviours. In this paper, we introduce the concept of the skill to explore the ability of heterogeneous behaviours. We propose a novel skill-based multi-agent reinforcement learning framework to enable agents to master diverse skills. Specifically, our framework consists of the skill representation mechanism, the skill selector and the skill-based policy learning mechanism. We design an auto-encoder model to generate the latent variable as the skill representation by incorporating the environment information, which ensures the distinguishable of agents for skill selection and the discriminability for the skill learning. With the representation, a skill selection mechanism is invented to realize the assignment from agents to skills. Meanwhile, diverse skill-based policies are generated through a novel skill-based policy learning method. To promote efficient skill discovery, a mutual information based intrinsic reward function is constructed. Empirical results show that our framework obtains the best performance on three challenging benchmarks, i.e., StarCraft II micromanagement tasks, Google Research Football and GoBigger, over state-of-the-art MARL methods.

The distributed system Ray has attracted much attention in decision-making applications, which could greatly accelerate the training efficiency for intelligent algorithms. Task scheduling is one of the critical technologies in Ray, in which the number of resource nodes could be auto-scaling, i.e., automatically increasing or decreasing according to the workload. The adopted scheduling strategy is simple, which leaves much space to be optimized. In this paper, we consider designing a reinforcement learning method to optimize the scheduling problem in Ray. We propose an attentive reinforcement learning method, designing an attention-based state encoder that could efficiently extract the system state in the situation of the varying number of resource nodes. At the same time, an action mask mechanism filters invalid actions. Further, to improve the learning efficiency in the environment with the varied number of nodes, we design a curriculum learning method, which trains the method by gradually increasing the number of nodes in the scheduling process. Finally, we use the real data generated by the Alibaba Cluster Trace Program to test in the simulation platform CloudSim. The experimental results show that the proposed method effectively scales down the completion time of tasks compared to the original algorithm in Ray.

The problem of scheduling transmission in single hop and multi-hop wireless networks with arbitrary topology under the physical interference model has been extensively studied. The focus has been on optimizing the efficiency of transmission parallelization through a minimum-frame-length schedule that meets a given set of traffic demands using the smallest number of time slots, each of which is associated with a set of compatible (according to the interference model) transmissions. This approach maximizes the resource reuse efficiency, but in general does not correspond to the best performance in terms of end-to-end packet delivery delay for multiple source-destination pairs, due to the inherent restriction of frame periodicity. In this paper, we study the problem of scheduling to minimize the end-to-end delay in wireless networks under the Signal to Interference plus Noise Ratio (SINR) constraints, and propose two schemes. The first scheme extends the minimum-frame-length approach with a phase of time slot ordering to account for the delay metric. The second scheme directly optimizes delay without the constraint of periodic framing. We propose novel mixed integer programming models for the two schemes and study their properties and complexity. Moreover, we present an efficient heuristic method that provides good quality solutions time-efficiently.

With the densification of nodes in cellular networks, free space optic (FSO) connections are becoming an appealing low cost and high rate alternative to copper and fiber backhaul solutions for wireless communication systems. To ensure a reliable cellular backhaul, provisions for redundant disjoint paths between the nodes must be made in the design phase. This paper aims at finding a cost-effective solution to upgrade the cellular backhaul with pre-deployed optical fibers using FSO links and mirror components. Since the quality of the FSO links depends on several factors, such as transmission distance, power, and weather conditions, we adopt an elaborate formulation to calculate link reliability. We present a novel integer linear programming model to approach optimal FSO backhaul design, guaranteeing $K$-disjoint paths connecting each node pair. Next, we derive a column generation method to a path-oriented mathematical formulation. Applying the method in a sequential manner enables high computational scalability. We use realistic scenarios to demonstrate that our approaches efficiently provide optimal or near-optimal solutions, and thereby allow for accurately dealing with the trade-off between cost and reliability.

We consider a combinatorial optimization problem in passive In-Building Distributed Antenna Systems (IB-DAS) deployment for indoor mobile broadband service. These systems have a tree topology, in which a central base station is connected to a number of antennas located at tree leaves via cables represented by the tree edges. Each inner node corresponds to a power equipment, of which the available types differ in the number of output ports and/or by power gain at the ports. This paper focuses on the equipment selection problem that amounts to, for a given passive DAS tree topology, selecting a power equipment type for each inner node and assigning the outgoing edges of the node to the equipment ports. The performance metric is the power deviation at the antennas from the target values. We consider as objective function the minimization of either the total or the largest power deviation over all antennas. Our contributions are the development of exact pseudo-polynomial time algorithms and (additive) fully-polynomial time approximation schemes for both objectives. Numerical results are provided to illustrate the algorithms. We also extend some results to account for equipment cost.

For some time, directional antennas have been considered to solve connectivity and interference issues in wireless networks. Several scenarios have been presented and often the conclusions drawn are positive, showing increase in capacity. However, to date there has been no effort to assess a holistic picture of the benefit/cost tradeoff and previous work mainly concerns either link scheduling or antenna placement but not the two combined. Such consideration will become increasingly important in the near future with the advent of heterogeneous networks and other possible combinations of mesh and public access networks.

Network I/O workloads are dominating in most of the Cloud datacenters today. One way to improve inter-VM communication efficiency is to support co-resident VM communication using a faster communication protocol than the traditional TCP/IP commonly used regardless whether VMs are located on the same

Sparse LU decomposition is the core computation in the direct method that solves sparse systems of linear equations. Only little work has been conducted on parallelizing it on FPGAs. In this paper, we study parallelization strategies for sparse LU decomposition on FPGAs. We first analyze how to parallelize the right-looking algorithm and find that this algorithm is not suitable for FPGAs. Then the left-looking algorithm is analyzed and considered as better candidate than the right-looking version. Our design derived from the left-looking algorithm is based on a simple yet efficient parallel computational model for FPGAs. Our design mainly consists of multiple parallel processing elements (PEs). A total of 14 PEs can be integrated into a Xilinx Virtex-5 XC5VLX330. Unlike related work, where their designs are applied to sparse matrices from particular application domains, our hardware design can be applied to any symmetric positive definite or diagonally dominant matrices.

BGP prefix hijacking is one of the main threatens for the Internet. It is important to identify the impact factors for prefix hijacking. This paper studies the problem from the view of AS logical topology by analysis of the data from the snapshots of CAIDA. We propose a hierarchical model based on AS relationship to classify the AS nodes into different level and define core size of each node to prioritize them in each level. Two metrics named infected number and infected diameter are introduced to analyze the relationship between the logical structural characters of AS node and the impact of prefix hijacking. The results show that core size, which reflects the relation of an AS node with Tier-1 AS nodes, and AS level are two main important factors. AS nodes in higher level or with bigger core size are able to infect more nodes. However, AS node in lower level has longer infected diameters. This phenomenon indicates that the prefix hijacking with attacker in lower level is harder to detect.