This paper studies coordinated behaviors which arise when a team of robots must traverse hazardous environments in the presence of an adversary. We formulate the scenario as a novel non-cooperative stochastic game in which the "blue" team of robots moves in an environment modeled by a time-varying graph, attempting to reach some goal with minimum cost, while the "red" player controls how the graph changes to maximize the cost. In addition to a numerical method to compute the Nash equilibrium, we also present novel theoretical analysis on security strategies that provides performance bounds in a more computationally efficient way. Through numerical simulations, we demonstrate the emergence of beneficial coordinated behavior, where the robots split up and/or synchronize to traverse risky edges.

This paper studies Reinforcement Learning (RL) techniques to enable team coordination behaviors in graph environments with support actions among teammates to reduce the costs of traversing certain risky edges in a centralized manner. While classical approaches can solve this non-standard multi-agent path planning problem by converting the original Environment Graph (EG) into a Joint State Graph (JSG) to implicitly incorporate the support actions, those methods do not scale well to large graphs and teams. To address this curse of dimensionality, we propose to use RL to enable agents to learn such graph traversal and teammate supporting behaviors in a data-driven manner. Specifically, through a new formulation of the team coordination on graphs with risky edges problem into Markov Decision Processes (MDPs) with a novel state and action space, we investigate how RL can solve it in two paradigms: First, we use RL for a team of agents to learn how to coordinate and reach the goal with minimal cost on a single EG. We show that RL efficiently solves problems with up to 20/4 or 25/3 nodes/agents, using a fraction of the time needed for JSG to solve such complex problems; Second, we learn a general RL policy for any N-node EGs to produce efficient supporting behaviors. We present extensive experiments and compare our RL approaches against their classical counterparts.

This paper aims to solve the coordination of a team of robots traversing a route in the presence of adversaries with random positions. Our goal is to minimize the overall cost of the team, which is determined by (i) the accumulated risk when robots stay in adversary-impacted zones and (ii) the mission completion time. During traversal, robots can reduce their speed and act as a ‘guard’ (the slower, the better), which will decrease the risks certain adversary incurs. This leads to a trade-off between the robots’ guarding behaviors and their travel speeds. The formulated problem is highly non-convex and cannot be efficiently solved by existing algorithms. We employ reinforcement learning techniques by developing new encoding and policy-generating methods. Simulations demonstrate that our learning methods can efficiently produce team coordination behaviors. We discuss the reasoning behind these behaviors and explain why they reduce the overall team cost.

In multi-robot systems, achieving coordinated missions remains a significant challenge due to the coupled nature of coordination behaviors and the lack of global information for individual robots. To mitigate these challenges, this paper introduces a novel approach, Bi-level Coordination Learning (Bi-CL), that leverages a bi-level optimization structure within a CTDE paradigm. Our bi-level reformulation decomposes the original problem into a reinforcement learning level with reduced action space, and an imitation learning level that gains demonstrations from a global optimizer. Bi-CL further integrates an alignment penalty mechanism, aiming to minimize the discrepancy between the two levels without degrading their training efficiency. We introduce a running example to conceptualize the problem formulation. Simulation results demonstrate that Bi-CL can learn more efficiently and achieve comparable performance with traditional multi-agent reinforcement learning baselines for multi-robot coordination.

Team Coordination on Graphs with Risky Edges (TCGRE) is a recently emerged problem, in which a robot team collectively reduces graph traversal cost through support from one robot to another when the latter traverses a risky edge. Resembling the traditional Multi-Agent Path Finding (MAPF) problem, both classical and learning-based methods have been proposed to solve TCGRE, however, they lacked either computational efficiency or optimality assurance. In this paper, we reformulate TCGRE as a constrained optimization problem and perform a rigorous mathematical analysis. Our theoretical analysis shows the NP-hardness of TCGRE by reduction from the Maximum 3D Matching problem and that efficient decomposition is a key to tackle this combinatorial optimization problem. Furthermore, we design three classes of algorithms to solve TCGRE, i.e., Joint State Graph (JSG) based, coordination based, and receding-horizon sub-team based solutions. Each of these proposed algorithms enjoys different provable optimality and efficiency characteristics that are demonstrated in our extensive experiments.

This paper studies a team coordination problem in a graph environment. Specifically, we incorporate “support” action which an agent can take to reduce the cost for its teammate to traverse some high cost edges. Due to this added feature, the graph traversal is no longer a standard multi-agent path planning problem. To solve this new problem, we propose a novel formulation that poses it as a planning problem in a joint state space: the joint state graph (JSG). Since the edges of JSG implicitly incorporate the support actions taken by the agents, we are able to now optimize the joint actions by solving a standard single-agent path planning problem in JSG. One main drawback of this approach is the curse of dimensionality in both the number of agents and the size of the graph. To improve scalability in graph size, we further propose a hierarchical decomposition method to perform path planning in two levels. We provide both theoretical and empirical complexity analyses to demonstrate the efficiency of our two algorithms.

In this article, we propose an approach to construct a time-varying communication topology with a resilient consensus performance for robot swarms with limited communication ranges. The robots are deployed to explore a large task space and achieve consensus despite the existence of a finite number of noncooperative members in the team. Existing methods encouraged robots to stay close to each other to achieve certain robustness requirements on the connectivity of the communication topology. We leverage on the robots’ mobility to design a time-varying integrated topology composed of several subgroups of robots deployed on nonoverlapping closed-loop paths. Robots are spread out and move along the paths, forming periodic communication links within or across groups. We analyze the time-varying topology synthesized and provide sufficient conditions for individual subgroups and the interconnection between them. We show designs satisfying the conditions with simulated examples in a lattice space, as well as in a task space with predefined paths.

We formulate a scenario in which an autonomous defender is tasked with intercepting an intruder that tries to reach a circular target region. This is a variant of the target defense problem proposed by Isaacs as a pursuit-evasion game. Unlike the original target guarding problem and its various extensions, we consider the effect of partial information by imposing sensing limitation to the robots. We analyze the game by decomposing it into three game phases: deployment, asymmetric information, and engagement phase. Focusing on a particular parameter regime, we propose a simple defender strategy together with the lower bound on the probability that it wins the game. The defender strategy in each phase is constructed so that the subsequent phase starts in a desired initial configuration. The proposed problem is rich in terms of the parameter regimes that it contains, and thus is expected to be a useful platform in exploring effective control policies.

We present an algorithm for the concurrent assignment and planning of collision-free trajectories (DC-CAPT) for robots whose kinematics can be modeled as Dubins cars, i.e., robots constrained in terms of their initial orientation and their minimum turning radius. Coupling the assignment and trajectory planning subproblems allows for a computationally tractable solution. This solution is guaranteed to be collision- free through the use of a single constraint: the start and goal locations have separation distance greater than some threshold. We derive this separation distance by extending a prior work that assumed holonomic robots. We demonstrate the validity of our approach, and show its efficacy through simulations and experiments where groups of robots executing Dubins curves travel to their assigned goal locations without collisions.

Multi-Robot Systems have been recently employed in different applications and have advantages over single-robot systems, such as increased robustness and task performance efficiency. We consider such assemblies specifically in the scenario of perimeter defense, where the task is to defend a circular perimeter by intercepting radially approaching targets. Possible intruders appear randomly at a fixed distance from the perimeter and with azimuthal location determined by some unknown probability density. Coordination among multiple defenders is a complex combinatorial optimization problem. In this work, we focus on the following two aspects: (i) estimating the probability density that describes the direction from which the next intruders are going to arrive, and (ii) partitioning of the space so that the defenders focus on capturing a disjoint subset of intruders. Results show that the proposed strategy increases the number of captures over a naive baseline strategy, especially in scenarios with non-uniform spatial distributions of intruder arrival. The proposed approach is also efficient and able to quickly adapt to time-varying intruder distributions.

We study strategies to deploy defenders/sensors to detect intruders that approach a targeted region. This scenario is formulated as a barrier coverage, which aims to minimize the number of unseen paths. The problem becomes challenging when the number of defenders is insufficient for a full coverage, requiring us to find the most effective location to deploy them. To this end, we use ideas from game theory to account for various paths that the intruders may take. Specifically, we propose an iterative algorithm to refine the set of candidate defender formations, which uses the payoff matrix to directly evaluate the utility of different formations. Given the set of candidate formations, a mixed Nash equilibrium gives a stochastic policy to deploy the defenders. The efficacy of the proposed strategy is demonstrated by a numerical analysis that compares our method with an existing graph-theoretic method.

Effective communication is required for teams of robots to solve sophisticated collaborative tasks. In practice it is typical for both the encoding and semantics of communication to be manually defined by an expert; this is true regardless of whether the behaviors themselves are bespoke, optimization based, or learned. We present an agent architecture and training methodology using neural networks to learn task-oriented communication semantics based on the example of a communication-unaware expert policy. A perimeter defense game illustrates the system's ability to handle dynamically changing numbers of agents and its graceful degradation in performance as communication constraints are tightened or the expert's observability assumptions are broken.