Robotic fleets such as unmanned aerial and ground vehicles have been widely used for routine inspections of static environments, where the areas of interest are known and planned in advance. However, in many applications, such areas of interest are unknown and should be identified online during exploration. Thus, this paper considers the problem of simultaneous exploration, inspection of unknown environments and then real-time communication to a mobile ground control station to report the findings. The heterogeneous robots are equipped with different sensors, e.g., long-range lidars for fast exploration and close-range cameras for detailed inspection. Furthermore, global communication is often unavailable in such environments, where the robots can only communicate with each other via ad-hoc wireless networks when they are in close proximity and free of obstruction. This work proposes a novel planning and coordination framework (SLEI3D) that integrates the online strategies for collaborative 3D exploration, adaptive inspection and timely communication (via the intermittent or proactive protocols). To account for uncertainties w.r.t. the number and location of features, a multi-layer and multi-rate planning mechanism is developed for inter-and-intra robot subgroups, to actively meet and coordinate their local plans. The proposed framework is validated extensively via high-fidelity simulations of numerous large-scale missions with up to 48 robots and 384 thousand cubic meters. Hardware experiments of 7 robots are also conducted. Project website is available at https://junfengchen-robotics.github.io/SLEI3D/.

Robotic fleets can be extremely efficient when working concurrently and collaboratively, e.g., for delivery, surveillance, search and rescue. However, it can be demanding or even impractical for an operator to directly control each robot. Thus, autonomy of the fleet and its online interaction with the operator are both essential, particularly in dynamic and partially unknown environments. The operator might need to add new tasks, cancel some tasks, change priorities and modify planning results. How to design the procedure for these interactions and efficient algorithms to fulfill these needs have been mostly neglected in the related literature. Thus, this work proposes a human-centric coordination and supervision scheme (HECTOR) for large-scale robotic fleets under continual and uncertain temporal tasks. It consists of three hierarchical layers: (I) the bidirectional and multimodal protocol of online human-fleet interaction, where the operator interacts with and supervises the whole fleet; (II) the rolling assignment of currently-known tasks to teams within a certain horizon, and (III) the dynamic coordination within a team given the detected subtasks during online execution. The overall mission can be as general as temporal logic formulas over collaborative actions. Such hierarchical structure allows human interaction and supervision at different granularities and triggering conditions, to both improve computational efficiency and reduce human effort. Extensive human-in-the-loop simulations are performed over heterogeneous fleets under various temporal tasks and environmental uncertainties.

Many robots are not equipped with a manipulator and many objects are not suitable for prehensile manipulation (such as large boxes and cylinders). In these cases, pushing is a simple yet effective non-prehensile skill for robots to interact with and further change the environment. Existing work often assumes a set of predefined pushing modes and fixed-shape objects. This work tackles the general problem of controlling a robotic fleet to push collaboratively numerous arbitrary objects to respective destinations, within complex environments of cluttered and movable obstacles. It incorporates several characteristic challenges for multi-robot systems such as online task coordination under large uncertainties of cost and duration, and for contact-rich tasks such as hybrid switching among different contact modes, and under-actuation due to constrained contact forces. The proposed method is based on combinatorial hybrid optimization over dynamic task assignments and hybrid execution via sequences of pushing modes and associated forces. It consists of three main components: (I) the decomposition, ordering and rolling assignment of pushing subtasks to robot subgroups; (II) the keyframe guided hybrid search to optimize the sequence of parameterized pushing modes for each subtask; (III) the hybrid control to execute these modes and transit among them. Last but not least, a diffusion-based accelerator is adopted to predict the keyframes and pushing modes that should be prioritized during hybrid search; and further improve planning efficiency. The framework is complete under mild assumptions. Its efficiency and effectiveness under different numbers of robots and general -shaped objects are validated extensively in simulations and hardware experiments, as well as generalizations to heterogeneous robots, planar assembly and 6D pushing.

Multi-agent systems can be extremely efficient when working concurrently and collaboratively, e.g., for delivery, surveillance, search and rescue. Coordination of such teams often involves two aspects: (i) selecting appropriate subteams for different tasks in various areas; (ii) coordinating agents in the subteams to execute the associated subtasks. Existing work often assumes that the tasks are static and known beforehand, where an integer program can be formulated and solved offline. However, in many applications, the team-wise tasks are generated online continually by external requests; and the amount of subtasks within each task is uncertain (e.g., the number of packages to deliver, and victims to rescue). The aforementioned offline solution becomes inadequate as it would require constant re-computation for the whole team and global communication to broadcast the results. Thus, this work tackles the large-scale coordination problem under continual and uncertain temporal tasks, specified as temporal logic formulas over collaborative actions. The proposed hierarchical framework (HULK) consists of two interleaved layers: the rolling assignment of currently-known tasks to subteams within a certain horizon, and the dynamic coordination within a sub-team given the detected subtasks during online execution. Thus, the coordination is performed hierarchically at different granularities and triggering conditions, to improve the computational efficiency and robustness. It is validated rigorously over large-scale heterogeneous systems under various temporal tasks and environment uncertainties.

Automated synthesis of local controllers for multi-agent systems to satisfy complex task specifications has attracted extensive attention. However, it remains unclear how to formally guarantee that their composed behavior satisfies the specified global task. In this paper, we aim to synthesize an automated and distributed control strategy for fleet-wise tasks specified as linear temporal logic formulas, such that the agents act asynchronously and synchronize only on shared actions via local coordination. The proposed method consists of three main steps. First, the set of satisfying global control strategies is computed via parallel composition. Among these strategies the conditions for decomposability are evaluated, based on which the global strategy over a maximum synchronization scheme is found. Then the synchronization scheme is further refined to obtain more efficient local control strategies. It is formally proven that the resulting global behavior along with the synchronization scheme satisfies the specified global task. Also, the synthesized local controllers are reactive to changes in the workspace or in the fleet during online execution, without the need for replanning. Thus, collaborative relations among the agents are adaptive as needed. Numerical simulations and hardware experiments are conducted for nontrivial scenarios.Note to Practitioners—This paper was motivated by the problem of coordinating a fleet of autonomous robots in collaborative search and delivery processes, where local controllers are synthesized for each robot such that the specified global task specification is satisfied. Existing approaches to address such problems often rely on a fully-connected communication topology and fixed collaborative relations among the robots, thus limiting efficiency of the multi-robot execution and yielding difficulty of coordination. Also, most of existing approaches cannot directly deal with the changes in the workspace or in the fleet during task execution unless a task replanning is performed, which leads to longer time of system-wide communication and task completion, yielding failures during online execution. In this paper, we propose a novel distributed control architecture to tackle these issues, where the robots are controlled to operate asynchronously and achieve online local coordination by synchronization on collaborative actions. Moreover, compared with the common solutions, the collaborative relations among the robots are formed and removed dynamically as needed and the robots are robust to uncertainty during task execution. It is formally proven that the resulted global behavior of the robot fleet along with the synchronization scheme is consistent with the specified global task. We have shown that it is particularly useful for complex and coupled multi-robot applications, where the inter-robot collaborations are feasible and local. Experimental results suggest that this approach is applicable to multi-robot systems which greatly improves the concurrency and efficiency of task execution. In the future research, we will draw inspiration from decentralized approaches for task decomposition to alleviate the computational burden as the number of robots increases.

Fleets of unmanned robots can be beneficial for the long-term monitoring of large areas, e.g., to monitor wild flocks, detect intruders, search and rescue. Monitoring numerous dynamic targets in a collaborative and efficient way is a challenging problem that requires online coordination and information fusion. The majority of existing works either assume a passive all-to-all observation model to minimize the summed uncertainties over all targets by all robots, or optimize over the jointed discrete actions while neglecting the dynamic constraints of the robots and unknown behaviors of the targets. This work proposes an online task and motion coordination algorithm that ensures an explicitly-bounded estimation uncertainty for the target states, while minimizing the average number of active robots. The robots have a limited-range perception to actively track a limited number of targets simultaneously, of which their future control decisions are all unknown. It includes: (i) the assignment of monitoring tasks, modeled as a flexible size multiple vehicle routing problem with time windows (m-MVRPTW), given the predicted target trajectories with uncertainty measure in the road-networks; (ii) the nonlinear model predictive control (NMPC) for optimizing the robot trajectories under uncertainty and safety constraints. It is shown that the robots can switch between active and inactive roles dynamically online as required by the unknown monitoring task. The proposed methods are validated via large-scale simulations of up to 100 robots and targets.

Learning from demonstration (LfD) provides a fast, intuitive and efficient framework to program robot skills, which has gained growing interest both in research and industrial applications. Most complex manipulation tasks are long-term and involve a set of skill primitives. Thus it is crucial to have a reliable coordination scheme that selects the correct sequence of skill primitive and the correct parameters for each skill, under various scenarios. Instead of relying on a precise simulator, this work proposes a human-in-the-loop coordination framework for LfD skills that: builds parameterized skill models from kinesthetic demonstrations; constructs a geometric task network (GTN) on-the-fly from human instructions; learns a hierarchical control policy incrementally during execution. This framework can reduce significantly the manual design efforts, while improving the adaptability to new scenes. We show on a 7-DoF robotic manipulator that the proposed approach can teach complex industrial tasks such as bin sorting and assembly in less than 30 minutes.

Complex manipulation tasks can contain various execution branches of primitive skills in sequence or in parallel under different scenarios. Manual specifications of such branching conditions and associated skill parameters are not only error-prone due to corner cases, but also quickly untraceable given a large number of objects and skills. On the other hand, learning from demonstration has increasingly shown to be an intuitive and effective way to program such skills for industrial robots. Parameterized skill representations allow generalization over new scenarios, which however makes the planning process much slower thus unsuitable for online applications. In this article, we propose a hierarchical and compositional planning framework that learns a geometric task network (GTN) from exhaustive planners, without any manual inputs. A GTN is a goal-dependent task graph that encapsulates both the transition relations among skill representations and the geometric constraints underlying these transitions. This framework has shown to improve dramatically the offline learning efficiency, the online performance, and the transparency of decision process, by leveraging the task-parameterized models. We demonstrate the approach on a 7-DoF robot arm both in simulation and on hardware solving various manipulation tasks.

Learning from Demonstration (LfD) provides an intuitive and fast approach to program robotic manipulators. Task parameterized representations allow easy adaptation to new scenes and online observations. However, this approach has been limited to pose-only demonstrations and thus only skills with spatial and temporal features. In this work, we extend the LfD framework to address forceful manipulation skills, which are of great importance for industrial processes such as assembly. For such skills, multi-modal demonstrations including robot end-effector poses, force and torque readings, and operation scene are essential. Our objective is to reproduce such skills reliably according to the demonstrated pose and force profiles within different scenes. The proposed method combines our previous work on task-parameterized optimization and attractor-based impedance control. The learned skill model consists of (i) the attractor model that unifies the pose and force features, and (ii) the stiffness model that optimizes the stiffness for different stages of the skill. Furthermore, an online execution algorithm is proposed to adapt the skill execution to real-time observations of robot poses, measured forces, and changed scenes. We validate this method rigorously on a 7-DoF robot arm over several steps of an E-bike motor assembly process, which require different types of forceful interaction such as insertion, sliding and twisting.

Enabling robots to quickly learn manipulation skills is an important, yet challenging problem. Such manipulation skills should be flexible, e.g., be able adapt to the current workspace configuration. Furthermore, to accomplish complex manipulation tasks, robots should be able to sequence several skills and adapt them to changing situations. In this work, we propose a rapid robot skill-sequencing algorithm, where the skills are encoded by object-centric hidden semi-Markov models. The learned skill models can encode multimodal (temporal and spatial) trajectory distributions. This approach significantly reduces manual modeling efforts, while ensuring a high degree of flexibility and re-usability of learned skills. Given a task goal and a set of generic skills, our framework computes smooth transitions between skill instances. To compute the corresponding optimal end-effector trajectory in task space we rely on Riemannian optimal controller. We demonstrate this approach on a 7 DoF robot arm for industrial assembly tasks.

A wide range of discrete planning problems can be solved optimally using graph search algorithms. However, optimal search quickly becomes infeasible with increased complexity of a problem. In such a case, heuristics that guide the planning process towards the goal state can increase performance considerably. Unfortunately, heuristics are often unavailable or need manual and time-consuming engineering. Building upon recent results on applying deep learning to learn generalized reactive policies, we propose to learn heuristics by imitation learning. After learning heuristics based on optimal examples, they are used to guide a classical search algorithm to solve unseen tasks. However, directly applying learned heuristics in search algorithms such as A∗ breaks optimality guarantees, since learned heuristics are not necessarily admissible. Therefore, we (i) propose a novel method that utilizes learned heuristics to guide Focal Search A∗, a variant of A∗ with guarantees on bounded suboptimality; (ii) compare the complexity and performance of jointly learning individual policies for multiple robots with an approach that learns one policy for all robots; (iii) thoroughly examine how learned policies generalize to previously unseen environments and demonstrate considerably improved performance in a simulated complex dynamic coverage problem.

This paper considers the motion control and task planning problem of mobile robots under complex high-level tasks and human initiatives. The assigned task is specified as Linear Temporal Logic (LTL) formulas that consist of hard and soft constraints. The human initiative influences the robot autonomy in two explicit ways: with additive terms in the continuous controller and with contingent task assignments. We propose an online coordination scheme that encapsulates (i) a mixed-initiative continuous controller that ensures all-time safety despite of possible human errors, (ii) a plan adaptation scheme that accommodates new features discovered in the workspace and short-term tasks assigned by the operator during run time, and (iii) an iterative inverse reinforcement learning (IRL) algorithm that allows the robot to asymptotically learn the human preference on the parameters during the plan synthesis. The results are demonstrated by both realistic human-in-the-loop simulations and experiments.

We consider a team of heterogeneous robots, which are deployed within a common workspace to gather different types of data. The robots have different roles due to different capabilities: some gather data from the workspace (source robots) and others receive data from source robots and upload them to a data center (relay robots). The data-gathering tasks are specified locally to each source robot as high-level linear temporal logic formulas, which capture the different types of data that need to be gathered at different regions of interest. All robots have a limited buffer to store the data. Thus, the data gathered by source robots should be transferred to relay robots before their buffers overflow, respecting at the same time a limited communication range for all robots. The main contribution of this work is a distributed motion coordination and intermittent communication scheme that guarantees the satisfaction of all local tasks, while obeying the above constraints. The robot motion and interrobot communication are closely coupled and coordinated during runtime by scheduling intermittent meeting events to facilitate the local plan execution. We present both numerical simulations and experimental studies to demonstrate the advantages of the proposed method over existing approaches that predominantly require all-time network connectivity.

We consider a team of multiple dynamical and heterogeneous robots which are deployed for gathering different types of data within a common workspace. The robots have different roles due to different capabilities: some gather data from the workspace (Type-A robots) and others receive data from Type-A robots and upload them to a data center (Type-B robots). The data-gathering tasks are specified locally to each Type-A robot as high-level Linear Temporal Logic (LTL) formulas. All robots have a limited buffer to store the data. Thus the data gathered by Type-A robots should be transferred to Type-B robots before the buffers overflow, respecting at the same time limited communication range for all robots. The main contribution of this work is a distributed task coordination and intermittent meeting scheme that guarantees the satisfaction of all local tasks while obeying the above constraints. We present numerical simulations to demonstrate the advantages of the proposed method over most existing approaches that require all-time network connectivity.

We consider a multiagent system that consists of heterogeneous groups of homogeneous agents. Instead of defining a global task for the whole team, each agent is assigned a local task as syntactically cosafe linear temporal logic formulas that specify both motion and action requirements. Interagent dependence is introduced by collaborative actions, of which the execution requires multiple agents' collaboration. To ensure the satisfaction of all local tasks without central coordination, we propose a bottom-up motion and task coordination strategy that contains an off-line initial plan synthesis and an online coordination scheme based on real-time exchange of request and reply messages. It facilitates not only the collaboration among heterogeneous agents but also the task swapping between homogeneous agents to reduce the total execution cost. It is distributed as any decision is made locally by each agent based on local computation and communication within neighboring agents. It is scalable and resilient to agent failures as the dependence is formed and removed dynamically based on agent capabilities and their plan execution status, instead of preassigned agent identities. The overall scheme is demonstrated by a simulated scenario of 20 agents with loosely coupled local tasks.

We propose a distributed and cooperative motion and task control scheme for a team of mobile robots that are subject to dynamic constraints including inter-robot collision avoidance and connectivity maintenance of the communication network. Moreover, each agent has a local high-level task given as a Linear Temporal Logic (LTL) formula of desired motion and actions. Embedded graph grammars (EGGs) are used as the main tool to specify local interaction rules and switching control modes among the robots, which is then combined with the model-checking-based task planning module. It is ensured that all local tasks are satisfied while the dynamic constraints are obeyed at all time. The overall approach is demonstrated by simulation and experimental results.

We propose a cooperative motion and task planning scheme for multi-agent systems where the agents have independently-assigned local tasks, specified as Linear Temporal Logic (LTL) formulas. These tasks contain hard and soft sub-specifications. A least-violating initial plan is synthesized first for the potentially infeasible task and the partially-known workspace. While the system runs, each agent updates its knowledge about the workspace via its sensing capability and shares this knowledge with its neighboring agents. Based on this update, each agent verifies and revises its plan in real time. It is ensured that the hard specification is always fulfilled and the satisfaction for the soft specification is improved gradually. The design is distributed as only local interactions are assumed. The overall framework is demonstrated by a case study.

In this paper we propose a generic framework for real-time motion planning based on model-checking and revision. The task specification is given as a Linear Temporal Logic formula over a finite abstraction of the robot motion. A preliminary motion plan is first generated based on the initial knowledge of the system model. Then real-time information obtained during the runtime is used to update the system model, verify and further revise the motion plan. The implementation and revision of the motion plan are performed in real-time. This framework can be applied to partially-known workspaces and workspaces with large uncertainties. Computer simulations are presented to demonstrate the efficiency of the framework.

We propose a novel framework to combine model-checking-based motion planning with action planning using action description languages, aiming to tackle task specifications given as Linear Temporal Logic (LTL) formulas. The specifications implicitly require both sequential regions to visit and the desired actions to perform at these regions. The robot's motion is abstracted based on sphere regions of interest in the workspace and the structure of navigation function(NF)-based controllers, while the robot's action map is constructed based on precondition and effect functions associated with the actions. An optimal planner is designed that generates the discrete motion-and-action plan fulfilling the specification, as well as the low-level hybrid controllers that implement this plan. The whole framework is demonstrated by a case study.