This paper addresses the problem of task planning for robots that must comply with operational manuals in real-world settings. Task planning under these constraints is essential for enabling autonomous robot operation in domains that require adherence to domain-specific knowledge. Current methods for generating robot goals and plans rely on common sense knowledge encoded in large language models. However, these models lack grounding of robot plans to domain-specific knowledge and are not easily transferable between multiple sites or customers with different compliance needs. In this work, we present SayComply, which enables grounding robotic task planning with operational compliance using retrievalbased language models. We design a hierarchical database of operational, environment, and robot embodiment manuals and procedures to enable efficient retrieval of the relevant context under the limited context length of the LLMs. We then design a task planner using a tree-based retrieval augmented generation (RAG) technique to generate robot tasks that follow user instructions while simultaneously complying with the domain knowledge in the database. We demonstrate the benefits of our approach through simulations and hardware experiments in real-world scenarios that require precise context retrieval across various types of context, outperforming the standard RAG method. Our approach bridges the gap in deploying robots that consistently adhere to operational protocols, offering a scalable and edge-deployable solution for ensuring compliance across varied and complex real-world environments. Project website: saycomply.github.io.

Autonomous exploration of unknown environments is fundamentally a problem of decision making under uncertainty where the agent must account for uncertainty in sensor measurements, localization, action execution, as well as many other factors. For large-scale exploration applications, autonomous systems must overcome the challenges of sequentially deciding which areas of the environment are valuable to explore while safely evaluating the risks associated with obstacles and hazardous terrain. In this work, we propose a risk-aware meta-level decision making framework to balance the tradeoffs associated with local and global exploration. Meta-level decision making builds upon classical hierarchical coverage planners by switching between local and global policies with the overall objective of selecting the policy that is most likely to maximize reward in a stochastic environment. We use information about the environment history, traversability risk, and kinodynamic constraints to reason about the probability of successful policy execution to switch between local and global policies. We have validated our solution in both simulation and on a variety of large-scale real world hardware tests. Our results show that by balancing local and global exploration we are able to significantly explore large-scale environments more efficiently.

This paper addresses the problem of autonomous robotic inspection in complex and unknown environments. This capability is crucial for efficient and precise inspections in various real-world scenarios, even when faced with perceptual uncertainty and lack of prior knowledge of the environment. Existing methods for real-world autonomous inspections typically rely on predefined targets and waypoints and often fail to adapt to dynamic or unknown settings. In this paper, we introduce the Semantic Belief Behavior Graph (SB2G) framework as a new approach to semantic-aware autonomous robot inspection. SB2G generates a control policy for the robot, using behavior nodes that encapsulate various semantic-based policies designed for inspecting different classes of objects. We design an active semantic search behavior to guide the robot in locating objects for inspection while reducing semantic information uncertainty. The edges in the SB2G encode transitions between these behaviors. We validate our approach through simulation and real-world urban inspections using a legged robotic platform. Our results show that SB2G enables a more efficient object inspection policy, exhibiting similar behaviors comparable to human-operated inspections.

In active source seeking, a robot takes repeated measurements in order to locate a signal source in a cluttered and unknown environment. A key component of an active source seeking robot planner is a model that can produce estimates of the signal at unknown locations with uncertainty quantification. This model allows the robot to plan for future measurements in the environment. Traditionally, this model has been in the form of a Gaussian process, which has difficulty scaling and cannot represent obstacles. We propose a global and local factor graph model for active source seeking, which allows the model to scale to a large number of measurements and represent unknown obstacles in the environment. We combine this model with extensions to a highly scalable planner to form a system for large-scale active source seeking. We demonstrate that our approach outperforms baseline methods in both simulated and real robot experiments.

To achieve autonomy in unknown and unstruc-tured environments, we propose a method for semantic-based planning under perceptual uncertainty. This capability is cru-cial for safe and efficient robot navigation in environment with mobility-stressing elements that require terrain-specific locomotion policies. We propose the Semantic Belief Graph (SBG), a geometric- and semantic-based representation of a robot's probabilistic roadmap in the environment. The SBG nodes comprise of the robot geometric state and the semantic-knowledge of the terrains in the environment. The SBG edges represent local semantic-based controllers that drive the robot between the nodes or invoke an information gathering action to reduce semantic belief uncertainty. We formulate a semantic-based planning problem on SBG that produces a policy for the robot to safely navigate to the target location with min-imal traversal time. We analyze our method in simulation and present real-world results with a legged robotic platform navigating multi-level outdoor environments.

Robust mission planning is an essential component for mission autonomy to perform complicated tasks in extreme environments. In this paper, we are interested in the role of semantic abstractions for guiding autonomous mission planning. In particular, we focus on how semantics can be leveraged to transition, at the mission level, in between individually robust task plans. We present a mission autonomy framework wherein a task plan adaptation policy leverages up-to-date semantics information in order to adapt to changes that occur during run-time, which endows the robot with better resiliency to unexpected events and improves the overall efficiency of mission operations. Under this new perspective, we provide a concrete and challenging application of autonomous exploration and radio source seeking in a complex multi-level building environment. Experimental results over simulations and real hardware tests demonstrate that the presented semantics-aware mission adaptation more effectively completes the mission with better qualitative results compared to a non-adaptive baseline.

We present a method for solving the coverage problem with the objective of autonomously exploring an unknown environment under mission time constraints. Here, the robot is tasked with planning a path over a horizon such that the accumulated area swept out by its sensor footprint is maximized. Because this problem exhibits a diminishing returns property known as submodularity, we choose to formulate it as a tree-based sequential decision making process. This formulation allows us to evaluate the effects of the robot's actions on future world coverage states, while simultaneously accounting for traversability risk and the dynamic constraints of the robot. To quickly find near-optimal solutions, we propose an effective approximation to the coverage sensor model which adapts to the local environment. Our method was extensively tested across various complex environments and served as the local exploration algorithm for a competing entry in the DARPA Subterranean Challenge.

We present a method for autonomous exploration of large-scale unknown environments under mission time con-straints. We start by proposing the Frontloaded Information Gain Orienteering Problem (FIG-OP) - a generalization of the traditional orienteering problem where the assumption of a reliable environmental model no longer holds. The FIG-OP ad-dresses model uncertainty by frontloading expected information gain through the addition of a greedy incentive, effectively expe-diting the moment in which new area is uncovered. In order to reason across multi-kilometer environments, we solve FIG-OP over an information-efficient world representation, constructed through the aggregation of information from a topological and metric map. Our method was extensively tested and field-hardened across various complex environments, ranging from subway systems to mines. In comparative simulations, we observe that the FIG-OP solution exhibits improved coverage efficiency over solutions generated by greedy and traditional orienteering-based approaches (i.e. severe and minimal model uncertainty assumptions, respectively).

We consider the task of autonomously unloading boxes from trucks using an industrial manipulator robot. There are multiple challenges that arise: (1) real-time motion planning for a complex robotic system carrying two articulated mechanisms, an arm and a scooper, (2) decision-making in terms of what action to execute next given imperfect information about boxes such as their masses, (3) accounting for the sequential nature of the problem where current actions affect future state of the boxes, and (4) real-time execution that interleaves high-level decision-making with lower level motion planning. In this work, we propose a planning, learning, and reasoning framework to tackle these challenges, and describe its components including motion planning, belief space planning for offline learning, online decision-making based on offline learning, and an execution module to combine decision-making with motion planning. We analyze the performance of the framework on real-world scenarios. In particular, motion planning and execution modules are evaluated in simulation and on a real robot, while offline learning and online decision-making are evaluated in simulated real-world scenarios.

This paper serves as one of the first efforts to enable large-scale and long-duration autonomy using the Boston Dynamics Spot robot. Motivated by exploring extreme environments, particularly those involved in the DARPA Subterranean Challenge, this paper pushes the boundaries of the state-of-practice in enabling legged robotic systems to accomplish real-world complex missions in relevant scenarios. In particular, we discuss the behaviors and capabilities which emerge from the integration of the autonomy architecture NeBula (Networked Belief-aware Perceptual Autonomy) with next-generation mobility systems. We will discuss the hardware and software challenges, and solutions in mobility, perception, autonomy, and very briefly, wireless networking, as well as lessons learned and future directions. We demonstrate the performance of the proposed solutions on physical systems in real-world scenarios.3The proposed solution contributed to winning 1st-place in the 2020 DARPA Subterranean Challenge, Urban Circuit.4

Search-based planning for relatively low-dimensional motion-planning problems such as for autonomous navigation and autonomous flight has been shown to be very successful. Such framework relies on laying a grid over a state-space and constructing a set of actions (motion primitives) that connect the centers of cells. However, in some cases such as kinodynamic motion planning, planning for bipedal robots with high balance requirements, computing these actions can be highly non-trivial and often impossible depending on the dynamic constraints. In this paper, we explore a soft version of discretization, wherein the state-space remains to be continuous but the search tries to avoid exploring states that are likely to be duplicates of states that have already been explored. We refer to this property of the search as soft duplicate detection and view it as a relaxation of the standard notion of duplicate detection. Empirically, we show that the search can efficiently compute paths in highly-constrained settings and outperforms alternatives on several domains.

Simultaneous localization and planning (SLAP) is a crucial ability for an autonomous robot operating under uncertainty. In its most general form, SLAP induces a continuous partially observable Markov decision process (POMDP), which needs to be repeatedly solved online. This paper addresses this problem and proposes a dynamic replanning scheme in belief space. The underlying POMDP, which is continuous in state, action, and observation space, is approximated offline via sampling-based methods, but operates in a replanning loop online to admit local improvements to the coarse offline policy. This construct enables the proposed method to combat changing environments and large localization errors, even when the change alters the homotopy class of the optimal trajectory. It further outperforms the state-of-the-art Feedback-based Information RoadMap (FIRM) method by eliminating unnecessary stabilization steps. Applying belief space planning to physical systems brings with it a plethora of challenges. A key focus of this paper is to implement the proposed planner on a physical robot and show the SLAP solution performance under uncertainty, in changing environments and in the presence of large disturbances, such as a kidnapped robot situation.

Parts assembly, in a broad sense, is to make multiple objects to be in specific relative poses in contact with each other. One of the major reasons that make it difficult is uncertainty. Because parts assembly involves physical contact between objects, it requires higher precision than other manipulation tasks like collision avoidance. The key idea of this paper is to use simulation-aided physical reasoning while planning with the goal of finding a robust motion plan for parts assembly. Specifically, in the proposed approach, a) uncertainty between object poses is represented as a distribution of particles, b) the motion planner estimates the transition of particles for unit actions (motion primitives) through physics-based simulation, and c) the performance of the planner is sped up using Multi-Heuristic A* (MHA*) search that utilizes multiple inadmissible heuristics that lead to fast uncertainty reduction. To demonstrate the benefits of our framework, motion planning and physical robot experiments for several parts assembly tasks are provided.

In a cluttered scene, an object is often occluded by other objects, and a robot cannot figure out what the object is and perceive its pose exactly. We assume that the robot is equipped with a depth sensor and given a database of 3D object models and their grasping poses, but yet there is uncertainty about object's class and pose. In this paper, we study the problem of how to predict the class and pose of an occluded object by carefully taking a sequence of observations. To find the best sequence of viewpoints by the robot, we construct hypotheses of the states of the target and occluding objects, and update our belief state as new observations come in. Every time selecting the next robot pose, we greedily choose the one that is expected to reduce the uncertainty the most. Based on the theoretical analysis of adaptive submodular maximization problems, this process is guaranteed to find a near-optimal sequence of robot poses in terms of observation and traverse costs. To validate the proposed method, we present simulation and robot experiments using a PR2.

When one uses a hand-held tool, the fingers often make the tool to be in contact with the palm in the form of multi-contact manipulation. Multi-contact manipulation is useful for object-environment interaction tasks because it can provide both powerful grasping of the object body and dexterous manipulation of the object end-effector. However, dealing with the internal link contact with the object is not trivial. In this paper, we propose Reactional Internal Contact Hypothesis that regards the internal contact force as a reaction force so that the desired finger force can be reduced. By taking a handwriting task as an example, optimal configuration search and grasping force computation problems are addressed based on this hypothesis and validated via dynamic simulation.

In this paper, the grasping force control of a robotic hand based on a torque to velocity transformation using force/torque (F/T) sensors with gravity compensation is addressed. The force controller is designed and the task force is transformed into command torque. Then, the torque is converted into command velocity for the velocity servo control through torque to velocity transformation. Inner velocity controller is modeled, and the additional torque due to the gravity effect is augmented using superposition principle; then, the analysis of the overall system and controller is conducted through the grasping experiment. As a result of this, competent results are obtained.

Human's action strategy is a good source of robot controller design. For there is no decisive criterion on balance control during manipulation tasks, human motion data are obtained and analyzed in this paper. Based on the observation of the center of mass (CoM) being proportional to target object distance but limited inside the supporting polygon, the bound-proportional CoM planner is proposed. Along with the CoM planner, whole-body balance and grasping controller for a dualarm robot is suggested in a simple and computationally efficient structure. Dynamic simulation is conducted for validation, and showed competent results.

There are many difficulties in dexterous object manipulation by multi-fingered hands, due to redundant degree-of-freedom of the entire system and uncertainties in interaction with the object. In this paper, however, 3D object manipulation without any external sensors are attempted. Under the assumption of point contact without rolling, the object position and orientation are computed in relative sense and used as the feedback for object manipulation. Overall system dynamics including two cone-shaped finger robots and an arbitrary object is modeled, and the closed-loop system stability is analyzed based with the proposed controllers for stable grasping and object position/orientation control. In order to validate the proposed method, dynamic simulation is conducted, and showed complacent results.

This paper addresses a torque control method in a high friction robot manipulator. A stiction feed-forward compensator is proposed to eliminate the control problem caused by the nonlinear friction and disturbance. In order to control a robot manipulator with unknown effects, a time-delay control method is used to control the torque. One degree of freedom flexible joint robot manipulator with a joint torque sensor is used to show the performance of the proposed control method.

Redundant manipulator control usually brings about a lot of complexity. This paper proposes a quite simple approach for concurrent set-point regulation of position and orientation of a redundant manipulator using the virtual spring-damper hypothesis, which offers human-like movements without solving the inverse dynamics. For orientation, quaternion representation is applied to give singularity-free orientation control. Augmenting quaternion feedback to the state vector of the system, the stability is analytically proved, and dynamic simulation is conducted for validation of the method. Competition between position and orientation control is also briefly discussed based on the simulation results.

There are many difficulties in operating a humanoid which has high degree-of-freedom and instability in balancing its body. In addition, due to the shape of a humanoid, it is expected to have motions like a human. In order to overcome its operational difficulties and to provide a humanlike motion, a teleoperation with the motion imitation is studied in this paper. Specifically, a framework for online generation of a footprint from a human walking motion is proposed. The human walking motions acquired from a motion capture device are parameterized and normalized to give a human independent foot motion. The normalized parameters are restored by a humanoid considering its hardware limit. The restored footprints generate a walking trajectory of a humanoid, which imitates the human walking motion in terms of the footprint. Experiments are conducted with MAHRU-R, a humanoid robot developed in KIST.