While recent work in scene reconstruction and understanding has made strides in grounding natural language to physical 3D environments, it is still challenging to ground abstract, high-level instructions to a 3D scene. High-Level instructions might not explicitly invoke semantic elements in the scene, and even the process of breaking a high-level task into a set of more concrete subtasks —a process called hierarchical task analysis— is environment-dependent. In this work, we propose ASHiTA, the first framework that generates a task hierarchy grounded to a 3D scene graph by breaking down high-level tasks into grounded subtasks. ASHiTA alternates LLM-assisted hierarchical task analysis —to generate the task breakdown— with task-driven 3D scene graph construction to generate a suitable representation of the environment. Our experiments show that ASHiTA performs significantly better than LLM baselines in breaking down high-level tasks into environment-dependent subtasks and is additionally able to achieve grounding performance comparable to state-of-the-art methods.

Designing planners and controllers for contact-rich manipulation is extremely challenging as contact violates the smoothness conditions that many gradient-based controller synthesis tools assume. Contact smoothing approximates a non-smooth system with a smooth one, allowing one to use these synthesis tools more effectively. However, applying classical control synthesis methods to smoothed contact dynamics remains relatively under-explored. This paper analyzes the efficacy of linear controller synthesis using differential simulators based on contact smoothing. We introduce natural baselines for leveraging contact smoothing to compute (a) open-loop plans robust to uncertain conditions and/or dynamics, and (b) feedback gains to stabilize around open-loop plans. Using robotic bimanual whole-body manipulation as a testbed, we perform extensive empirical experiments on over 300 trajectories and analyze why LQR seems insufficient for stabilizing contact-rich plans.

Understanding how humans leverage semantic knowledge to navigate unfamiliar environments and decide where to explore next is pivotal for developing robots capable of human-like search behaviors. We introduce a zero-shot navigation approach, Vision-Language Frontier Maps (VLFM), which is inspired by human reasoning and designed to navigate towards unseen semantic objects in novel environments. VLFM builds occupancy maps from depth observations to identify frontiers, and leverages RGB observations and a pre-trained vision-language model to generate a language-grounded value map. VLFM then uses this map to identify the most promising frontier to explore for finding an instance of a given target object category. We evaluate VLFM in photo-realistic environments from the Gibson, Habitat-Matterport 3D (HM3D), and Matterport 3D (MP3D) datasets within the Habitat simulator. Remarkably, VLFM achieves state-of-the-art results on all three datasets as measured by success weighted by path length (SPL) for the Object Goal Navigation task. Furthermore, we show that VLFM’s zero-shot nature enables it to be readily deployed on real-world robots such as the Boston Dynamics Spot mobile manipulation platform. We deploy VLFM on Spot and demonstrate its capability to efficiently navigate to target objects within an office building in the real world, without any prior knowledge of the environment. The accomplishments of VLFM underscore the promising potential of vision-language models in advancing the field of semantic navigation. Videos of real world deployment can be viewed at naoki.io/vlfm.

Traversing terrain with good traction is crucial for achieving fast off-road navigation. Instead of manually designing costs based on terrain features, existing methods learn terrain properties directly from data via self-supervision to automatically penalize trajectories moving through undesirable terrain, but challenges remain in properly quantifying and mitigating the risk due to uncertainty in the learned models. To this end, we present evidential off-road autonomy (EVORA), a unified framework to learn uncertainty-aware traction model and plan risk-aware trajectories. For uncertainty quantification, we efficiently model both aleatoric and epistemic uncertainty by learning discrete traction distributions and probability densities of the traction predictor's latent features. Leveraging evidential deep learning, we parameterize Dirichlet distributions with the network outputs and propose a novel uncertainty-aware squared Earth Mover's Distance loss with a closed-form expression that improves learning accuracy and navigation performance. For risk-aware navigation, the proposed planner simulates state trajectories with the worst-case expected traction to handle aleatoric uncertainty and penalizes trajectories moving through terrain with high epistemic uncertainty. Our approach is extensively validated in simulation and on wheeled and quadruped robots, showing improved navigation performance compared to methods that assume no slip, assume the expected traction, or optimize for the worst-case expected cost.

We present a semi-supervised anomaly detection system for humanoid robots that operates on trajectories with varying lengths, resolutions, and time shifts. The proposed approach utilities optimization to extract a model from joint trajectories under normal operation and seek to identify anomalous behaviors that deviates significantly from the known model. Compared to previously proposed approaches in humanoid anomaly detection that identified only high-level faults, our approach can detect subtle defects in the robot and at the same time, is capable of generalizing to higher-level behaviors. The system is demonstrated on a simulated model of the Atlas humanoid robot, with several experimental scenarios demonstrating detection of both joint-level anomalies and behaviors such as falling.

We address the problem of dynamic stabilization and push recovery for humanoid robots using robust control through convex optimization. By formulating the simultaneous search for a controller and the associated domain of attraction as a single problem, we provide a unified framework in which full-body push recovery controllers can be designed and their performance analyzed. The resulting controller explicitly models external disturbances in the system dynamics and guarantees stabilization under bounded disturbances as well as physical constraints on the robot. Through numerical simulations, we demonstrate full-body push recovery for a planar, three-link, bipedal humanoid in the sagittal plane.

We present successful control strategies for dynamically stable robots that avoid low ceilings and other vertical obstacles in a manner similar to limbo dances. Given the parameters of the mission, including the goal and obstacle dimensions, our method uses a sequential composition of IO-linearized controllers and applies stochastic optimization to automatically compute the best controller gains and references, as well as the times for switching between the different controllers. We demonstrate this system through numerical simulations, validation in a physics-based simulation environment, as well as on a novel two-wheeled platform. The results show that the generated control strategies are successful in mission planning for this challenging problem domain and offer significant advantages over hand-tuned alternatives.

Heterogeneous multi-robot networks require novel tools for applications that require achieving and maintaining formations. This is the case for distributing sensing devices with heterogeneous mobile sensor networks. Here, we consider a heterogeneous multi-robot network of mobile robots. The robots have a limited range in which they can estimate the relative position of other network members. The network is also heterogeneous in that only a subset of robots have localization ability. We develop a method for automatically configuring the heterogeneous network to deploy a desired formation at a desired location. This method guarantees that network members without localization are deployed to the correct location in the environment for the sensor placement.

This paper describes our successful implementation of a robot that autonomously and strategically removes multiple blocks from an unstable Jenga tower. We present an integrated strategy for perception, planning and control that achieves repeatable performance in this challenging physical domain. In contrast to previous implementations, we rely only on low-cost, readily available system components and use strategic algorithms to resolve system uncertainty. We present a three-stage planner for block extraction which considers block selection, extraction order, and physics-based simulation that evaluates removability. Existing vision techniques are combined in a novel sequence for the identification and tracking of blocks within the tower. Discussion of our approach is presented following experimental results on a 5-DOF robot manipulator.