Selma Liliane Wanna, Agnes Luhtaru, Jonathan Salfity, Ryan Barron, Juston Moore, Cynthia Matuszek, Mitch Pryor. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

In this work, we present PlaceNet: a deep learning framework for mobile manipulator base placement which provides solutions to the shortcomings common in the state-of-the-art. Our method addresses the lack of obstacle awareness of reachability methods and the limited generalization of learning methods. Using only the raw pointcloud and task pose data as input, PlaceNet learns the concepts of reachability and obstacle occlusions in an environment-independent manner, enabling its use in situations outside its training experiences. Tests comparing PlaceNet to inverse reachability and heuristic methods demonstrated state-of-the-art performance in both the In-Distribution and Out-Of-Distribution test sets, achieving as high as 98% success rate for problems with many solutions, and an 82% success rate overall. PlaceNet can be trained on grounded pointcloud data from any source without the need for dynamic simulation, marking it as an accessible alternative to similar frameworks which require expensive, high-performance GPUs for running simultaneous simulation and training or which depend on labor intensive data collection. PlaceNet is lightweight during deployment and can easily run with low latency on affordable hardware, including laptop GPUs and the NVIDIA Jetson line for embedded deployment.

Solving the Simultaneous Localization and Mapping (SLAM) problem is essential for most mobile robotics applications that do not have an a priori environment representation. The SLAM problem is well-studied, with previous works demonstrating impressive results with a variety of robots, sensors, and environments. However, despite the widespread need for SLAM solutions across a broad spectrum of robotics disciplines and applications, it remains challenging to quickly and easily scale existing solutions to new robots, environments, and tasks. In this paper, we address this problem by introducing Hyla-SLAM, a framework for 3D LiDAR-based SLAM that uses dynamic memory management to efficiently create and manage maps of environments of arbitrary size and density. Hyla-SLAM also scales to diverse systems and applications by using behavior trees to maximize runtime flexibility and extensibility. We demonstrate the scalability of Hyla-SLAM in experiments using datasets collected in North America, Europe, and Asia to generate a single, unified global-scale map thousands of kilometers across that can be efficiently accessed and expanded. We also show experiments using the behavior tree interface to make robot- or task-informed modifications that enable deployment on heterogeneous robots with varying system constraints. These results demonstrate the framework’s ability to efficiently create and manage huge maps while generalizing to a wide range of systems and applications with minimal reconfiguration. We release Hyla-SLAM’s code implementation1open-source.

Manipulating articulated objects is especially difficult if the robot is operating autonomously or far from any human operator. Object articulation imposes strict constraints on robot motion, making it a challenge to generate valid trajectories to complete the task. Problems compound when the robot is mobile and operates in an uncontrolled environment, where the location or articulation model is unknown a priori. In this work, we leverage screw theory to model constraints imposed on a generic manipulator by simple articulated objects and present two novel, fast, and robust methods–Sequential Path Stepping (SPS) and Direct Screw Sampling (DSS)–for planning trajectories by directly sampling these constraints. We show that these methods are hardware-agnostic and work in cluttered environments using long, complex paths modeled by multiple screw-axis constraints. We demonstrate that modeling constraints using multiple screw axes handles objects with multiple DoF, or multi-step tasks (e.g., turning a knob before opening the door). In addition, the direct sampling component of the proposed approaches is implemented as a module that used with existing well-known probabilistic planning methods, allowing customization across different hardware, domains, or planning problems. We validate our methods across many planning and inverse kinematic elements, with three different mobile and stationary manipulators, and on a set of challenging planning problems that include single- and multiple-screw constraints. Results demonstrate a 97.6% success rate planning in cluttered environments, in less than 0.2 seconds.Note to Practitioners—This paper was motivated by the articulated manipulation problem for mobile manipulators. The solutions also apply to non-articulated object manipulation and task planning. Existing approaches do not include helical constraints (e.g., turning a threaded bolt), require operator oversight, or lack integration with MoveIt: the de facto kinematic manipulation standard. We address these issues with two methods that utilize screw theory to enable helical (inclusive of revolute and prismatic) constraints, require little input from operators, and align with standard plan-execute, task definition, and robot configuration capabilities offered by MoveIt and ROS. Through experimentation, we show that our methods easily plan manipulations of arbitrary articulated objects-including those with multiple DoF-are relatively quick, successful, and hardware agnostic.

Robots operating in unpredictable environments require versatile, hardware-agnostic frameworks capable of adapting to various tasks. While a recent screw-based affordance approach shows promise, it faces challenges in avoiding undesirable configurations, singularity navigation, and task success prediction. To address these limitations, we propose a novel framework that incorporates gripper orientation control and generates complete joint trajectories in real time for screw-based task affordance execution. Our method models the affordance and manipulator as a closed-chain mechanism, introducing an innovative approach to solving closed-chain inverse kinematics. It encapsulates task constraints and simplifies task definitions, while remaining hardware and robot agnostic, robust to errors, and invariant to the initial grasp. We validate our framework with simulations on a UR5 robot and real-world implementation on a Boston Dynamics Spot robot. Our experiments demonstrate rapid joint trajectory generation (0.0077 to 0.098 seconds) for various tasks, including a 420-degree valve turn with consideration of the gripper orientation. Comparison with the state-of-the-art methods shows a 4x improvement in planning time, reduced joint movement and achievement of greater task goals.

Multiple object tracking (MOT) is a valuable perception function for robots and intelligent systems. Despite rapid improvements in metrics such as Average Multiple Object Tracking Accuracy (AMOTA) on MOT benchmarks, many trackers are application-specific or run at speeds10 FPS. MaRMOT is open source and can be extended with new detectors, process models, matching algorithms, and track management techniques.

As end users interact with increasing numbers of autonomous or semi-autonomous systems, collaboration and supervision become more complicated. People must simultaneously manage multiple systems, and most current interfaces do not scale. Augmented Reality (AR) offers a promising solution by placing information over the real world, allowing users to concurrently track the scene and robot(s) – potentially improving scaling for these devices. In this work, we leverage user gaze - a powerful indicator of attention suitable for reactive systems - to lower cognitive burden and improve performance such that scaling to multiple agents is possible. Gaze is probed in two modalities. In active mode, the user looks at a menu and presses a button to request additional information. The passive mode provides more information when the user’s gaze dwells on the menu. We performed two studies: 1) participants complete a visual search task with increasing numbers of virtual robotic agents and 2) participants must track the dynamic status of a team with physical agents. Results from the first study show that the passive and active interfaces provide better scaling compared to a non-interactive interface as the number of robots increases. In both studies, users preferred the passive mode, citing a lower mental demand, effort, and frustration.

Human–robot interaction (HRI) in human social environments (HSEs) poses unique challenges for robot perception systems, which must combine asynchronous, heterogeneous data streams in real time. Multimodal perception systems are well-suited for HRI in HSEs and can provide more rich, robust interaction for robots operating among humans. In this article, we provide an overview of multimodal perception systems being used in HSEs, which is intended to be an introduction to the topic and summary of relevant trends, techniques, resources, challenges, and terminology. We surveyed 15 peer-reviewed robotics and HRI publications over the past 10+ years, providing details about the data acquisition, processing, and fusion techniques used in 65 multimodal perception systems across various HRI domains. Our survey provides information about hardware, software, datasets, and methods currently available for HRI perception research, as well as how these perception systems are being applied in HSEs. Based on the survey, we summarize trends, challenges, and limitations of multimodal human perception systems for robots, then identify resources for researchers and developers and propose future research areas to advance the field.

Performing a manipulation contact task in an unknown and unstructured environment is still a challenge. Learning from Demonstration (LfD) techniques provide an intuitive means to define difficult-to-model contact tasks, but have attributes that make them undesirable for novice users in uncertain environments. We present a novel end-to-end system that captures a single manipulation task demonstration from an augmented reality (AR) head-mounted display (HMD), computes an affordance primitive (AP) representation of the task, and sends the task parameters to a mobile manipulator for execution in realtime. Using an AR HMD for task demon-stration and APs for task representation has several distinct advantages. AR task demonstration is intuitive, practical, and can be accomplished without requiring sensor installment in the task environment. APs provide a compact and legible task representation, enabling scalability, generalization, and modification of the task without significant data processing overhead. In this effort, we demonstrate system generalization with 10 object manipulation tasks, confirming the computed parameters from all tasks fit within AP tolerances. Secondly, we evaluate a mobile manipulator robot's ability to perform human-demonstrated tasks using AP representation. To increase robustness, we devised and tested four methods to correct for inherent, irreducible position errors in the system. A final study shows the system has a manipulation success rate of 96 % from a single manipulation demonstration on an industrial wheel valve.

We propose a novel Augmented Reality (AR) Head Mounted Display (HMD) haptic-enabled device which is capable of providing visual and vibrotactile directional cues to locate objects of interest. Using the vibrotactile cues, the device communicates prioritization information to users without the need for additional graphics. This work builds upon a human-robot teaming AR application, AugRE, which provides both situational awareness and control interfaces for any number of ROS-enabled robotic systems. The vibrotactile haptic component developed attaches to the AR-HMD and uses a sequence of vibrations to direct the user to specific objects in their proximity. The visual haptic component does the same by overlaying a holographic arrow on the HMD. We present results from a pilot study and discuss system limitations and research areas that may help direct future development for human-robot teaming applications. Results indicate that visual haptic cues provide the best response times. However, high-frequency vibrotactile haptic cues may be a viable alternative for some tasks where the visual space is already saturated.

Target search via Bayesian estimation is a commonly studied problem in which the goal is to locate and possibly characterize a lost “target” within a given space using noisy sensors. Example applications include locating gas leaks, radio sources, ships at sea, etc, using appropriate sensors for the specific phenomena. The motivating application for this effort is detecting and locating nearby radioactive material in nuclear facilities. Past work has produced several implementations for addressing this problem, with varying levels of accuracy and sophistication. Here we present both new and previously developed solution methods to the radioactive source localization problem. The solutions have diverse strategies for selecting advantageous positions at which to collect measurements. The objective is to minimize the number of measurements needed to complete the target localization. We empirically compare the effectiveness of various strategies, three of which are novel and prove superior over prior methods of measurement selection. The relative advantages of the novel strategies are discussed as well as their broader applicability to a general class of target search problems where measurement intensity correlates with target proximity. Note to Practitioners—This paper was motivated by the desire to automate tasks in radiation work spaces of nuclear facilities, particularly laboratory facilities such as those at Los Alamos National Laboratory. Prior work by the authors studied the problem of autonomously locating and characterizing radiation sources in a survey region. This paper presents enhancements to the prior method, which reduce the number of radiation measurements needed to produce the solution. This increases the efficiency and practicality of the method for end users.

Augmented Reality (AR) provides a method to superimpose real-time information on the physical world. AR is well-suited for complex robotic systems to help users understand robot behavior, status, and intent. This paper presents an AR system, Augmented Robot Environment (AugRE), that combines ROS-based robotic systems with Microsoft HoloLens 2 AR headsets to form a scalable multi-agent human-robot teaming system for indoor and outdoor exploration. The system allows multiple users to simultaneously localize, supervise, and receive labeled images from robotic clients. An overview of AugRE and details of the novel system architecture that allows for large-scale human-robot teaming is presented below. Studies showcasing system performance with multiple robotic clients are presented. Results show that AugRE can scale to 50 robotic clients with minimal performance degradation, due in part to key components that leverage a recent advancement in robotic client-to-client communication called Robofleet. Finally we discuss new capabilities enabled by AugRE.

Both routine radiation surveys and incident response—as currently performed by human workers—are time-consuming and involve significant, potentially unanticipated radiation dose (especially for accident response where levels prohibit human presence entirely). Previous efforts addressed routine surveys using robotic systems, but they must also localize and characterize discrete sources when anomalies occur. To be effective, characterization must account for real-world complications, including multiple sources, environmental attenuation, spatial localization, and isotopic identification. Recursive Bayesian estimation (RBE) using grid-based estimators and particle filters has been previously investigated, but we eliminate previous open-space assumptions by including attenuation models and sensor position height in the sequential sampling strategy. The method also incorporates autonomous isotopic identification via gamma spectroscopy, which supports the attenuation modeling and improves the computational efficiency in multisource cases. In addition, survey points are autonomously optimized using the Fisher information. A radioactive decay model is implemented to generalize the method by addressing short-lived nuclides. The developed hardware is evaluated in multiple scenarios where no operator intervention is required, effectively eliminating operator dose uptake from localization/characterization in real-world scenarios. Two different mobile robots are used, demonstrating the portability of the data collection and analysis software. Also, the generalized complexity is documented to support planning in future scenarios.Note to Practitioners—This article was motivated by the desire to automate tasks in radiation work spaces of nuclear facilities, particularly laboratory facilities such as those at Los Alamos National Laboratory. While there is a great deal of prior literature dealing with the simple problem of collecting radiation readings in some regular spatial patterns, there remains the more advanced problem of inferring characteristics of the radiation sources. The localization problem has been studied in academia, but the developed solutions fall short of being practically deployable in real facilities. Our goal with this work is to enhance the sophistication of the radiation data analysis and modeling in order to shrink the gap between the state of the art and the demands of the target environments.

To facilitate continuous development of novel HRI systems, it is beneficial to have tools that enable quick adjustments, flexibility, or re-invention of the human interfaces when system requirements change due to updates in the state-of-the-art, application domain, etc. Thus, modularity is a key design principle which promotes software reuse and scalability, and reduces development time and cost. Hence, a robot’s autonomous capabilities should not depend on the command interface and should be decoupled via a common format that possesses the descriptive capabilities for outlining tasks and has a sensible syntax for HRI. In this paper, we propose the Unified Meaning Representation Format (UMRF) , which provides the syntax and semantics for passing both simple and complex commands modelled as control flow graphs. UMRF is a standalone meaning representation container that supports embedding other meaning representation formalisms, such as predicate-argument semantics and graphical meaning representation formats, making it adoptable as a standard task description format for semi-autonomous systems in HRI domains. In this article, we define the UMRF syntax and semantics, summarize its unique aspects relative to related task description formats, and demonstrate its descriptiveness by navigating a robot via concurrent (e.g., gestures and speech) and interchangeable input systems (e.g., Google Assistant, Amazon Alexa).

Using robotic manipulators to remotely perform real-world complex contact tasks is challenging whether tasks are known (due to uncertainty) or unknown a priori (lack of motion waypoints, force profiles, etc.). For known tasks we can integrate and utilize Affordance Templates with a selective compliance jogger to remotely perform high dimensional velocity/force tasks - such as turning valves, opening doors, etc. Affordance Templates (ATs) contain virtual visual representations of task-relevant objects and waypoints for interacting with visualized objects. Operators and/or developers align pre-defined ATs with real-world objects to complete complex tasks, potentially reducing the operator's input dimension to a single initiation command. In this work, we integrate a compliant controller with existing ATs to reduce the operator's burden by 1) reducing the dimension of commanded inputs, 2) internally managing contact forces even for complex tasks, and 3) providing situational awareness in the task frame. Since not all tasks can be modeled for general teleoperation, we also introduce Affordance Primitives which reduce the command dimensionality of complex spatial tasks to as low as 1-dimensional input gestures as demonstrated for this effort. To enable reduction of the command input's dimension, the same compliant jogger used to robustly handle uncertainty with ATs is used with Affordance Primitives to autonomously maintain force constraints associated with complex contact tasks. Both Affordance Templates and Affordance Primitives - when used in tandem with a compliant jogger - provide a safe, intuitive, and efficient teleoperation system for general use including using primitives to easily develop new Affordance Templates from newly completed teleoperation tasks.

Drilling and completing an oil/gas well is a time-sensitive and high-value operation, in which environment/system parameters change in unseen, unpredictable environments. Safety issues arise at every stage. Drilling principles can be taught using traditional methods, but safety and event response are difficult to teach in such formats. Here, in this article, we integrate a hardware-in-the-loop simulator, downhole physics, and auxiliary touchscreen interfaces (similar to a rig's add-on equipment) to develop a realistic, real-time drilling simulator for well control operation training. Realistic operational data are supplied to the simulator representative of downhole operations, including unplanned well events. The well plan accounts for drilling parameter changes, the pore-pressure fracture-gradient drilling window, mud weights, etc., which occur in response to the unplanned events. The developed simulator is used for hands-on training, human factor studies, model verification, and evaluating new auxiliary equipment and/or operational procedures. A critical research objective was evaluating the accuracy/realism of the developed system. To do so, eight petroleum engineering students and 11 certified drillers were trained and asked to complete a comprehensive (>6 h) drilling operation. System accuracy was measured by comparing how new versus experienced operators learned to operate the simulator, execute mission-critical tasks, and respond to unplanned events. The results validate the realism of the developed simulator and scenarios, since personnel with prior drilling experience took significantly less time to master the system.

One of the main difficulties in teleoperated systems is providing an operator with sufficient Situational Awareness (SA). This paper introduces three open-source packages that improve the operator's SA using the Robot Operating System (ROS). The first package-rviz_textured_sphere-allows rendering panospheric camera outputs as spherical images in the ROS visualization software RViz. A system where the visualization of this spherical data using an open-source virtual reality (OSVR) headset in the ROS framework is achieved with the second package: rviz_plugin_osvr. Finally, the third package-pointcloud_painter-projects spherical data onto a 3D depth cloud scan of the scene generated from a rotating lidar. This package outputs a XYZRGB pointcloud that can be visualized either in RViz or using the virtual reality headset. Together, these technologies address the wider issue of limited SA in robotics and represent a substantial advancement in the environment visualization capabilities available to open-source robotics developers.

Los Alamos National Laboratory and the University of Texas at Austin recently implemented a robotically controlled nondestructive testing (NDT) system for X-ray and neutron imaging. This system is intended to address the need for accurate measurements for a variety of parts and, be able to track measurement geometry at every imaging location, and is designed for high-throughput applications. This system was deployed in a beam port at a nuclear research reactor and in an operational inspection X-ray bay. The nuclear research reactor system consisted of a precision industrial seven-axis robot, 1.1-MW TRIGA research reactor, and a scintillator-mirror-camera-based imaging system. The X-ray bay system incorporated the same robot, a 225-keV microfocus X-ray source, and a custom flat panel digital detector. The robotic positioning arm is programmable and allows imaging in multiple configurations, including planar, cylindrical, as well as other user defined geometries that provide enhanced engineering evaluation capability. The imaging acquisition device is coupled with the robot for automated image acquisition. The robot can achieve target positional repeatability within 17 μm in the 3-D space. Flexible automation with nondestructive imaging saves costs, reduces dosage, adds imaging techniques, and achieves better quality results in less time. Specifics regarding the robotic system and imaging acquisition and evaluation processes are presented. This paper reviews the comprehensive testing and system evaluation to affirm the feasibility of robotic NDT, presents the system configuration, and reviews results for both X-ray and neutron radiography imaging applications.

Telerobotics (i.e., remote-controlling robots) is highly attractive for tasks in potentially dangerous situations, e.g., search and rescue, space exploration, and handling hazardous materials. However, when telerobots are deployed to complete tasks, the human operator needs to develop task plan and figure out how to execute it using the available control interface. Inappropriate controls can lead to excessive cognitive load and long task completion times. If the human operator can interact with the robot in an intuitive way, he or she can focus more on the task. For that reason, we have designed a human-centered control interface that allows the operator to modify the user perspective, command via hand gestures and natural language, and scale human input motion to any suitable range on the robot. The interface consists of a Leap Motion Controller for hand tracking, microphone for speech detection, and a simple turn knob for varying the scaling factor between the human and robot motions. The teleoperator software utilizes the Robot Operating System (ROS) which enables open-source development and hardware agnosticism. In this paper we demonstrate the feasibility of the proposed system by executing a high-precision task of threading a needle. Furthermore, we present results from a usability study in where people were asked to complete high-precision tasks with both the developed human-centered gestural control input and a conventional functionality-centered drag-and-drop interface.

This paper describes the integration of several techniques for cooperative control of both tele-operated and autonomous redundant manipulators with overlapping workspaces. Motivating this research is a tele-operated surgical manipulator(s) supported by autonomous robot(s) that insert/remove items from the surgical workspace. The dynamic and unpredictable location of obstacles in a small workspace requires a complete strategy to avoid collisions when completing critical tasks and minimizes the need for user (i.e. the surgeon) intervention to make path planning decisions or resolve impasse situations. Three techniques are integrated into the decision-making for the manipulators: an intelligent and intuitive EEF velocity scaling, coordinated null-space optimization across affected manipulators, and collision detection. Central to all three techniques is an estimated time- to-collision formulation that combines distances between objects with their higher order properties, thus only objects currently moving towards each other are included in the collision avoidance techniques. The use of multiple techniques derived from the terms of a single metric results in a computationally efficient strategy for tele-operated and autonomous manipulators sharing the same workspace.

In this paper, a metrology method is presented to calibrate modular robots. It is to perform metrology at the modular level and measure the kinematic parameters of each module so that the acquired information may be used for obtaining an "as built" model every time either a modular robot is reconfigured or maintenance has been made. Based on the product-of-exponentials formula and the modified dyad kinematics, two calibration models with or without the interface error compensation are proposed. The success of this method is based on two key technologies; the first key is an accurate and interchangeable standardized interface and the second key is to develop a state-of-the-art 3-D metrology system which is accurate, flexible, and easy to use. This paper presents these topics with kinematic models for modular robot calibration.

This paper examines the commonalities of manual controllers used in robotics for teleoperation. These include devices ranging from simple joysticks to force-reflecting controllers. The similarities in functionality and behaviour of these controllers is further exploited to develop a reusable software architecture for manual controller interfacing. The development of this architecture is based on object-oriented design. The application of this design philosophy led to the development of a hierarchy of software components that are manual controller independent and also have a standardized interface. Reusability of these components is supported through generality and extensibility. The key design requirements for this architecture were: open-system, reusable, application independent, extensive error-handling and safety checking, applicability to real-time control and simulation, and reduction in program development time. This paper discusses the software analysis and design issues that were faced to meet the architecture requirements. Further, this architecture is demonstrated using four different manual controllers and a teleoperated dual-arm robotic manipulator.