In this work, we combine the strengths of humans and robots by developing MARCER, a novel interactive and multimodal end-user robot programming system. MARCER utilizes a Large Language Model to translate users' natural language task descriptions and environmental context into Action Plans for robot execution, based on a trigger-action programming paradigm that facilitates authoring reactive robot behaviors. MARCER also affords interaction via augmented reality to help users parameterize and validate robot programs and provide real-time, visual previews and feedback directly in the context of the robot's operating environment. We present the design, implementation, and evaluation of MARCER to explore the usability of such systems and demonstrate how trigger-action programming, Large Language Models, and augmented reality hold deep-seated synergies that, when combined, empower users to program general-purpose robots to perform everyday tasks.

Human-involved robot learning has made significant strides in performing everyday tasks. However, long-horizon human-robot collaborative tasks remain challenging due to ambiguous subtask goals, which we refer to as intention misalignment between the user and the robot. To address this, we propose a novel human-robot collaboration (HRC) paradigm that utilizes augmented reality (AR) as a bidirectional communication channel. This channel enables the robot to communicate its intentions for ambiguous subtasks to users and adapt based on their feedback. To operationalize these aligned intentions as actionable goals, we employ a goal-conditioned reinforcement learning model, where the goal reflects the user-aligned intention. We validate our approach in a classic pick-and-place task involving three distinct objects, where the user specifies the desired goal object and its target pose.

As autonomous robots become integrated into society, they must socially navigate around humans. We propose that effective social robot navigation relies on three key principles: social norms, perceived safety, and legibility. Our framework, Overlapping Social Navigation Principles, suggests that the strength of each principle is influenced by the presence of other principles. To test our framework, we implemented SRN behaviors on an autonomous robot in a passing scenario and conducted an online study where participants ranked videos of different SRN behavior combinations. Our findings show that incorporating all three principles enhances SRN, with social norms having the greatest impact.

In this work, we present the design and evaluation of an immersive Cyber-Physical Control Room interface for remote mobile robots that provides users with both robot-egocentric and robot-exocentric 3D perspectives. We evaluate the Cyber-Physical Control room against a traditional robot interface in a mock disaster response scenario that features a mixed human-robot field team. In our evaluation, we found that the Cyber-Physical Control Room improved robot operator effectiveness by 28% while navigating a complex warehouse environment and performing a visual search. The Cyber-Physical Control Room also enhanced various aspects of human-robot teaming, including social engagement, the ability of a remote robot teleoperator to track their human partner in the field, and opinions of human teammate leadership qualities.

Robot Imitation Learning (IL) is a crucial technique in robot learning, where agents learn by mimicking human demonstrations. However, IL encounters scalability challenges stemming from both non-user-friendly demonstration collection methods and the extensive time required to amass a sufficient number of demonstrations for effective training. In response, we introduce the Augmented Reality for Collection and generAtion of DEmonstrations (ARCADE) framework, designed to scale up demonstration collection for robot manipulation tasks. Our framework combines two key capabilities: 1) it leverages AR to make demonstration collection as simple as users performing daily tasks using their hands, and 2) it enables the automatic generation of additional synthetic demonstrations from a single human-derived demonstration, significantly reducing user effort and time. We assess ARCADE’s performance on a real Fetch robot across three robotics tasks: 3-Waypoints-Reach, Push, and Pick-And-Place. Using our framework, we were able to rapidly train a policy using vanilla Behavioral Cloning (BC), a classic IL algorithm, which excelled across these three tasks. We also deploy ARCADE on a real household task, Pouring-Water, achieving an 80% success rate.

Lab assignments, in which students build and program robots to accomplish tasks in various environments, are a central component in many undergraduate robotics classes. Such activities require that students operationalize concepts learned in class. However, physical robots are prone to uncertain real-world behavior, making debugging challenging and causing many students to feel stressed about being graded based on their robot's performance. Therefore, we incorporated retakes into our undergraduate robotics class, allowing students to learn from their mistakes and master class content while improving their robots. Initial results show that students widely embrace retakes, use the opportunity to improve, and feel less stressed about the assignments.

The field of end-user robot programming seeks to develop methods that empower non-expert programmers to task and modify robot operations. In doing so, researchers may enhance robot flexibility and broaden the scope of robot deployments into the real world. We introduce PRogramAR (Programming Robots using Augmented Reality), a novel end-user robot programming system that combines the intuitive visual feedback of augmented reality (AR) with the simplistic and responsive paradigm of trigger-action programming (TAP) to facilitate human-robot collaboration. Through PRogramAR, users are able to rapidly author task rules and desired reactive robot behaviors, while specifying task constraints and observing program feedback contextualized directly in the real world. PRogramAR provides feedback by simulating the robot’s intended behavior and providing instant evaluation of TAP rule executability to help end users better understand and debug their programs during development. In a system validation, 17 end users ranging from ages 18 to 83 used PRogramAR to program a robot to assist them in completing three collaborative tasks. Our results demonstrate how merging the benefits of AR and TAP using elements from prior robot programming research into a single novel system can successfully enhance the robot programming process for non-expert users.

Programming robots is a challenging task exacer-bated by software bugs, faulty hardware, and environmental factors. When coding issues arise, traditional debugging techniques are not always useful for roboticists. Robots often have an array of sensors that output complex data, which can be difficult to decipher as raw text. Augmented reality (AR) provides a unique medium for conveying data to the user by displaying information directly in the scene as their corresponding visual definition. In my research, I am exploring various design approaches towards AR visualizations for expert robotics debugging support. From my initial work, I developed design guidelines to inform two future bodies of work which investigate better ways of visualizing robot sensor and state data for debugging.

Programming robots is a challenging task exacer-bated by software bugs, faulty hardware, and environmental fac-tors. When coding issues arise, traditional debugging techniques such as output logs or print statements that may help in typical computer applications are not always useful for roboticists. As a result, roboticists often leverage visualizations that depict various aspects of robot, sensor, and environment states. In this paper, we explore various design approaches towards such visualizations for robotics debugging support, including 3D visualizations presented on 2D displays, as in the popular RViz tool within the ROS ecosystem, visualizations in a two-dimensional graphical user interfaces (2D GUI), and emerging immersive three-dimensional (3D) augmented reality (AR). We present a qualitative evaluation of feedback gathered from 24 roboticists across two universities who used one of these debugging tools and synthesize design guidelines for advancing robotics debugging interfaces.