This paper presents recent advances to the Affordance Template (AT) task description language. Affordance Templates provide standardized, easy-to-use tools for defining robot manipulation tasks that provide a high level of augmented reality capabilities to facilitate human-in-the-loop operation, but can also be used to support robot autonomy when coupled with various planning tools. While initially defined in terms of end effector waypoint sequences for bimanual robots, such as the NASA Valkyrie and Robonaut 2, this paper extends the original specification to support integrated mobile manipulation, object-centric template definitions, autonomous grasp determination, and integration with custom and off-the-shelf collision free motion planners. ATs have proved highly adaptable to new robots and new domains by both the authors and third-party groups, and have been used in numerous contexts for NASA, DOD, and industry. As such, we believe that the AT framework provides a strong foundation for robot development in multiple real-world contexts that can be increasingly built upon and expanded to meet the challenges of many new applications.

Reliable, efficient shared autonomy requires balancing human operation and robot automation on complex tasks, such as dexterous manipulation. Adding to the difficulty of shared autonomy is a robot’s limited ability to perceive the 6 degree-of-freedom pose of objects, which is essential to perform manipulations those objects afforded. Inspired by Monte Carlo Localization, we propose a generative human-in-the-loop approach to estimating object pose. We characterize the performance of our mixed-initiative 3D registration approach using 2D pointing devices via a user study. Seeking an analog for Fitts’s Law for 3D registration, we introduce a new evaluation framework that takes the entire registration process into account instead of only the outcome. When combined with estimates of registration confidence, we posit that mixed-initiative registration will reduce the human workload while maintaining or even improving final pose estimation accuracy.

As robotic systems grow in complexity and autonomy, the need for effective and intuitive operator interfaces and task programming tools becomes more acute. The operator must be able to preview, verify, adjust, and observe the robot's plans and actions, and make system-level decisions about the appropriate level of shared autonomy-both before and during a mission. Operator interfaces must be usable by non-experts, even in the presence of degraded situational awareness and environmental uncertainty. Indeed, when used in mission-critical domains such as space or other hazardous environments, these tools must engender confidence that tasks will be performed safely and effectively by providing transparency into the robot's decision-making processes. In this paper, we introduce the PHARAOH system for robot operation. PHARAOH integrates the PRIDE electronic procedure software developed by TRACLabs for NASA field operations with the affordance template visual task programming framework developed to command remote robot systems performing manipulation tasks with varying levels of autonomy. Together, these tools enable an operator to define and follow task procedures while providing access to state-of-the-art robot operation tools as necessary. This blending of human and robot capabilities in a well-defined architecture, covering levels of abstraction from teleoperation to autonomous execution, provides enormous flexibility for mission planners and operators, and represents a significant new tool for robot operation in the field.

This paper introduces the Affordance Template ROS package for quickly programming, adjusting, and executing robot applications in the ROS RViz environment. This package extends the capabilities of RViz interactive markers [1] by allowing an operator to specify multiple end-effector waypoint locations and grasp poses in object-centric coordinate frames and to adjust these waypoints in order to meet the run-time demands of the task (specifically, object scale and location). The Affordance Template package stores task specifications in a robot-agnostic JSON description format such that it is trivial to apply a template to a new robot. As such, the Affordance Template package provides a robot-generic ROS tool appropriate for building semi-autonomous, manipulation-based applications. Affordance Templates were developed by the NASA-JSC DARPA Robotics Challenge (DRC) team and have since successfully been deployed on multiple platforms including the NASA Valkyrie and Robonaut 2 humanoids, the University of Texas Dreamer robot and the Willow Garage PR2. In this paper, the specification and implementation of the affordance template package is introduced and demonstrated through examples for wheel (valve) turning, pick-and-place, and drill grasping, evincing its utility and flexibility for a wide variety of robot applications.

This paper introduces the Robot Task Commander (RTC) framework for defining, developing, and deploying robot application software for use in different run-time contexts. RTC was created by NASA-JSC in conjunction with General Motors for use with the Robonaut-2 and Valkyrie humanoid robot platforms. RTC provides a robot programming syntax and an IDE appropriate for use by experts and non-experts for implementation and execution. An expert developer can implement a new application with a combination of scripts, called process nodes, and state machines that set the control mode of the robot. A non-expert developer can assemble process nodes and controller state machines into novel hierarchical applications using a visual programming language (VPL). This VPL also allows developers to interface with other RTC applications or with third-party software packages using a variety of network transport mechanisms (ROS, TCP, shared memory, etc.). RTC represents an advantage over other robot programming frameworks by providing multiple levels of flexibility for development. The efficacy of RTC is demonstrated through examples of sophisticated behaviors, such as programming the Valkyrie robot to grab objects and turn a valve.

We present a framework for the programming of manipulation behavior by means of an intrinsic reward function that encourages the building of deep control knowledge. We show how this framework can be used to teach new manipulation skills in a hierarchical and incremental fashion. We demonstrate the contributions of this paper on a humanoid robot through three incremental learning stages.

Any given task can be solved in a number of ways, whether through path-planning, modeling, or control techniques. In this paper, we present a methodology for natural task decomposition through the use of intrinsically meaningful potential fields. Specifically, we demonstrate that using classical conditioning measures in a concurrent control framework provides a domain-general means for solving tasks. Among the conditioning measures we use are manipulability [T. Yoshikawam, 1985], localizability [J. Uppala et al., 2002], and range of motion. To illustrate the value of our approach we demonstrate its applicability to an industrially relevant inspection task.