Diffusion models have revolutionized imitation learning, allowing robots to replicate complex behaviours. However, diffusion often relies on cameras and other exteroceptive sensors to observe the environment and lacks long-term memory. In space, military, and underwater applications, robots must be highly robust to failures in exteroceptive sensors, operating using only proprioceptive information. In this paper, we propose ProDapt, a method of incorporating long-term memory of previous contacts between the robot and the environment in the diffusion process, allowing it to complete tasks using only proprioceptive data. This is achieved by identifying “keypoints”, essential past observations maintained as inputs to the policy. We test our approach using a UR10e robotic arm in both simulation and real experiments and demonstrate the necessity of this long-term memory for task completion.

This work demonstrates dual-arm lifting of bulky objects based on inferred object properties (center of mass (COM) location, weight, and shape) using proprioception (i.e. force torque measurements). Data-driven Bayesian models describe these quantities, which enables subsequent behaviors to depend on confidence of the learned models. Experiments were conducted using the NASA Jet Propulsion Laboratory's (JPL) RoboSimian to lift a variety of cumbersome objects ranging in mass from 7kg to 25kg. The position of a supporting second manipulator was determined using a particle set and heuristics that were derived from inferred object properties. The supporting manipulator decreased the initial manipulator's load and distributed the wrench load more equitably across each manipulator, for each bulky object. Knowledge of the objects came from pure proprioception (i.e. without reliance on vision or other exteroceptive sensors) throughout the experiments.

This paper presents a model based approach to autonomous dexterous manipulation, developed as part of the DARPA Autonomous Robotic Manipulation (ARM) program. The developed autonomy system uses robot, object, and environment models to identify and localize objects, and well as plan and execute required manipulation tasks. Deliberate interaction with objects and the environment increases system knowledge about the combined robot and environmental state, enabling high precision tasks such as key insertion to be performed in a consistent framework. This approach has been demonstrated across a wide range of manipulation tasks, and in independent DARPA testing archived the most successfully completed tasks with the fastest average task execution of any evaluated team.

This paper describes a system design concept to enable automated sample acquisition on planetary surfaces without the use of additional sensor information such as cameras and force/torque sensors. A simple and robust solution is presented to the unique problems encountered in in situ science applications with its tight constraints on mass, power, computing resources and lack of flight-qualified hardware. Problems addressed include end-effector contact detection, trajectory generation, hardware fault monitoring and autonomous determination of terrain profile.

The Science Activity Planner (SAP) is the primary science operations tool for the Mars Exploration Rover mission and NASA’s Software of the Year for 2004. SAP utilizes a variety of visualization and planning capabilities to enable the mission operations team to direct the activities of the Spirit and Opportunity rovers. This paper outlines some of the challenging requirements that drove the design of SAP and discusses lessons learned from the development and use of SAP in mission operations.

We present an account of the scientific visualization techniques that were used to support science planning for NASA's 2003 Mars Exploration Rover Mission. The great sophistication of the rover's Athena Science Payload required a wide variety of visualization modalities to make the large amounts of scientific data accessible and readily understood by the science planners under the tight time constraints of the tactical planning schedule. A number of techniques were also newly developed to support science operations, including on-the-fly mosaic stitching and rendering, image cube visualization, and data fusion. Many other well-established visualization techniques were applied to science operations also, such as anaglyph stereo, high performance 3D visualization, and optimized I/O and memory management techniques for the loading, processing, and visualization of very large data sets. We discuss each of these topics and demonstrate their application in the MER mission with examples.

Group collaboration capabilities have been developed for Internet-based Mars rover mission operations. Internet-based operations enables scientists to participate in daily Mars rover mission operations from their home institutions. Group collaboration enables geographically separated users to collaboratively analyze downlinked data and plan new activities for the rover. The motivation for group collaboration in Mars rover mission operations and the technologies developed to provide group collaboration are discussed. The group collaboration capability was developed for use in rover mission operations in the 2003 NASA Mars Exploration Rover mission.

This paper discusses a proof-of-concept prototype for ground-based automatic generation of validated rover command sequences. This prototype is based on ASPEN (Automated Scheduling and Planning Environment). This Artificial Intelligence (AI) based planning and scheduling system will automatically generate a command sequence that will execute: within resource constraints and satisfy flight rules. An automated planning and scheduling system encodes rover design knowledge and uses the search and reasoning techniques to automatically generate low-level command sequences while respecting the rover operability constraints. This prototype planning system has been field-tested using the Rocky-7 rover at JPL, and will be field-tested on more complex rovers to prove its effectiveness before transferring the technology to flight operations for an upcoming NASA mission. The goal-driven commanding of planetary rovers greatly reduces the requirements for highly skilled rover engineering personnel. This in turn greatly reduces mission operations costs and permits a faster response to changes in rover states.

The Mars Polar Lander (MPL) mission was the first planetary mission to use Internet-based distributed ground operations where scientists and engineers collaborate in daily mission operations from multiple geographically distributed locations via the Internet. This paper describes the operations system, the Web interface for telescience (WITS), which was used by the MPL mission for Internet-based operations. WITS was used for generating command sequences for the lander's robotic arm and robotic arm camera, and as a secondary tool for sequence generation for the stereo camera on the lander. WITS was also used as a public outreach tool. Results are shown from the January 2000 field test in Death Valley, California.

Automated planning and scheduling, including automated path planning, has been integrated with an Internet-based distributed operations system for planetary rover operations. The resulting prototype system enables faster generation of valid rover command sequences by a distributed planetary rover operations team. The Web Interface for Telescience provides Internet-based distributed collaboration, the automated scheduling and planning environment provides automated planning and scheduling, and an automated path planner provides path planning. The system was demonstrated on the Rocky7 research rover at JPL.

The Web Interface for Telescience (WITS) is an Internet-based tool that the Mars Pathfinder mission used for both mission operations at JPL and public outreach. WITS enables the viewing of downlinked images and results in various ways, terrain feature measurement and annotation, and planning of daily mission activities. WITS is written in the Java language and is accessible by mission scientists and the general public via a web browser. The public can use WITS to plan and simulate their own rover missions. WITS will also be used in the 1998 lander and 2001, 2003, and 2005 rover missions to Mars.

A new robotic crawler is being developed for inspection of large aircraft external surfaces. The multifunction automated crawling system (MACS) has been designed and fabricated to carry miniature instrumentation to perform a wide variety of tasks while being attached to an aircraft's surface. The immediate application of MACS is inspection of the exterior of large military aircraft such as the C-5. Various inspection payload modules could be incorporated onto the MACS crawler to perform the desired inspections. MACS employs ultrasonic motors for mobility and suction cups for surface adherence. MACS has two legs for linear motion and a rotation element for turning, enabling any simultaneous combination of motion from linear to rotation about a central axis.

The Web Interface for Telescience (WITS) has been developed to enable scientists to participate in planetary rover missions from their home institutions, rather than having to travel to JPL as has been required in the past. A scientist accesses WITS via a commercial web browser. WITS provides various views of the scene: including a descent image view which allows the user to see where the rover has been and currently is in a global perspective, a panoramic view which is an overhead view of the area immediately around the rover, and a wedge view which is an image from a rover-mounted camera. Scientists can select science targets in a wedge image and science subtasks to perform at those targets. A mission planner can select rover waypoints to traverse between science targets, and waypoint subtasks to perform at those locations. A public version allows the general public to view the same information and plan their own simulated missions.

This paper describes the design and implementation at the Jet Propulsion Laboratory of a small rover for future Mars missions requiring long traverses and rover-based science experiments. The small rover prototype, called Rocky 7, is capable of long traverses, autonomous navigation, and science instrument control. This rover carries three science instruments, and can be commanded from any computer platform from any location using the World Wide Web. In this paper we describe the mobility system, the sampling system, the sensor suite, navigation and control, onboard science instruments, and the ground command and control system. We also present key accomplishments of a recent field test of Rocky 7 in the Mojave Desert in California.

The new task lines and motion guides approaches to telerobotics are described. Motion guides is a new paradigm for teleoperation of a robot where the path is teleoperated rather than the robot, and the robot is constrained to follow the path. Continuous commands to the robot are only one dimensional: forward, back, or halt along the motion guide. Task lines have subtasks attached to motion guides. The task lines and motion guides have been implemented in a virtual reality environment to enable task description and execution to be done in a natural virtual reality graphics environment rather than via direct interaction with a command program. Subtasks are represented in the virtual reality environment by icons attached to the motion guides. The combination of task lines and motion guides is valuable for ground control of Space Station robots, which is the initial application for this technology.

Time delay and limited communication capacity are the primary constraints in super-long distance telerobotic systems such as space telerobotic systems. Intelligent monitoring is efficient for this problem to provide a function which selects important scenes to help the operator through a monitoring camera. We constructed a telerobotic testbed which includes a connection through the international ISDN and typical space structure (space robot, truss structure land ORU). We conducted trans-Pacific teleoperation experiments using the testbed in ETL as the remote site and a telerobotic console at JPL (Jet Propulsion Laboratory in Pasadena, California) as a local site. Experimental results showed intelligent monitoring to be effective for the above problems.

The development of a ground control station for Earth based control of the dexterous orbital servicing system (DOSS) is described. The DOSS flight test program provides a manipulator which is mounted on a multi-purpose experiment support structure (MPESS) in a Space Shuttle payload bay and may execute tasks from the MPESS or while held by the SSRMS manipulator. The DOSS demonstrates the ability to perform dexterous robotic operations both from an on-board control station and from an Earth-based control station. The initial prototype ground control station for DOSS is described.

A unified control scheme for execution of robotic supervisory and shared control tasks using impedance control with multiple sensory inputs is presented. The control scheme has been designed to provide general Cartesian task execution capability for space telerobotics, both for contact and noncontact tasks. Multiple sensors can be utilized concurrently to enable the execution of a wide range of teleoperation, shared control, and autonomous control telerobotic tasks. Control is based on simultaneous inputs from a trajectory generator, teleoperation, dither generator, proximity sensor, and joint space joint limit and joint singularity avoidance, as well as inertia, damping and stiffness associated with impedance control and force control based on a reference force setpoint.>

Orbital replacement unit (ORU) changeout experiments were performed with the JPL/NASA telerobot testbed system for three different control modes: force reflecting teleoperation (FR), shared compliant control (SCC), and autonomous control (AC). In FR, the operator was provided with kinesthetic force feedback, feeling contact forces through a force-reflecting hand controller. In SCC, the operator controlled a compliant robot, where the robot compliance was implemented by an autonomous compliant control force feedback. In AC, telemanipulation tasks were performed autonomously by using a sequence of macros. Experimental results indicate that SCC greatly reduces the magnitudes and durations of contact forces as compared to FR. AC took the least time to complete the ORU changeout task and, with appropriate operator intervention capabilities, it could be very pleasant to use in some applications with well-structured task environments.>

An interactive user interface, UMI (User-Macro-Interface), and its complete task execution system have been developed and implemented for use by an operator to describe, simulate, and execute tasks on a multiarm telerobot. The operator interactively sets up the execution environment and specifies input parameters for a variety of available task primitives. Several task primitives can be stored together as a task sequence for later sequence execution. The primitives or sequences can be executed on a graphics simulator or on the JPL Telerobot, which includes two task execution manipulators, one manipulator used for positioning cameras, and two six-degree-of-freedom force reflecting hand controllers used for shared and teleoperation modes of execution. In supervisory control, the operator sets up a task primitive or a sequence of primitives and executes the task autonomously. In teleoperation, the user specifies desired initialization parameters and controls the arms with the hand controllers. In shared control, teleoperated inputs from the operator hand controllers are merged during task execution with inputs from user parameterized autonomous primitives. In all modes of operation, UMI allows the operator to stop execution at any time. The UMI interface is intended as the operator interface at a local command site to prepare and send commands to a remote space telerobot.>

A description is given of a complete robot control facility built at the CIT Jet Propulsion Laboratory as part of a NASA telerobotic program to develop a state-of-the-art robot control environment for laboratory-based spacelike experiments. This system has the following features: separation of the computing facilities into local and remote sites, autonomous motion generation in joint or Cartesian coordinates, dual-arm force reflecting teleoperation with voice interaction between the operator and the robots, shared control between the autonomously generated motions and operator controlled teleoperation, and dual-arm coordinated trajectory generation. The system has been used to carry out realistic experiments such as the exchange of an orbital replacement unit, bolt turning, and door opening, using a mixture of autonomous actions and teleoperation, with either a single arm or two cooperating arms.>

Real time hybrid control results with resolved motion force control (RMFC) and task coordinate pole-placement self-tuning (TST) are given. The control methods calculate corrective forces in a Cartesian coordinate frame to eliminate position and force errors. These forces are transformed to equivalent joint inputs for actuation. RMFC uses stochastic approximation to adapt to changing manipulator dynamics. TST uses self-tuning in the task frame to estimate dynamic system models and place closed loop poles. Results show both methods capable of the hybrid control application but friction has a large effect on the control.