The use of robots in high-risk and extreme environments is crucial for tasks that are dangerous or inaccessible to humans and require high precision. Particularly in scenarios where the cost of failure is high, remote human teleoperation can be the preferred method of robot control due to the adaptability and high-level decision making of humans. Teleoperation brings many challenges including lack of accurate prior knowledge about the environment, limited views of the environment by on-board sensors, and especially inconsistent latency. 7-DOF (degrees of freedom) manipulators provide redundancy which can be utilized for increased flexibility in manipulation, and may be preferred to 6-DOF manipulators in many scenarios. The redundancy, however, must be considered by the teleoperation system. We present an extension to an existing Interactive Planning and Supervised Execution (IPSE) system that enables full teleoperation of a 7-DOF robot by encoding the redundant degree of freedom with a Shoulder-Elbow-Wrist (SEW) angle, which is user-manipulable via an SEW angle graph. Additionally, we introduce a novel user interface feature that encodes robot state information into a 2D image which is displayed directly on the SEW angle graph. We conduct a user-study which demonstrates that the addition of this SEW graph significantly reduces task completion time.

Teleoperation of robots in space is challenging due to high latency and limited workspace visibility. Previously, the Interactive Planning and Supervised Execution (IPSE) and Augmented Virtuality systems were developed to reduce failure risk. These tools were visualized on a 3D da Vinci surgical console and operated using the da Vinci manipulators or visualized on conventional monitors and operated with a keyboard and mouse. Experimental studies indicated operator preference for the latter. In this work, we develop a 3D virtual reality (VR) interface for IPSE, implemented on a Meta Quest 2 head-mounted display (HMD), and evaluate it against the prior 2D, keyboard-and-mouse-based interface. The results demonstrate improved operator load with the 3D VR interface, with no decrease in task performance, while also providing cost and portability benefits compared to the conventional 2D interface.

Exerting forces and torques instantaneously on rigid magnetic bodies with no physical connection is an attractive feature of magnetic robotics. This demonstrates great potential for manipulating tools that are externally controlled through the use of magnetic fields in minimally invasive surgeries. The magnetic field can be controlled by the application of currents to electromagnets positioned around the surgical site, and the necessary currents for a specific desired manipulation can be derived from magnetic field models. However, the magnetic field generated by electromagnetic coils are highly nonlinear, especially in the vicinity of the magnetic field sources, which complicates the modeling process. While simple dipole models provide a good approximation for these fields far away from the electromagnets, these models tend to be highly inaccurate near the sources. Magnetic surgical applications benefit from models which accurately describe fields and gradients both near and far from the field source. Particularly, since forces and torques decay inversely proportionally with the cube of the distance to the coil, inaccurate modeling near the coil makes large regions near the coil unfit for applications requiring precisely predicted motion. Estimation errors near coils generate inaccuracies in field models that significantly reduce control performance for rigid magnetic bodies. In order to tackle this problem, we utilize Zernike basis functions to analytically represent the nonlinear magnetic field distribution more accurately. The accuracy of the controller is tested experimentally by driving a magnetic surgical suture needle with a length of 22 mm in the MagnetoSuture™ system along a lemniscate trajectory. The magnetic needle's tip position and the needle orientation, autonomously controlled by the proposed controller, shows RMS tracking error of 2.35 mm using typical dipole models and 1.71 mm for the Zernike fitting approach, a 27% improvement in tracking error. This suggests that the use of Zernike basis functions to capture the nonlinearities of the magnetic field may assist in implementing fast and precise autonomous control strategies for magnetic suture needles.

Real-time visual localization of needles is necessary for various surgical applications, including surgical automation and visual feedback. In this study we investigate localization and autonomous robotic control of needles in the context of our magneto-suturing system. Our system holds the potential for surgical manipulation with the benefit of minimal invasiveness and reduced patient side effects. However, the nonlinear magnetic fields produce unintuitive forces and demand delicate position-based control that exceeds the capabilities of direct human manipulation. This makes automatic needle localization a necessity. Our localization method combines neural network-based segmentation and classical techniques, and we are able to consistently locate our needle with 0.73 mm RMS error in clean environments and 2.72 mm RMS error in challenging environments with blood and occlusion. The average localization RMS error is 2.16 mm for all environments we used in the experiments. We combine this localization method with our closed-loop feedback control system to demonstrate the further applicability of localization to autonomous control. Our needle is able to follow a running suture path in (1) no blood, no tissue; (2) heavy blood, no tissue; (3) no blood, with tissue; and (4) heavy blood, with tissue environments. The tip position tracking error ranges from 2.6 mm to 3.7 mm RMS, opening the door towards autonomous suturing tasks.

We propose a system for visually monitoring and servoing the cutting of a multi-layer insulation (MLI) blanket that covers the envelope of satellites and spacecraft. The main contributions of this paper are: 1) to propose a model for relating visual features describing the engagement depth of the blade to the force exerted on the MLI blanket by the cutting tool, 2) a blade design and algorithm to reliably detect the engagement depth of the blade inside the MLI, and 3) a servoing mechanism to achieve the desired applied force by monitoring the engagement depth. We present results that validate these contributions by comparing forces estimated from visual feedback to measured forces at the blade. We also demonstrate the robustness of the blade design and vision processing under challenging conditions.

Ground-based teleoperation of robot manipulators for on-orbit servicing of spacecraft represents an example of high-payoff, high-risk operations that are challenging to perform due to high latency communications, with telemetry time delays of several seconds. In these scenarios, confidence of operating without failure is paramount. We report the development of an Interactive Planning and Supervised Execution (IPSE) system that takes advantage of accurate 3D reconstruction of the remote environment to enable operators to plan motions in the virtual world, evaluate and adjust the plan, and then supervise execution with the ability to pause and return to the planning environment at any time. We report the results of an experimental evaluation of a representative on-orbit telerobotic servicing task from NASA's upcoming OSAM-1 mission to refuel a satellite in low earth orbit; specifically, to change the robot tool to acquire the fuel supply line and then to insert it into the satellite fill/drain valve. Results of a pilot study show that the operators preferred, and were more successful with, the IPSE system when compared to a conventional teleoperation implementation.

On-orbit servicing of satellites is complicated by the fact that almost all existing satellites were not designed to be serviced. This creates a number of challenges, one of which is to cut and partially remove the protective thermal blanketing that encases a satellite prior to performing the servicing operation. A human operator on Earth can perform this task telerobotically, but must overcome difficulties presented by the multi-second round-trip telemetry delay between the satellite and the operator and the limited, or even obstructed, views from the available cameras.This paper reports the results of ground-based experiments with trained NASA robot teleoperators to compare our recently-reported augmented virtuality visualization to the conventional camera-based visualization. We also compare the master console of a da Vinci surgical robot to the conventional teleoperation interface. The results show that, for the cutting task, the augmented virtuality visualization can improve operator performance compared to the conventional visualization, but that operators are more proficient with the conventional control interface than with the da Vinci master console.