Teleoperation of robots in space is challenging due to high latency and limited workspace visibility. Previously, the Interactive Planning and Supervised Execution (IPSE) and Augmented Virtuality systems were developed to reduce failure risk. These tools were visualized on a 3D da Vinci surgical console and operated using the da Vinci manipulators or visualized on conventional monitors and operated with a keyboard and mouse. Experimental studies indicated operator preference for the latter. In this work, we develop a 3D virtual reality (VR) interface for IPSE, implemented on a Meta Quest 2 head-mounted display (HMD), and evaluate it against the prior 2D, keyboard-and-mouse-based interface. The results demonstrate improved operator load with the 3D VR interface, with no decrease in task performance, while also providing cost and portability benefits compared to the conventional 2D interface.

Robot-Assisted systems for Minimally Invasive Surgery enhance the surgeon capability, however, direct control over both the surgical tools and the endoscope results in an increased workload that leads to longer operation times. This work investigates the introduction of SCAN (System for Camera Autonomous Navigation) to overcome this limitation. An experimental study involving 12 participants was carried out with the da Vinci Research Kit. Each user tested two novel camera control modalities, autonomous and semi-autonomous, as well as the current manual control of the camera, while carrying out a dry-lab task. Among the camera control modalities, the autonomous navigation achieved better objective performances and the highest user confidence. Moreover, the autonomous control (along with the semi-autonomous one) was able to optimize some metrics related to the robotic surgery workflow.

Ground-based teleoperation of robot manipulators for on-orbit servicing of spacecraft represents an example of high-payoff, high-risk operations that are challenging to perform due to high latency communications, with telemetry time delays of several seconds. In these scenarios, confidence of operating without failure is paramount. We report the development of an Interactive Planning and Supervised Execution (IPSE) system that takes advantage of accurate 3D reconstruction of the remote environment to enable operators to plan motions in the virtual world, evaluate and adjust the plan, and then supervise execution with the ability to pause and return to the planning environment at any time. We report the results of an experimental evaluation of a representative on-orbit telerobotic servicing task from NASA's upcoming OSAM-1 mission to refuel a satellite in low earth orbit; specifically, to change the robot tool to acquire the fuel supply line and then to insert it into the satellite fill/drain valve. Results of a pilot study show that the operators preferred, and were more successful with, the IPSE system when compared to a conventional teleoperation implementation.

In adult laparoscopy, robot-aided surgery is a reality in thousands of operating rooms worldwide, owing to the increased dexterity provided by the robotic tools. Many robots and robot control techniques have been developed to aid in more challenging scenarios, such as pediatric surgery and microsurgery. However, the prevalence of case-specific solutions, particularly those focused on non-redundant robots, reduces the reproducibility of the initial results in more challenging scenarios. In this paper, we propose a general framework for the control of surgical robotics in constrained workspaces under teleoperation, regardless of the robot geometry. Our technique is divided into a slave-side constrained optimization algorithm, which provides virtual fixtures, and with Cartesian impedance on the master side to provide force feedback. Experiments with two robotic systems, one redundant and one non-redundant, show that smooth teleoperation can be achieved in adult laparoscopy and infant surgery.

On-orbit servicing of satellites is complicated by the fact that almost all existing satellites were not designed to be serviced. This creates a number of challenges, one of which is to cut and partially remove the protective thermal blanketing that encases a satellite prior to performing the servicing operation. A human operator on Earth can perform this task telerobotically, but must overcome difficulties presented by the multi-second round-trip telemetry delay between the satellite and the operator and the limited, or even obstructed, views from the available cameras.This paper reports the results of ground-based experiments with trained NASA robot teleoperators to compare our recently-reported augmented virtuality visualization to the conventional camera-based visualization. We also compare the master console of a da Vinci surgical robot to the conventional teleoperation interface. The results show that, for the cutting task, the augmented virtuality visualization can improve operator performance compared to the conventional visualization, but that operators are more proficient with the conventional control interface than with the da Vinci master console.

In robotic-assisted laparoscopic surgery, the first assistant (FA) stands at the bedside assisting the intervention, while the surgeon sits at the console teleoperating the robot. Tasks for the FA include navigating new instruments into the surgeon's field-of-view and passing in or retracting materials from the body using hand-held tools. We previously developed ARssist, an augmented reality application based on an optical see-through head-mounted display, to aid the FA. In this paper, we refine the system and first perform a pilot study with three experienced surgeons for two specific tasks: instrument insertion and tool manipulation. The results suggest that ARssist would be especially useful for less experienced assistants and for difficult hand-eye configurations. We then perform a multi-user study with inexperienced subjects. The results show that ARssist can reduce navigation time by 34.57%, enhance insertion path consistency by 41.74%, reduce root-mean-square path deviation by 40.04%, and reduce tool manipulation time by 72.25%. Thus, ARssist has the potential to improve efficiency, safety and hand-eye coordination, especially for novice assistants.

Robot control algorithms often rely on measurements of robot joint velocities, which can be estimated by measuring the time between encoder edges. When encoder edges occur infrequently, such as at low velocities and/or with low resolution encoders, this measurement delay may affect the stability of closed-loop control. This is evident in both the joint position control and Cartesian impedance control of the da Vinci Research Kit (dVRK), which contains several low-resolution encoders. We present a hardware-based method that gives more frequent velocity updates and is not affected by common encoder imperfections such as non-uniform duty cycles and quadrature phase error. The proposed method measures the time between consecutive edges of the same type but, unlike prior methods, is implemented for the rising and falling edges of both channels. Additionally, it estimates acceleration to enable software compensation of the measurement delay. The method is shown to improve Cartesian impedance control of the dVRK.

Suturing is a challenging and highly dexterous task in minimally invasive surgery, even with the assistance of robotic surgical systems. In this work, we propose a simple yet versatile impedance virtual fixture framework, which can be applied on the master manipulator in a tele-operated robotic surgical system. With this framework, we further develop two types of virtual fixtures that assist with the needle passing and knot tying sub-tasks in suturing. The paper also presents the results of a 14-participant user study for both needle passing and knot tying sub-tasks, showing that virtual fixture assistance for novice users increases the needle passing exit point accuracy, reduces the number of errors (suture slip) in knot tying, and simultaneously decreases the task completion time and overall operator workload.

We present a telerobotics research platform that provides complete access to all levels of control via open-source electronics and software. The electronics employs an FPGA to enable a centralized computation and distributed I/O architecture in which all control computations are implemented in a familiar development environment (Linux PC) and low-latency I/O is performed over an IEEE-1394a (FireWire) bus at speeds up to 400 Mbits/sec. The mechanical components are obtained from retired first-generation da Vinci ® Surgical Systems. This system is currently installed at 11 research institutions, with additional installations underway, thereby creating a research community around a common open-source hardware and software platform.

Component-based software has been proposed as a methodology for improving software reuse and has increasingly been adopted by robot software developers. At the same time, robot systems typically have real-time performance requirements and performance gains can often be obtained by multi-threading. It is challenging, however, to create correct multi-threaded software, especially when standard mutual exclusion primitives, such as mutexes and semaphores, are eschewed in favor of more efficient, lock-free mechanisms. It is even more difficult to find these errors, as they can remain dormant for years until triggered by just the “right” conditions. Our approach, therefore, is to apply Formal Methods to reason about the correctness of these mechanisms. As a first step, we adopted a recently-developed program logic called History for Local Rely/Guarantee (HLRG) and applied it to prove the correctness (after first finding and fixing an error) of one such mechanism in the open source cisst software package. This strategy is not specific to cisst and can be applied to other packages.

This paper presents an augmented reality framework, implemented on the master console of a modified da Vinci® surgical robot, that enables the operator to design and implement assistive virtual fixtures during teleoperation. Our specific goal is to facilitate teleoperation with large time delays, such as the delay of several seconds that occurs with ground-based control of robotic systems in earth orbit. The virtual fixtures give immediate visual feedback and motion guidance to the operator, while the remote slave performs motions consistent with those constraints. This approach is suitable for tasks in unstructured environments, such as servicing of existing on-orbit spacecraft that were not designed for servicing. We conducted a pilot study by teleoperating a remote slave robot for a thermal barrier blanket cutting task using virtual fixtures with and without time delay. The results show that virtual fixtures reduce the time required to complete the task while also eliminating significant manipulation errors, such as tearing the blanket. The improvement in performance is especially dramatic when a simulated time delay (4 seconds) is introduced.

While a robot control framework generally focuses on real-time performance and efficient data exchange between cooperating tasks or processes, an application such as robot-assisted surgery often demands information from, and integration with, a number of other devices. Thus, the software framework for the integrated system may have different requirements and priorities than a framework for real-time robot control. This paper reports on a component-based architecture that seamlessly bridges the gap between real-time robot control and a distributed, integrated system. The starting point is the cisst library, which provides a component-based framework for lock-free and efficient data exchange between multiple threads within a single process, which is suitable for real-time robot control. This paper describes the extension of the cisst library to support distributed systems, while keeping the same programming model as the single-process, multi-threaded scenario. Thus, application software does not need to know whether the component providing services is within the same process, in a different process, or on a different computer. In comparison, most standard middleware packages support components that fall within the last two categories (different processes on the same computer or different computers). This does not allow them to take advantage of the higher performance that can be achieved using standard lock-free data structures that do not rely on the operating system or on middleware services. Thus, the novelty of this approach is that the same component-based architecture and associated programming model extends from a multi-threaded scenario (which provides the best real-time performance) to a standard multi-process distributed system.

Robots are increasingly being used in computer integrated surgery (CIS) systems, yet to our knowledge, there is no open source software that is specifically targeted at this application domain. In this paper, we derive unique requirements for medical robot controllers that are based on our experiences in the field. We present an overview of the second-generation open source software package (the cisst package), currently under development, that would include a family of application frameworks with dynamically loaded software components. We then focus on some key design details, which include extensive application of the command pattern as well as a dynamic interface query mechanism, and present a simple example to illustrate the concepts

We describe models and algorithms designed to produce efficient and physically consistent dynamic simulations. These models and algorithms have been implemented in a unique framework, modeling both deformations and contacts through visco-elastic relations. Since this model of interaction (known as "penalty based") is much debated, we present it in detail. The "penalty" based model is said to have two major drawbacks: the difficulty in determining viscoelastic parameters, and in choosing the computation time step. We present a solution for both problems based on physical concepts. Finally, we show results comparing real data, "impulse" based simulation and "penalty" based simulation.

We describe models and algorithms designed to produce efficient and physically consistent dynamic simulations. These models and algorithms have been implemented in a unique framework modeling both deformations and contacts through visco-elastic relations. Since this model of interaction (known as "penalty based") is much debated, we present a detailed study of this model. Indeed, the "penalty" based model is supposed to have two major drawbacks: determining the visco-elastic parameters and choosing the computation time step. We present a solution for both problems based on physical concepts. Finally, we present results comparing real data, "impulse" based simulation and "penalty" based simulation.