The use of robots in high-risk and extreme environments is crucial for tasks that are dangerous or inaccessible to humans and require high precision. Particularly in scenarios where the cost of failure is high, remote human teleoperation can be the preferred method of robot control due to the adaptability and high-level decision making of humans. Teleoperation brings many challenges including lack of accurate prior knowledge about the environment, limited views of the environment by on-board sensors, and especially inconsistent latency. 7-DOF (degrees of freedom) manipulators provide redundancy which can be utilized for increased flexibility in manipulation, and may be preferred to 6-DOF manipulators in many scenarios. The redundancy, however, must be considered by the teleoperation system. We present an extension to an existing Interactive Planning and Supervised Execution (IPSE) system that enables full teleoperation of a 7-DOF robot by encoding the redundant degree of freedom with a Shoulder-Elbow-Wrist (SEW) angle, which is user-manipulable via an SEW angle graph. Additionally, we introduce a novel user interface feature that encodes robot state information into a 2D image which is displayed directly on the SEW angle graph. We conduct a user-study which demonstrates that the addition of this SEW graph significantly reduces task completion time.

Accurate and efficient surgical robotic tool pose estimation is of fundamental significance to downstream applications such as augmented reality (AR) in surgical training and learning-based autonomous manipulation. While significant advancements have been made in pose estimation for humans and animals, it is still a challenge in surgical robotics due to the scarcity of published data. The relatively large absolute error of the da Vinci end effector kinematics and arduous calibration procedure make calibrated kinematics data collection expensive. Driven by this limitation, we collected a dataset, dubbed SurgPose, providing instance-aware semantic keypoints for visual surgical tool pose estimation and tracking. By marking keypoints using ultraviolet (UV) reactive paint, which is invisible under white light and fluorescent under UV light, we execute the same trajectory under different lighting conditions to collect raw videos and keypoint annotations, respectively. The SurgPose dataset consists of approximately 120 K surgical instrument instances of 6 categories as shown in Fig. 1. Since the videos are collected in stereo pairs, the 2D pose can be lifted to 3D based on stereo-matching depth. In addition to releasing the dataset, we tested a few baseline approaches to surgical instrument tracking to demonstrate the utility of SurgPose. More details can be found at surgpose.github.io.

Data scarcity has long been an issue in the robot learning community. Particularly, in safety-critical domains like surgical applications, obtaining high-quality data can be especially difficult. It poses challenges to researchers seeking to exploit recent advancements in reinforcement learning and imitation learning, which have greatly improved generalizability and enabled robots to conduct tasks autonomously. We introduce dARt Vinci, a scalable data collection platform for robot learning in surgical settings. The system uses Augmented Reality (AR) hand tracking and a high-fidelity physics engine to capture subtle maneuvers in primitive surgical tasks: By eliminating the need for a physical robot setup and providing flexibility in terms of time, space, and hardware resources-such as multiview sensors and actuators-specialized simulation is a viable alternative. At the same time, AR allows the robot data collection to be more egocentric, supported by its body tracking and content overlaying capabilities. Our user study confirms the proposed system’s efficiency and usability, where we use widely-used primitive tasks for training teleoperation with da Vinci surgical robots. Data throughput improves across all tasks compared to real robot settings by 41% on average. The total experiment time is reduced by an average of 10%. The temporal demand in the task load survey is improved. These gains are statistically significant. Additionally, the collected data is over 400 times smaller in size, requiring far less storage while achieving double the frequency. The source code for this project can be accessed at https://dartvinci.finite-state.com/.

Automation in surgical robotics has the potential to improve patient safety and surgical efficiency, but it is difficult to achieve due to the need for robust perception algorithms. In particular, 6D pose estimation of surgical instruments is critical to enable the automatic execution of surgical maneuvers based on visual feedback. In recent years, supervised deep learning algorithms have shown increasingly better performance at 6D pose estimation tasks; yet, their success depends on the availability of large amounts of annotated data. In household and industrial settings, synthetic data, generated with 3D computer graphics software, has been shown as an alternative to minimize annotation costs of 6D pose datasets. However, this strategy does not translate well to surgical domains as commercial graphics software have limited tools to generate images depicting realistic instrument-tissue interactions. To address these limitations, we propose an improved simulation environment for surgical robotics that enables the automatic generation of large and diverse datasets for 6D pose estimation of surgical instruments. Among the improvements, we developed an automated data generation pipeline and an improved surgical scene. To show the applicability of our system, we generated a dataset of 7.5k images with pose annotations of a surgical needle that was used to evaluate a state-of-the-art pose estimation network. The trained model obtained a mean translational error of 2.59mm on a challenging dataset that presented varying levels of occlusion. These results highlight our pipeline's success in training and evaluating novel vision algorithms for surgical robotics applications.

Skull base surgery is a demanding field in which surgeons operate in and around the skull while avoiding critical anatomical structures including nerves and vasculature. While image-guided surgical navigation is the prevailing standard, limitation still exists requiring personalized planning and recognizing the irreplaceable role of a skilled surgeon. This paper presents a collaboratively controlled robotic system tailored for assisted drilling in skull base surgery. Our central hypothesis posits that this collaborative system, enriched with haptic assistive modes to enforce virtual fixtures, holds the potential to significantly enhance surgical safety, streamline efficiency, and alleviate the physical demands on the surgeon. The paper describes the intricate system development work required to enable these virtual fixtures through haptic assistive modes. To validate our system’s performance and effectiveness, we conducted initial feasibility experiments involving a medical student and two experienced surgeons. The experiment focused on drilling around critical structures following cortical mastoidectomy, utilizing dental stone phantom and cadaveric models. Our experimental results demonstrate that our proposed haptic feedback mechanism enhances the safety of drilling around critical structures compared to systems lacking haptic assistance. With the aid of our system, surgeons were able to safely skeletonize the critical structures without breaching any critical structure even under obstructed view of the surgical site.

Despite advancements in robotic-assisted surgery, automating complex tasks like suturing remains challenging due to the need for adaptability and precision. Learning-based approaches, particularly reinforcement learning (RL) and imitation learning (IL), require realistic simulation environments for efficient data collection. However, current platforms often include only relatively simple, non-dexterous manipulations and lack the flexibility required for effective learning and generalization. We introduce SurgicAI, a novel platform for development and benchmarking that addresses these challenges by providing the flexibility to accommodate both modular subtasks and more importantly task decomposition in RL-based surgical robotics. Compatible with the da Vinci Surgical System, SurgicAI offers a standardized pipeline for collecting and utilizing expert demonstrations. It supports the deployment of multiple RL and IL approaches, and the training of both singular and compositional subtasks in suturing scenarios, featuring high dexterity and modularization. Meanwhile, SurgicAI sets clear metrics and benchmarks for the assessment of learned policies. We implemented and evaluated multiple RL and IL algorithms on SurgicAI. Our detailed benchmark analysis underscores SurgicAI's potential to advance policy learning in surgical robotics. Details: https://github.com/surgical-robotics-ai/SurgicAI

Teleoperation of robots in space is challenging due to high latency and limited workspace visibility. Previously, the Interactive Planning and Supervised Execution (IPSE) and Augmented Virtuality systems were developed to reduce failure risk. These tools were visualized on a 3D da Vinci surgical console and operated using the da Vinci manipulators or visualized on conventional monitors and operated with a keyboard and mouse. Experimental studies indicated operator preference for the latter. In this work, we develop a 3D virtual reality (VR) interface for IPSE, implemented on a Meta Quest 2 head-mounted display (HMD), and evaluate it against the prior 2D, keyboard-and-mouse-based interface. The results demonstrate improved operator load with the 3D VR interface, with no decrease in task performance, while also providing cost and portability benefits compared to the conventional 2D interface.

The introduction of image-guided surgical navigation (IGSN) has greatly benefited technically demanding surgical procedures by providing real-time support and guidance to the surgeon during surgery. To develop effective IGSN, a careful selection of the surgical information and the medium to present this information to the surgeon is needed. However, this is not a trivial task due to the broad array of available options. To address this problem, we have developed an open-source library that facilitates the development of multimodal navigation systems in a wide range of surgical procedures relying on medical imaging data. To provide guidance, our system calculates the minimum distance between the surgical instrument and the anatomy and then presents this information to the user through different mechanisms. The real-time performance of our approach is achieved by calculating Signed Distance Fields at initialization from segmented anatomical volumes. Using this framework, we developed a multimodal surgical navigation system to help surgeons navigate anatomical variability in a skull base surgery simulation environment. Three different feedback modalities were explored: visual, auditory, and haptic. To evaluate the proposed system, a pilot user study was conducted in which four clinicians performed mastoidectomy procedures with and without guidance. Each condition was assessed using objective performance and subjective workload metrics. This pilot user study showed improvements in procedural safety without additional time or workload. These results demonstrate our pipeline's successful use case in the context of mastoidectomy.

Telesurgery has a clear potential for providing high-quality surgery to medically underserved areas like rural areas, battlefields, and spacecraft; nevertheless, effective methods to overcome unreliable communication systems are still lacking. Furthermore, it is not well understood how users react at the moment of communication loss and also during the loss. In this paper, we aim to analyze human response by proposing a telesurgery simulation framework that models an environment incorporating local and remote sites. Furthermore, this framework generates structural data for human behavior analysis and can provide different forms of assistance during the communication failure and at the communication recovery. We investigated three different types of assistance: User-centered, Robot-centered and Hybrid. A 12-person user-study was carried out using the proposed telesurgery simulation where participants completed a peg transfer task with random communication loss. The collected data was used to analyze the human response to a communication failure. The proposed Hybrid method reduced temporal demand with no increase in completion time compared to the baseline control method where users were unable to move the input device during the communication loss. The Hybrid method also significantly reduced both the task completion time and workload compared to the other two proposed methods (User-centered and Robot-centered).

Studies of the human brain during natural activities, such as locomotion, would benefit from the ability to image deep brain structures during these activities. While Positron Emission Tomography (PET) can image these structures, the bulk and weight of current scanners are not compatible with the desire for a wearable device. This has motivated the design of a robotic system to support a PET imaging system around the subject's head and to move the system to accommodate natural motion. We report here the design and experimental evaluation of a prototype robotic system that senses motion of a subject's head, using parallel string encoders connected between the robot-supported imaging ring and a helmet worn by the subject. This measurement is used to robotically move the imaging ring (coarse motion correction) and to compensate for residual motion during image reconstruction (fine motion correction). Minimization of latency and measurement error are the key design goals, respectively, for coarse and fine motion correction. The system is evaluated using recorded human head motions during locomotion, with a mock imaging system consisting of lasers and cameras, and is shown to provide an overall system latency of about 80 ms, which is sufficient for coarse motion correction and collision avoidance, as well as a measurement accuracy of about 0.5 mm for fine motion correction.

Stress, anxiety, and depression negatively affect productivity and the global economy with an estimated annual cost of ${\$}$1 trillion U.S. dollars, according to the World Health Organization. Moreover, prolonged daily stress—even if minor—can lead to severe health consequences, including cancer and various mental disorders. Virtual reality (VR) has been shown to be a promising tool for relieving daily stressors given its accessibility and its projected availability as compared to visiting with mental health professionals. Prior work in this area has mostly focused on the restorative effects of nature simulations, demonstrating that passively experiencing immersive nature scenes improves positive affect. However, aside from providing opportunities for exercise, little is known about how active VR engagement can improve one’s mental health. To address this research gap, this paper presents a new, active form of VR therapy and assesses its effectiveness as compared to passive VR experiences. We developed VR Drawing—inspired by art therapy, which promotes positive emotions through artistic creation—and VR Throwing—inspired by “rage rooms”, which allow people to release negative emotions via intentional destruction. In a between- participants study (n = 64), we found that both VR Drawing and VR Throwing significantly reduced participants’ stress levels and increased positive affect when compared to passively watching nature scenes in VR. Linear regression models suggest that the total number of user interactions positively affects improvement in positive emotions for VR Drawing, but has a negative impact on positive emotions for VR Throwing. This study provides empirical evidence of how active VR experiences may reduce stress and offers guidelines for creating future VR applications to promote psychological well-being.

Head-mounted loupes can increase the user's visual acuity to observe the details of an object. On the other hand, optical see-through head-mounted displays (OST-HMD) are able to provide virtual augmentations registered with real objects. In this article, we propose AR-Loupe, combining the advantages of loupes and OST-HMDs, to offer augmented reality in the user's magnified field-of-vision. Specifically, AR-Loupe integrates a commercial OST-HMD, Magic Leap One, and binocular Galilean magnifying loupes, with customized 3D-printed attachments. We model the combination of user's eye, screen of OST-HMD, and the optical loupe as a pinhole camera. The calibration of AR-Loupe involves interactive view segmentation and an adapted version of stereo single point active alignment method (Stereo-SPAAM). We conducted a two-phase multi-user study to evaluate AR-Loupe. The users were able to achieve sub-millimeter accuracy ( 0.82 mm) on average, which is significantly ( ) smaller compared to normal AR guidance ( 1.49 mm). The mean calibration time was 268.46 s. With the increased size of real objects through optical magnification and the registered augmentation, AR-Loupe can aid users in high-precision tasks with better visual acuity and higher accuracy.

The da Vinci Research Kit (dVRK) was introduced in 2012 to provide an affordable, open-source platform for research in robotic minimally-invasive surgery. It provides access to all levels of control but, until now, has relied on an analog controller for the motor current, which cannot easily be customized to improve performance. This paper aims to implement the low-level control digitally and to improve the overall control performance. To enable model-based controller design, the system is first identified using measurements provided by the encoders and internal electronics. The digital current controller is then implemented on the existing field programmable gate array (FPGA). Experiments demonstrate that the new digital current controller yields superior performance compared to the original analog design. In addition, the identified system model is used to design an improved position controller that is also implemented on the FPGA and provides better trajectory tracking than the position controller currently implemented on the control PC. The comparison between simulation and measurement, for both the current and position control, verifies the validity of the model based on the system identification, enabling utilization for future adaptations. The improved low-level control enlarges the possibilities for more accurate operation and the achieved digital implementation enables researchers worldwide to easily adapt the low-level control in future versions of the dVRK.

Teleoperation platforms often require the user to be situated at a fixed location to both visualize and control the movement of the robot and thus do not provide the operator with much mobility. One example is in existing robotic surgery solutions that require the surgeons to be away from the patient, attached to consoles where their heads must be fixed and their arms can only move in a limited space. This creates a barrier between physicians and patients that does not exist in normal surgery. To address this issue, we propose a mobile telesurgery solution where the surgeons are no longer mechanically limited to control consoles and are able to teleoperate the robots from the patient bedside, using their arms equipped with wireless sensors and viewing the endoscope video via optical see-through head-mounted displays (HMDs). We evaluate the feasibility and efficiency of our user interaction method compared to a standard surgical robotic manipulator via two tasks with different levels of required dexterity. The results indicate that with sufficient training our proposed platform can attain similar efficiency while providing added mobility for the operator.

Most current surgical robotic systems lack the ability to sense tool/tissue interaction forces, which motivates research in methods to estimate these forces from other available measurements, primarily joint torques. These methods require the internal joint torques, due to the robot inverse dynamics, to be subtracted from the measured joint torques. This paper presents the use of neural networks to estimate the inverse dynamics of the da Vinci surgical robot, which enables estimation of the external environment forces. Experiments with motions in free space demonstrate that the neural networks can estimate the internal joint torques within 10% normalized rootmean-square error (NRMSE), which outperforms model-based approaches in the literature. Comparison with an external force sensor shows that the method is able to estimate environment forces within about 10% NRMSE.

Robot-Assisted systems for Minimally Invasive Surgery enhance the surgeon capability, however, direct control over both the surgical tools and the endoscope results in an increased workload that leads to longer operation times. This work investigates the introduction of SCAN (System for Camera Autonomous Navigation) to overcome this limitation. An experimental study involving 12 participants was carried out with the da Vinci Research Kit. Each user tested two novel camera control modalities, autonomous and semi-autonomous, as well as the current manual control of the camera, while carrying out a dry-lab task. Among the camera control modalities, the autonomous navigation achieved better objective performances and the highest user confidence. Moreover, the autonomous control (along with the semi-autonomous one) was able to optimize some metrics related to the robotic surgery workflow.

We propose a system for visually monitoring and servoing the cutting of a multi-layer insulation (MLI) blanket that covers the envelope of satellites and spacecraft. The main contributions of this paper are: 1) to propose a model for relating visual features describing the engagement depth of the blade to the force exerted on the MLI blanket by the cutting tool, 2) a blade design and algorithm to reliably detect the engagement depth of the blade inside the MLI, and 3) a servoing mechanism to achieve the desired applied force by monitoring the engagement depth. We present results that validate these contributions by comparing forces estimated from visual feedback to measured forces at the blade. We also demonstrate the robustness of the blade design and vision processing under challenging conditions.

Ground-based teleoperation of robot manipulators for on-orbit servicing of spacecraft represents an example of high-payoff, high-risk operations that are challenging to perform due to high latency communications, with telemetry time delays of several seconds. In these scenarios, confidence of operating without failure is paramount. We report the development of an Interactive Planning and Supervised Execution (IPSE) system that takes advantage of accurate 3D reconstruction of the remote environment to enable operators to plan motions in the virtual world, evaluate and adjust the plan, and then supervise execution with the ability to pause and return to the planning environment at any time. We report the results of an experimental evaluation of a representative on-orbit telerobotic servicing task from NASA's upcoming OSAM-1 mission to refuel a satellite in low earth orbit; specifically, to change the robot tool to acquire the fuel supply line and then to insert it into the satellite fill/drain valve. Results of a pilot study show that the operators preferred, and were more successful with, the IPSE system when compared to a conventional teleoperation implementation.

A flexible endoscope introduces more dexterity to the image capturing in endoscopic surgery. However, manual control or automatic control based on instrument tracking does not handle the misorientation between the endoscopic video and the surgeon. We propose an automatic flexible endoscope control method that tracks the surgeon's head with respect to the object in the surgical scene. The robotic flexible endoscope is actuated so that it captures the surgical scene from the same perspective as the surgeon. The surgeon wears a head-mounted display to observe the endoscopic video. The frustum of the flexible endoscope is rendered as an augmented reality overlay to provide surgical guidance. We developed the prototype, FlexiVision, integrating a 6-DOF robotic flexible endoscope based on the da Vinci Research Kit and Microsoft HoloLens. We evaluated the proposed automatic control method via a lesion observation task, and evaluated the AR surgical guidance in a lesion targeting task. The multi-user study results demonstrated that, for both tasks, FlexiVision significantly reduced the completion time (by 59% and 58%), number of errors (by 75% and 95%) and subjective task load level. With FlexiVision, the flexible endoscope could act as the surgeon's eyes teleported into the abdominal cavity of the patient.

In adult laparoscopy, robot-aided surgery is a reality in thousands of operating rooms worldwide, owing to the increased dexterity provided by the robotic tools. Many robots and robot control techniques have been developed to aid in more challenging scenarios, such as pediatric surgery and microsurgery. However, the prevalence of case-specific solutions, particularly those focused on non-redundant robots, reduces the reproducibility of the initial results in more challenging scenarios. In this paper, we propose a general framework for the control of surgical robotics in constrained workspaces under teleoperation, regardless of the robot geometry. Our technique is divided into a slave-side constrained optimization algorithm, which provides virtual fixtures, and with Cartesian impedance on the master side to provide force feedback. Experiments with two robotic systems, one redundant and one non-redundant, show that smooth teleoperation can be achieved in adult laparoscopy and infant surgery.

On-orbit servicing of satellites is complicated by the fact that almost all existing satellites were not designed to be serviced. This creates a number of challenges, one of which is to cut and partially remove the protective thermal blanketing that encases a satellite prior to performing the servicing operation. A human operator on Earth can perform this task telerobotically, but must overcome difficulties presented by the multi-second round-trip telemetry delay between the satellite and the operator and the limited, or even obstructed, views from the available cameras.This paper reports the results of ground-based experiments with trained NASA robot teleoperators to compare our recently-reported augmented virtuality visualization to the conventional camera-based visualization. We also compare the master console of a da Vinci surgical robot to the conventional teleoperation interface. The results show that, for the cutting task, the augmented virtuality visualization can improve operator performance compared to the conventional visualization, but that operators are more proficient with the conventional control interface than with the da Vinci master console.

In robotic-assisted laparoscopic surgery, the first assistant (FA) stands at the bedside assisting the intervention, while the surgeon sits at the console teleoperating the robot. Tasks for the FA include navigating new instruments into the surgeon's field-of-view and passing in or retracting materials from the body using hand-held tools. We previously developed ARssist, an augmented reality application based on an optical see-through head-mounted display, to aid the FA. In this paper, we refine the system and first perform a pilot study with three experienced surgeons for two specific tasks: instrument insertion and tool manipulation. The results suggest that ARssist would be especially useful for less experienced assistants and for difficult hand-eye configurations. We then perform a multi-user study with inexperienced subjects. The results show that ARssist can reduce navigation time by 34.57%, enhance insertion path consistency by 41.74%, reduce root-mean-square path deviation by 40.04%, and reduce tool manipulation time by 72.25%. Thus, ARssist has the potential to improve efficiency, safety and hand-eye coordination, especially for novice assistants.

Robot control algorithms often rely on measurements of robot joint velocities, which can be estimated by measuring the time between encoder edges. When encoder edges occur infrequently, such as at low velocities and/or with low resolution encoders, this measurement delay may affect the stability of closed-loop control. This is evident in both the joint position control and Cartesian impedance control of the da Vinci Research Kit (dVRK), which contains several low-resolution encoders. We present a hardware-based method that gives more frequent velocity updates and is not affected by common encoder imperfections such as non-uniform duty cycles and quadrature phase error. The proposed method measures the time between consecutive edges of the same type but, unlike prior methods, is implemented for the rising and falling edges of both channels. Additionally, it estimates acceleration to enable software compensation of the measurement delay. The method is shown to improve Cartesian impedance control of the dVRK.

One major cause of preventable death is a lack of proper skills for providing critical care. Conventional training for advanced emergency medical procedures is often limited to a verbal block of instructions and/or an instructional video. In this study, we evaluate the benefits of using an optical see-through head-mounted display (OST-HMD) for training of caregivers in an emergency medical environment. A rich user interface was implemented that provides 3D visual aids including images, text and tracked 3D overlays for each task. A user study with 20 participants was conducted for two medical tasks, where each subject received conventional training for one task and HMD training for the other task. Our results indicate that using a mixed reality HMD is more engaging, improves the time-on-task, and increases the confidence level of users.

Recent technical advancements support the application of Optical See-Through Head-Mounted Displays (OST-HMDs) in critical situations like navigation and manufacturing. However, while the form-factor of an OST-HMD occupies less of the user's visual field than in the past, it can still result in critical oversights, e.g., missing a pedestrian while driving a car. In this paper, we design and compare two methods to compensate for the loss of awareness due to the occlusion caused by OST-HMDs. Instead of presenting the occluded content to the user, we detect motion that is not visible to the user and highlight its direction either on the edge of the HMD screen, or by activating LEDs placed in the user's peripheral vision. The methods involve an offline stage, where the occluded visual field and location of each indicator and its associated occluded region of interest (OROI) are determined, and an online stage, where an enhanced optical flow algorithm tracks the motion in the occluded visual field. We have implemented both methods on a Microsoft HoloLens and an ODG R-9. Our prototype systems achieved success rates of 100% in an objective evaluation, and 98.90% in a pilot user study. Our methods are able to compensate for the loss of safety-critical information in the occluded visual field for state-of-the-art OST-HMDs and can be extended for their future generations.

One of the risks of the endonasal approach to skull base surgery is inadvertent damage to one of the two carotid arteries that are located behind the bone being drilled. Photoacoustic imaging, which combines a pulsed laser with an ultrasound receiver probe, has been shown to be able to image blood vessels behind bone. We therefore integrated a photoacoustic imaging system with a telerobotic system, where the pulsed laser is delivered via an optical fiber attached to the drill held by one robot arm and the ultrasound receiver is positioned, at some distance from the drilling site, by another robot arm. This paper describes a new method for accurately determining the safe region for drilling, which is defined by the center-line between the two carotid arteries, and presents the first phantom experiments with this system. The results show that the system can determine the center point with an accuracy better than 2 mm, which suggests that it may be sufficient for clinical scenarios where the two carotid arteries can be within 8 mm of each other.

Ground-based teleoperation of robots in space is subject to time delays of several seconds or more. This leads to the use of model-based approaches, where the operator interacts with a model (simulation) of the remote environment and the remote robot attempts to reproduce the results of that interaction. But, it is also desirable for the operator to view (delayed) images from the remote scene. These images, however, are often from one or more monocular cameras mounted on the robot end-effector, which leads to several other problems: unintuitive teleoperation due to the eye-in-hand configuration, limited field of view, and lack of stereo visualization. We present an augmented virtuality interface for teleoperation, which can solve these problems by projecting the real camera images onto a registered 3D model of the environment and allowing the operator to select any desired viewpoint. This approach is suitable when there is at least a partial model of the environment, as is the case for satellite servicing. The proposed method begins with a video survey to register the 3D model to the physical environment, followed by a user interface that presents a stereo visualization of the model, augmented by projections of real camera images onto the model. We quantitatively and qualitatively compare the augmented virtuality images to real camera images taken from the same viewpoint and perform experiments to evaluate the efficacy of the augmented virtuality paradigm for teleoperation. The results suggest that this approach can improve operator situation awareness, potentially leading to better performance, especially when the camera views are unintuitive or limited.

Uncertainty in measurement of point correspondences negatively affects the accuracy and precision in the calibration of head-mounted displays (HMD). In general, the distribution of alignment errors for optical see-through calibration are not isotropic, and one can estimate its distribution based on interaction requirements of a given calibration process and the user's measurable head motion and hand-eye coordination characteristics. Current calibration methods, however, mostly utilize the Direct Linear Transformation (DLT) method which minimizes Euclidean distances for HMD projection matrix estimation, disregarding the anisotropicity in the alignment errors. We utilize the error covariance in order to take the anisotropic nature of error distribution into account. The main hypothesis of this study is that using Mahalonobis distance within the nonlinear optimization can improve the accuracy of the HMD calibration. The simulation results indicate that our new method outperforms the standard DLT method both in accuracy and precision, and is more robust against user alignment errors. To the best of our knowledge, this is the first time that anisotropic noise has been accommodated in the optical see-through HMD calibration.

An effective and simple method is proposed for multi-camera collaborative tracking, based on the prioritization of all tracking units, and then modeling the discrepancy between different tracking units as a locally static transformation error. Static error compensation is applied to the lower-priority tracking systems when high-priority trackers are not available. The method does not require high-end or carefully calibrated tracking units, and is able to effectively provide a comfortable augmented reality experience for users. A pilot study demonstrates the validity of the proposed method.

Medical and surgical robot systems are examples of safety-critical systems due to the potential hazards that can lead to severe injury or the loss of human life. Furthermore, robot safety is becoming increasingly important in other domains as robots begin to share their workspace with humans. At the same time, the complexity of robot hardware and software has been increasing to endow them with capabilities for safe human-robot collaboration and other demanding tasks. While component-based frameworks have been widely adopted in robotics to manage the increasing scale and complexity of these systems, it is still challenging for both academic researchers and industrial developers to develop robot systems with safety in a reusable and structured manner. To address these issues, we reformulate safety as a visible, reusable, and verifiable property, rather than an embedded, hard-to-reuse, and hard-to-test property that is tightly coupled with the system. Specifically, we present a state-based model and a safety-oriented software architecture that allow us to reuse the design of safety features across different systems and different applications without relying on a particular component model. To demonstrate and evaluate the proposed methods, we built SAFECASS, an open source software framework that enables reuse of experience and knowledge on safety, and applied it to two medical robot systems, one of which is presented in this paper.

Model-based teleoperation is suitable for systems with large communication delay because the operator interacts with a model of the task while the remote robot uses sensor-based control to replicate that interaction on the physical system. When the model geometry is accurately known, it is only necessary to register it to the remote physical system. If registration errors can be detected during the task, it is possible to update the local task frame (as in model mediated teleoperation) or the remote task frame. This paper proposes the latter approach for the case of telerobotic satellite servicing where the remote (on-orbit) robot cuts the tape that secures the patch of insulation covering the satellite access panel. This task can be modeled as sliding a tool along a planar surface (or one that is locally planar). The remote task frame is used by a hybrid position/force controller to maintain contact with the planar surface. Registration errors, however, can affect the orientation of the cutting tool and cause cutting failures. Therefore, the registration is updated during the task by using position measurements under the effect of hybrid control. The contributions of this paper are in the application of this method to teleoperation, where it must handle user-specified motions, and in the experimental verification during cutting, where compliance of the environment must be estimated and taken into account.

Suturing is a challenging and highly dexterous task in minimally invasive surgery, even with the assistance of robotic surgical systems. In this work, we propose a simple yet versatile impedance virtual fixture framework, which can be applied on the master manipulator in a tele-operated robotic surgical system. With this framework, we further develop two types of virtual fixtures that assist with the needle passing and knot tying sub-tasks in suturing. The paper also presents the results of a 14-participant user study for both needle passing and knot tying sub-tasks, showing that virtual fixture assistance for novice users increases the needle passing exit point accuracy, reduces the number of errors (suture slip) in knot tying, and simultaneously decreases the task completion time and overall operator workload.

In this paper, a real-time control network based on Ethernet and FireWire is presented, where Ethernet provides a convenient, cross-platform interface between a central control PC and a FireWire subnetwork that contains multiple distributed nodes (I/O boards). Real-time performance is achieved because this architecture limits the number of Ethernet transactions on the host PC, benefits from the availability of real-time Ethernet drivers, and uses the broadcast and peer-to-peer capabilities of FireWire to efficiently transfer data among the distributed nodes. This approach and resulting benefits are comparable to EtherCAT, but preserves existing investments in FireWire-based controllers and relies only on conventional, vendor-neutral network hardware and protocols. The system performance is demonstrated on the da Vinci Research Kit (dVRK), which consists of 8 FireWire nodes that control 2 Master Tool Manipulators (MTMs) and 2 Patient Side Manipulators (PSMs), for a total of 28 axes. This approach is generally applicable to interface existing FireWire-based systems to new control PCs via Ethernet or to serve as an open-source alternative to EtherCAT for new designs.

We are investigating the use of photoacoustic (PA) imaging to detect critical structures, such as the carotid artery, that may be located behind the bone being drilled during robot-assisted endonasal transsphenoidal surgery. In this system, the laser is mounted on the drill (via an optical fiber) and the 2D ultrasound (US) probe is placed elsewhere on the skull. Both the drill and the US probe are tracked relative to the patient reference frame. PA imaging provides two advantages compared to conventional B-mode US: (1) the laser penetrates thin layers of bone, and (2) the PA image displays targets that are in the laser path. Thus, the laser can be used to (non-invasively) extend the drill axis, thereby enabling reliable detection of critical structures that may reside in the drill path. This setup creates a challenging alignment problem, however, because the US probe must be placed so that its image plane intersects the laser line in the neighborhood of the target anatomy (as estimated from preoperative images). This paper reports on a navigation system developed to assist with this task, and the results of phantom experiments that demonstrate that a critical structure can be detected with an accuracy of approximately 1 mm relative to the drill tip.

The ability to refuel satellites on-orbit using teleoperated robots could extend the life of satellite missions, potentially saving time and resources for both industry and space agencies. One critical step in the refueling process is to gain access to the satellite's fueling port by cutting through a seam of tape that adheres two sections of multi-layer insulation (MLI) covering the satellite body. The deformable, delicate, specular nature of the metalized tape used to adhere sections of MLI to one another makes cutting the tape seams a difficult task to perform remotely, a difficulty that is compounded by unavoidable multi-second communications delays. Virtual-fixture-assisted teleoperation can help mitigate some of these issues by constraining the motion of the cutting blade based on a priori knowledge of the task. Adding force control may further assist teleoperators by automatically setting a desired normal cutting force. This paper presents the results of a 20-participant user study for a representative satellite tape-cutting task, showing that force control based on a virtual fixture mapped to the MLI blanket plane can decrease both tape bunching and operator workload.

For satellite servicing, it is necessary to remove a patch of multi-layer insulation (MLI) that covers the access panel. We consider the case where this patch is secured by tape and desire to use ground-based teleoperation to carefully cut the tape on three sides of the patch. Communication delays of several seconds motivate the development of an online method to enable failure detection by the remote (on-orbit) robot system, so that cutting can be stopped without having to wait several seconds for the ground-based operator to observe the failure. This method is based on a model that predicts the force in the direction of cutting. The model parameters are provided by a recursive least squares estimator, with vector-like forgetting factors, that also includes a throttling mechanism to ensure that the estimator is used only when operating conditions and measurements enable reasonable outcomes. During cutting, the predicted force is compared to the measured force to detect various types of failures. Experiments are conducted on a ground-based platform to demonstrate that the proposed estimation system can reliably detect these failures.

Direct teleoperation with multisecond time-delayed telemetry between master and slave is challenging for humans to perform. When controlling a holonomic robot with many degrees of freedom, operators may incidentally provide commands in an intended direction without realizing their mistake until receiving feedback several seconds later. For some applications, imposing a virtual nonholonomic constraint (VNHC) on the motion of the end effector can help prevent operators from moving in an unintended direction by reducing the number of controllable degrees of freedom. This paper presents the development of a VNHC for a planar time-delayed telerobotic task, motivated by an on-orbit telerobotic satellite servicing operation. We also describe a nonholonomic virtual fixture (NHVF) that adheres to the VNHC to further reduce the potential for operators to input mistaken commands. We report the results of a pilot study in which teleoperation with a VNHC was found to have comparable task performance to holonomic planar teleoperation, while decreasing operator workload. The NHVF was found to decrease performance slightly, though user feedback indicated that a differently implemented virtual fixture and controller may improve performance.

A variety of medical and surgical robot systems have been developed in academia and industry and commercial products are actively used in modern operating rooms. However, there is no safety standard that specifically governs the design of medical robot systems. Despite the availability of several safety design guidelines, the absence of a basis or foundation for safety makes it difficult to describe safety designs in a systematic manner, and to share knowledge and experiences on safety with others. In the meantime, the scale and complexity of recent medical robot systems have been increasing and this further complicates the effective representation and sharing of safety designs. As an approach to this issue, we propose the Safety Design View, a conceptual framework that can capture and describe both the design-time and run-time characteristics of safety features of medical robot systems in a systematic and structured manner. To illustrate the application of the Safety Design View, we collected a set of frequently used safety features, based on our literature review of safety in the medical robotics domain, and show how we can more effectively describe and understand safety designs of medical robot systems.

We present a telerobotics research platform that provides complete access to all levels of control via open-source electronics and software. The electronics employs an FPGA to enable a centralized computation and distributed I/O architecture in which all control computations are implemented in a familiar development environment (Linux PC) and low-latency I/O is performed over an IEEE-1394a (FireWire) bus at speeds up to 400 Mbits/sec. The mechanical components are obtained from retired first-generation da Vinci ® Surgical Systems. This system is currently installed at 11 research institutions, with additional installations underway, thereby creating a research community around a common open-source hardware and software platform.

We applied quantified differential-dynamic logic (QdL) to analyze a control algorithm designed to provide directional force feedback for a surgical robot. We identified problems with the algorithm, proved that it was in general unsafe, and described exactly what could go wrong. We then applied QdL to guide the development of a new algorithm that provides safe operation along with directional force feedback. Using \KeYmaeraD (a tool that mechanizes QdL), we created a machine-checked proof that guarantees the new algorithm is safe for all possible inputs.

Our goal is to develop new methods for telerobotic on-orbit servicing of spacecraft under ground-based supervisory control of human operators to perform tasks in the presence of uncertainty and telemetry time delay of several seconds. We propose a new delay tolerant control methodology, using virtual fixtures, hybrid position/force control, task frame formalism, and environment modeling, that is robust to modeling and registration errors. The task model is represented by graphical primitives and virtual fixtures on the teleoperation master and by a hybrid position/force controller on the slave robot. The virtual fixtures guide the operator through a model-based simulation of the task, and the goal of the slave controller is to reproduce this action (after a few seconds of delay) or, if measurements are not consistent with the models, to stop motion and alert the operator. This approach is suitable for tasks in unstructured environments, such as servicing of existing on-orbit spacecraft that were not designed for servicing. We introduce the overall control concept, its main components, and an example application in which the remote slave robot cuts the tape that secures a flap of multi-layer insulation over the access panel of a satellite mockup.

Cooperatively controlled robots are used in many kinds of applications, including surgical robot applications where the surgeon can guide the robot end effector to a desired position. Often, a 6 degree-of-freedom (DOF) force/torque sensor is installed. However, in some cases, the sensor is only used to impose safety thresholds and to support the robot guidance task. In cases where high guidance accuracy is not required, it can be difficult to justify the added cost of a 6 DOF force sensor. One lower-cost solution is to incorporate a joystick or similar input device, but this requires additional hardware and removes the surgeon's hands from direct interaction with the robot end-effector. This paper presents a method for achieving cooperative force control without a force sensor. The method utilizes motor current feedback and uses a calibrated current value for force estimation. The novelty of this method is that it can be applied to non-backdrivable robots. It is implemented on a 2-DOF XY stage and experiments are conducted to demonstrate accuracy and performance on this non-backdrivable robot.

Image-guided radiation therapy (IGRT) involves two main procedures, performed in different rooms on different days: (1) treatment planning in the simulator room on the first day, and (2) radiotherapy in the linear accelerator room over multiple subsequent days. Both the simulator and the linear accelerator include CT imaging capabilities, which enables both treatment planning and reproducible patient setup, but does not provide good soft tissue contrast or allow monitoring of the target during treatment. We propose a cooperatively-controlled robot to reproducibly position an ultrasound (US) probe on the patient during simulation and treatment, thereby improving soft tissue visualization and allowing real-time monitoring of the target. A key goal of the robotic system is to produce consistent tissue deformations for both CT and US imaging, which simplifies registration of these two modalities. This paper presents the robotic system design and describes a novel control algorithm that employs virtual springs to implement guidance virtual fixtures during "hands on" cooperative control.

Component-based software has been proposed as a methodology for improving software reuse and has increasingly been adopted by robot software developers. At the same time, robot systems typically have real-time performance requirements and performance gains can often be obtained by multi-threading. It is challenging, however, to create correct multi-threaded software, especially when standard mutual exclusion primitives, such as mutexes and semaphores, are eschewed in favor of more efficient, lock-free mechanisms. It is even more difficult to find these errors, as they can remain dormant for years until triggered by just the “right” conditions. Our approach, therefore, is to apply Formal Methods to reason about the correctness of these mechanisms. As a first step, we adopted a recently-developed program logic called History for Local Rely/Guarantee (HLRG) and applied it to prove the correctness (after first finding and fixing an error) of one such mechanism in the open source cisst software package. This strategy is not specific to cisst and can be applied to other packages.

This paper presents an augmented reality framework, implemented on the master console of a modified da Vinci® surgical robot, that enables the operator to design and implement assistive virtual fixtures during teleoperation. Our specific goal is to facilitate teleoperation with large time delays, such as the delay of several seconds that occurs with ground-based control of robotic systems in earth orbit. The virtual fixtures give immediate visual feedback and motion guidance to the operator, while the remote slave performs motions consistent with those constraints. This approach is suitable for tasks in unstructured environments, such as servicing of existing on-orbit spacecraft that were not designed for servicing. We conducted a pilot study by teleoperating a remote slave robot for a thermal barrier blanket cutting task using virtual fixtures with and without time delay. The results show that virtual fixtures reduce the time required to complete the task while also eliminating significant manipulation errors, such as tearing the blanket. The improvement in performance is especially dramatic when a simulated time delay (4 seconds) is introduced.

Precise tumor identification is crucial in image-guided neurosurgical procedures. With existing navigation systems, the surgeon must turn away from the patient to view the imaging data on a separate monitor. In this study, an innovative system is introduced that illustrates the tumor boundaries precisely augmented on the spot where the tumor is located with regard to the patient. Additionally, it allows the surgeon to track the distal end of the tools contextually, where direct visualization is not possible. In this approach, the tracking system is compact and worn by the surgeon, eliminating the need for additional devices that are bulky and typically limited by line of sight constraints.

Surgical planning and navigation systems are vital for minimally invasive endoscopic surgeries but it is challenging to track the position and orientation of intrabody surgical instruments in these procedures. In order to address this problem, we propose a tracking system including multiple-sensor integration and data fusion. The proposed tracking approach is free of the constraints of line-of-sight, less subject to environmental distortion, and with higher update rate. By incorporating electromagnetic and inertial sensors, the system yields continuous 6-DOF information. Based on a system dynamic model and estimation theories, a new multisensor fusion algorithm, cascade orientation and position-estimation algorithm, is proposed for the integrated tracking device. The experimental results show that the proposed algorithms achieve accurate orientation and position tracking with robustness.

Intraoperative patient registration may significantly affect the outcome of image-guided surgery (IGS). Image-based registration approaches have several advantages over the currently dominant point-based direct contact methods and are used in some industry solutions in image-guided radiation therapy with fixed X-ray gantries. However, technical challenges including geometric calibration and computational cost have precluded their use with mobile C-arms for IGS. We propose a 2D/3D registration framework for intraoperative patient registration using a conventional mobile X-ray imager combining fiducial-based C-arm tracking and graphics processing unit (GPU)-acceleration. The two-stage framework 1) acquires X-ray images and estimates relative pose between the images using a custom-made in-image fiducial, and 2) estimates the patient pose using intensity-based 2D/3D registration. Experimental validations using a publicly available gold standard dataset, a plastic bone phantom and cadaveric specimens have been conducted. The mean target registration error (mTRE) was 0.34 ± 0.04 mm (success rate: 100%, registration time: 14.2 s) for the phantom with two images 90° apart, and 0.99 ± 0.41 mm (81%, 16.3 s) for the cadaveric specimen with images 58.5° apart. The experimental results showed the feasibility of the proposed registration framework as a practical alternative for IGS routines.

This paper presents a constrained optimization framework that enables the implementation of multi-robot constraints, as virtual fixtures, to assist human operators, in a teleoperated scenario. The collaborative constraints guide the motion of multiple robots such that the spatial and temporal relationships are maintained between them, while following human input motion objectives. We demonstrate this control architecture for the task of manipulating a surgical knot to a target point. The teleoperation system uses four arms from a da Vinci Surgical System® (two master manipulators and two slave manipulators), with custom electronics and software. It extends previous work, which focused on a cooperatively controlled system where the motions of two robots were directly controlled by user-applied forces. Our current system enables us to effectively evaluate the accuracy of the knot positioning task and completion time in a clinically realistic setup for Minimally Invasive Surgery

The Small Animal Radiation Research Platform (SARRP) is a novel and complete system capable of delivering multidirectional (focal), kilo-voltage radiation fields to targets in small animals under robotic control using cone-beam CT (CBCT) image guidance. The capability of the SARRP to deliver highly focused beams to multiple animal models provides new research opportunities that more realistically bridge laboratory research and clinical translation. This paper describes the design and operation of the SARRP for precise radiation delivery. Different delivery procedures are presented which enable the system to radiate through a series of points, representative of a complex shape. A particularly interesting case is shell dose irradiation, where the goal is to deliver a high dose of radiation to the shape surface, with minimal dose to the shape interior. The ability to deliver a dose shell allows mechanistic research of how a tumor interacts with its microenvironment to sustain its growth and lead to its resistance or recurrence.

Image-guided surgery offers great advantages to surgeons through the possibility to track tools in 3D space and to navigate based on the virtual model of the patient. In the case of robot-assisted procedures, both the inherent accuracy of the system components and the quality of the registration procedures are critical to provide high precision treatment delivery. One of the major barriers towards more technology-integrated procedures is the fact that alterations in the operating room environment can fundamentally change the performance of the system, decrease the accuracy, and therefore pose significant danger to the patient. Surgical events from the control point of view may include motion of the robot, motion of the camera, or motion of the patient. The paper describes a new concept to treat these events, to track and automatically compensate for abrupt changes that may affect the accuracy of a robot-integrated interventional system. Our solution is to use all available information at a given time, including the intra-operative tracker's internal base frame, to distinguish between different surgical events. The concept has been developed and tested on the neurosurgical robot system at the Johns Hopkins University. Initial experiments performed on data recordings from simulated scenarios showed that the algorithm was able to correctly identify the cases.

While a robot control framework generally focuses on real-time performance and efficient data exchange between cooperating tasks or processes, an application such as robot-assisted surgery often demands information from, and integration with, a number of other devices. Thus, the software framework for the integrated system may have different requirements and priorities than a framework for real-time robot control. This paper reports on a component-based architecture that seamlessly bridges the gap between real-time robot control and a distributed, integrated system. The starting point is the cisst library, which provides a component-based framework for lock-free and efficient data exchange between multiple threads within a single process, which is suitable for real-time robot control. This paper describes the extension of the cisst library to support distributed systems, while keeping the same programming model as the single-process, multi-threaded scenario. Thus, application software does not need to know whether the component providing services is within the same process, in a different process, or on a different computer. In comparison, most standard middleware packages support components that fall within the last two categories (different processes on the same computer or different computers). This does not allow them to take advantage of the higher performance that can be achieved using standard lock-free data structures that do not rely on the operating system or on middleware services. Thus, the novelty of this approach is that the same component-based architecture and associated programming model extends from a multi-threaded scenario (which provides the best real-time performance) to a standard multi-process distributed system.

This paper describes the development and initial testing of a new and optimized version of a steady-hand manipulator for retinal microsurgery. In the steady-hand paradigm, the surgeon and the robot share control of a tool attached to the robot through a force sensor. The robot controller senses forces exerted by the operator on the tool and uses this information in various control modes to provide smooth, tremor-free, precise positional control and force scaling. The steady-hand manipulator reported here has been specifically designed with the unique constraints of retinal microsurgery in mind. In particular, the system makes use of a compact wrist design that places the bulk of the robot away from the operating field. The resulting system has high efficacy, flexibility and ergonomics while meeting the accuracy and safety requirements of microsurgery. We have now tested this robot on a biological model system and we report a protocol for reliably cannulating ~80 mum OD veins (the size of veins in the human retina) using the system

We created an image-guided robot system to assist with skull base drilling by integrating our Steady Hand Robot with a Medtronic StealthStationreg Navigation System via its Stealthlink portal. The objective of this procedure is to create a opening in the skull base to allow access for neurosurgical interventions such as aneurysm clipping or tumor biopsy. The motivation for introducing an image-guided robot is to improve safety by preventing the surgeon from violating critical structures during the drilling procedure. Our approach is to attach the cutting tool to the robot end-effector and operate the robot in a cooperative control mode, where robot motion is determined from the forces and torques applied by the surgeon. We employ "virtual fixtures" to constrain the motion of the cutting tool so that it remains in the safe zone that was defined on a preoperative CT scan. This paper presents the system design and the results of phantom experiments performed with a synthetic skull.

This paper presents an incremental optical encoder based position and velocity measurements VLSI chip with a serial peripheral interface (SPI). It combines period and frequency countings to provide velocity estimates with good dynamic behavior over a wide speed range. By sensing the velocity of the encoder, it reserves the computational power of a supervisory microcontroller, and subsequently enhances the performance of the total system. Furthermore, multiple copies of the velocity encoder can access the behavior of multiple motors in parallel. It is compact with lower power consumption compared to traditional FPGA implementations. Although designed for use in the control unit of a medical robot with 34-axes and tight space and power constraints, it can be readily used in other applications. It is implemented in a 2P3M0.5μmCMOS process and consumes 4.82mW power with active area of0.45mm2.

Robots are increasingly being used in computer integrated surgery (CIS) systems, yet to our knowledge, there is no open source software that is specifically targeted at this application domain. In this paper, we derive unique requirements for medical robot controllers that are based on our experiences in the field. We present an overview of the second-generation open source software package (the cisst package), currently under development, that would include a family of application frameworks with dynamically loaded software components. We then focus on some key design details, which include extensive application of the command pattern as well as a dynamic interface query mechanism, and present a simple example to illustrate the concepts

Virtual fixtures are virtual constraints that enhance human performance in motion tasks. They can either confine and/or guide a user's motion. In this paper, we use a commercially available motion platform to explore the portability and applicability of virtual fixtures and document how people interact with them. Two micromanipulation tasks are analyzed and the effects of similarly designed virtual fixtures are discussed. One task simulates a medical task, retinal vein cannulation, and the other simulates a manufacturing task, fine leads soldering. Preliminary experimental results show that the virtual fixtures increase the accuracy of both medical and manufacturing tasks, lending support to its portability and applicability across unrelated tasks

Architecture and design parameters for a customizable VLSI chip to control a 34-axis robotic system are presented. The robot, which will be used in throat surgery, includes three small snake-like mechanisms, which places tough design challenges on the controller because of their requirement for precise sensing and control of low motor currents. The chip, in concert with control algorithms executed on a supervisory microprocessor, will deliver the appropriate control voltages to the motor to guide the end-effectors of the robot, while compensating for changes in resistance to motion as the robot interacts with tissue. The chip provides estimates of position, torque and velocity measurements to the microprocessor through a digital interface. Control signals, both generated on-chip and from the microprocessor, are conditioned and presented to the motor driver. The digital interface allows the chip to be configured for different motor specifications and various environmental conditions. Though designed for this particular robot, the chip can be applied to other robots with small DC brush motors

This paper describes an ongoing project to develop a computer-integrated system to assist surgeons in revision total hip replacement (RTHR) surgery. In RTHR surgery, a failing orthopedic hip implant, typically cemented, is replaced with a new one by removing the old implant, removing the cement and fitting a new implant into an enlarged canal broached in the femur. RTHR surgery is a difficult procedure fraught with technical challenges and a high incidence of complications. The goals of the computer-based system are the significant reduction of cement removal labor and time, the elimination of cortical wall penetration and femur fracture, the improved positioning and fit of the new implant resulting from precise, high-quality canal milling and the reduction of bone sacrificed to fit the new implant. Our starting points are the ROBODOC system for primary hip replacement surgery and the manual RTHR surgical protocol. We first discuss the main difficulties of computer-integrated RTHR surgery and identify key issues and possible solutions. We then describe possible system architectures and protocols for preoperative planning and intraoperative execution. We present a summary of methods and preliminary results in CT image metal artifact removal, interactive cement cut-volume definition and cement machining, anatomy-based registration using fluoroscopic X-ray images and clinical trials using an extended RTHR version of ROBODOC. We conclude with a summary of lessons learned and a discussion of current and future work.

We describe new methods for rigid registration of a preoperative computed tomography (CT)-scan image to a set of intraoperative X-ray fluoroscopic images, for guiding a surgical robot to its trajectory planned from CT. Our goal is to perform the registration, i.e., compute a rotation and translation of one data set with respect to the other to within a prescribed accuracy, based upon bony anatomy only, without external fiducial markers. With respect to previous approaches, the following aspects are new: 1) we correct the geometric distortion in fluoroscopic images and calibrate them directly with respect to the robot by affixing to it a new calibration device designed as a radiolucent rod with embedded metallic markers, and by moving the device along two planes, while radiographs are being acquired at regular intervals; 2) the registration uses an algorithm for computing the best transformation between a set of lines in three space, the (intraoperative) X-ray paths, and a set of points on the surface of the bone (imaged preoperatively), in a statistically robust fashion, using the Cayley parameterization of a rotation; and 3) to find corresponding sets of points to the X-ray paths on the surfaces, our new approach consists of extracting the surface apparent contours for a given viewpoint, as a set of closed three-dimensional nonplanar curves, before registering the apparent contours to X-ray paths. Aside from algorithms, there are a number of major technical difficulties associated with engineering a clinically viable system using anatomy and image-based registration. To detect and solve them, we have so far conducted two experiments with the surgical robot in an operating room (OR), using CT and fluoroscopic image data of a cadaver bone, and attempting to faithfully simulate clinical conditions. Such experiments indicate that intraoperative X-ray-based registration is a promising alternative to marker-based registration for clinical use with our proposed method.

The authors have developed an image-directed robotic system to augment the performance of human surgeons in precise bone machining procedures in orthopaedic surgery, initially targeted at cementless total hip replacement surgery. The total system consists of an interactive CT-based presurgical planning component and a surgical system consisting of a robot, redundant motion monitoring, and man-machine interface components. In vitro experiments conducted with this system have demonstrated an order-of-magnitude improvement in implant fit and placement accuracy, compared to standard manual preparation techniques. The first generation system described in this paper was used in a successful veterinary clinical trial on 26 dogs needing hip replacement surgery. It was the basis for subsequent development of a second-generation system that is now in human clinical trials.>

The authors describe the use of force feedback in a surgical robot system (ROBODOC). The application initially being addressed is total hip replacement (THR) surgery, where the robot must prepare a cavity in the femur for an artificial implant. In this system, force feedback is used to provide safety, tactile search capabilities, and an improved man-machine interface. Output of the force sensor is monitored by a safety processor, which initiates corrective action if any of several application-defined thresholds are exceeded. The robot is able to locate objects using guarded moves and force control (ball-in-cone strategy). In addition, the force control algorithm provides an intuitive man-machine interface which allows the surgeon to guide the robot by leading its tool to the desired location. An application of force control currently under development is described, where the force feedback is used to modify the cutter feed rate (force controlled velocity).>

The authors describe a robotic surgical system that has been designed to create femoral cavities that are precisely shaped and positioned for implantation of uncemented prostheses. This robotics system creates cavities with a dimensional accuracy more than 50 times greater than broached cavities, exceeds the tolerances to which implants are manufactured, and does not produce gaps that prevent bone ingrowth. A canine study was undertaken to evaluate the prosthesis fit and placement achieved by employing a surgical robot to prepare the femur. This study compared the results achieved on 15 dogs undergoing total hip replacement with manual broaching techniques and 25 dogs undergoing robotically assisted surgery. Among the 25 dogs, which ranged in age from 2/sup 1///sub 2/ to 11 years, there were no deaths, no infections, and no intraoperative complications. Human applications of this technique are also considered.>

The authors present the dual-drive control concept, which is a form of hybrid force/velocity control in which the constraint frame is automatically determined from feedback information. This allows compliant tasks to be conveniently specified in terms of a desired (normal) force and a desired (tangential) velocity. The proposed control algorithm is appropriate for compliant tasks that require motion orthogonal to the contact force; for example, turning a crank or tracking a surface. These basic tasks can be performed without the continuous involvement and corresponding overhead of a high-level planner. This is accomplished by the definition of a plane on which compliant motion is to occur and a point with which force and velocity directions are determined. The dual drive controller has been implemented on the SIERA system and is being used to control an IBM 7565 robot. Experimental results for crank-turning and surface-tracking problems are provided to illustrate the algorithm.>

A description is given of the software environment for SPARTA (signal processor architecture for real-time applications). The SPARTA hardware consists of a hybrid system with three different types of computers. The program development environment includes the PLH high-level language (on an IBM VM/CMS mainframe) for generating efficient real-time signal processor code. The runtime (user) interface (on an IBM PC) supports symbolic debugging, dynamic loading and linking, and synchronous switching of control algorithms during real-time program execution. It has been interfaced to the object-oriented AML/2 language interpreter. The real-time environment (on multiple IBM Hermes signal processors) has tasks with multiple entries and exits, linked by a method which is somewhat similar to threaded code, but which has lower overhead (0.1 mu s, nonpreemptive task switch). The real-time kernel also supports multiple threads of tasks categorized as critical or abortable, reflecting their importance to the real-time system. These threads are scheduled by a priority-based, preemptive scheduler, which has 1.3-5.1 mu s total overhead. A SPARTA system with one Hermes signal processor, executing a dozen real-time tasks, has achieved a sample rate of 5 kHz for a fully-coupled PID control, with nonlinear compensation, for the Eaglet II Cartesian robot (five motors achieving 2- mu m linear resolution and 0.02 degrees angular resolution).>

The authors describe the SPARTA (signal processor architecture for real-time applications) system, which is a hybrid computer development system including: (1) a program development environment on an IBM VM/CMS mainframe computer: (2) user interface and runtime support on an IBM PC; and (3) real-time computation and input/output using multiple IBM Hermes signal processors situated on the IBM PC bus. The program development environment includes the PLH high-level language for generating efficient real-time Hermes code. The runtime supports symbolic debugging, dynamic loading and linking, and synchronous switching of control algorithms during real-time program execution. The real-time environment includes a distributed operating system which supports foreground and background task execution and real-time data collection and display. The task switching overhead is 0.1 mu s, nonpreemptive, and 1.0-3.7 mu s when the context is changed by an interrupt (i.e. preemptive).>

The SPARTA (signal processor for advanced robot technology applications) system is a hybrid computer which includes: (1) program development environment on an IBM VM/CMS mainframe computer; (2) user interface and runtime support on an IBM PC; and (3) real-time computation and I/O using multiple IBM Hermes signal processors situated on the IBM PC bus. The program-development environment includes the PLH high-level language for generating efficient real-time Hermes code. The runtime supports symbolic debugging, dynamic loading, and synchronous switching of control algorithms during real-time program execution. The real-time environment includes an operating system which supports foreground and background task execution and real-time data collection and display. A SPARTA system with three Hermes signal processors is currently being used to develop control algorithms for a two-axis fast direct-drive Cartesian robot at sample rates in excess of 10 kHz;.>

SIERA (System for Implementing and Evaluating Robotic Algorithms) is a multiprocessor system that has been developed at the Laboratory for Engineering Man/Machine Systems (LEMS) at Brown University. It incorporates a tightly coupled bus-based system (the Real Time Servo System) and a loosely coupled point-to-point network (the Armstrong Multiprocessor System). SIERA is capable of controlling many types of commercially available robots because the modular construction of its hardware and software has minimized robot dependencies. Three programming levels are introduced to classify the different ways SIERA can be utilized-for simple robot operation, for robotics research, and for system enhancements. The environment associated with each of these levels provides the necessary programming support and interactive commands. The concepts of servo loops and controller modifications (cmods) are introduced to facilitate programming on the Real Time Servo System. A simple example is presented to illustrate operation of the system.