Robotic platforms provide consistent and precise tool positioning that significantly enhances retinal microsurgery. Integrating such systems with intraoperative optical coherence tomography (iOCT) enables image-guided robotic interventions, allowing autonomous performance of advanced treatments, such as injecting therapeutic agents into the subretinal space. However, tissue deformations due to tool-tissue interactions constitute a significant challenge in autonomous iOCT-guided robotic subretinal injections. Such interactions impact correct needle positioning and procedure outcomes. This paper presents a novel method for autonomous subretinal injection under iOCT guidance that considers tissue deformations during the insertion procedure. The technique is achieved through real-time segmentation and 3D reconstruction of the surgical scene from densely sampled iOCT B-scans, which we refer to as B5_ scans. Using B5-scans we monitor the position of the instrument relative to a virtual target layer between the ILM and RPE. Our experiments on ex-vivo porcine eyes demonstrate dynamic adjustment of the insertion depth and overall improved accuracy in needle positioning compared to prior autonomous insertion approaches. Compared to a 35% success rate in subretinal bleb generation with previous approaches, our method reliably created subretinal blebs in 90% our experiments. The source code and data used in this study are publicly available on GitHub11https://github.com/demirarikan/virtual-Iayer-retinal-surgery.

Performing retinal vein cannulation (RVC) as a potential treatment for retinal vein occlusion (RVO) without the assistance of a surgical robotic system is very challenging to do safely. The main limitation is the physiological hand tremor of surgeons. Robot-assisted eye surgery technology may resolve the problems of hand tremors and fatigue and improve the safety and precision of RVC. The Steady-Hand Eye Robot (SHER) is an admittance-based robotic system that can filter out hand tremors and enables ophthalmologists to manipulate a surgical instrument inside the eye cooperatively. However, the admittance-based cooperative control mode does not safely minimize the contact force between the surgical instrument and the sclera to prevent tissue damage. In addition, features such as haptic feedback or hand motion scaling, which can improve the safety and precision of surgery, require a teleoperation control framework. This work presents, for the first time in the field of robot-assisted retinal microsurgery research, a registration-free bimanual adaptive teleoperation (BMAT) control framework using SHER 2.0 and SHER 2.1 robotic systems. Both SHERs are integrated with an adaptive force control (AFC) algorithm that dynamically and automatically minimizes the tool-sclera interaction forces, enforcing them within a safe limit. The scleral forces are measured using two fiber Bragg grating (FBG)-based force-sensing tools. The performance of the proposed BMAT control framework is evaluated by comparison with a bimanual adaptive cooperative (BMAC) framework in a vessel-following experiment conducted under a surgical microscope. Experimental results demonstrate the effectiveness of the BMAT control framework in performing a safe bimanual telemanipulation of the eye without over-stretching it, even in the absence of registration between the two robots.

A surgeon's physiological hand tremor can significantly impact the outcome of delicate and precise retinal surgery, such as retinal vein cannulation (RVC) and epiretinal membrane peeling. Robot-assisted eye surgery technology provides ophthalmologists with advanced capabilities such as hand tremor cancellation, hand motion scaling, and safety constraints that enable them to perform these otherwise challenging and high-risk surgeries with high precision and safety. Steady-Hand Eye Robot (SHER) with cooperative control mode can filter out surgeon's hand tremor, yet another important safety feature, that is, minimizing the contact force between the surgical instrument and sclera surface for avoiding tissue damage cannot be met in this control mode. Also, other capabilities, such as hand motion scaling and haptic feedback, require a teleoperation control framework. In this work, for the first time, we implemented a teleoperation control mode incorporated with an adaptive sclera force control algorithm using a PHANTOM Omni haptic device and a force-sensing surgical instrument equipped with Fiber Bragg Grating (FBG) sensors attached to the SHER 2.1 end-effector. This adaptive sclera force control algorithm allows the robot to dynamically minimize the tool-sclera contact force. Moreover, for the first time, we compared the performance of the proposed adaptive teleoperation mode with the cooperative mode by conducting a vessel-following experiment inside an eye phantom under a microscope.

Skull base surgery is a demanding field in which surgeons operate in and around the skull while avoiding critical anatomical structures including nerves and vasculature. While image-guided surgical navigation is the prevailing standard, limitation still exists requiring personalized planning and recognizing the irreplaceable role of a skilled surgeon. This paper presents a collaboratively controlled robotic system tailored for assisted drilling in skull base surgery. Our central hypothesis posits that this collaborative system, enriched with haptic assistive modes to enforce virtual fixtures, holds the potential to significantly enhance surgical safety, streamline efficiency, and alleviate the physical demands on the surgeon. The paper describes the intricate system development work required to enable these virtual fixtures through haptic assistive modes. To validate our system’s performance and effectiveness, we conducted initial feasibility experiments involving a medical student and two experienced surgeons. The experiment focused on drilling around critical structures following cortical mastoidectomy, utilizing dental stone phantom and cadaveric models. Our experimental results demonstrate that our proposed haptic feedback mechanism enhances the safety of drilling around critical structures compared to systems lacking haptic assistance. With the aid of our system, surgeons were able to safely skeletonize the critical structures without breaching any critical structure even under obstructed view of the surgical site.

Productive human-robot partnerships are vital to successful integration of assistive robots into everyday life. Although prior research has explored techniques to facilitate collaboration during human-robot interaction, the work described here aims to forge productive partnerships prior to human-robot interaction, drawing upon team-building activities’ aid in establishing effective human teams. Through a 2 (group membership: ingroup and outgroup) ×3 (robot error: main task errors, side task errors, and no errors) online study ( N=62 ), we demonstrate that (1) a non-social pre-task exercise can help form ingroup relationships; (2) an ingroup robot is perceived as a better, more committed teammate than an outgroup robot (despite the two behaving identically); and (3) participants are more tolerant of negative outcomes when working with an ingroup robot. We discuss how pre-task exercises may serve as an active task failure mitigation strategy.

Image-based 2D/3D registration is a critical technique for fluoroscopic guided surgical interventions. Conventional intensity-based 2D/3D registration approa- ches suffer from a limited capture range due to the presence of local minima in hand-crafted image similarity functions. In this work, we aim to extend the 2D/3D registration capture range with a fully differentiable deep network framework that learns to approximate a convex-shape similarity function. The network uses a novel Projective Spatial Transformer (ProST) module that has unique differentiability with respect to 3D pose parameters, and is trained using an innovative double backward gradient-driven loss function. We compare the most popular learning-based pose regression methods in the literature and use the well-established CMAES intensity-based registration as a benchmark. We report registration pose error, target registration error (TRE) and success rate (SR) with a threshold of 10mm for mean TRE. For the pelvis anatomy, the median TRE of ProST followed by CMAES is 4.4mm with a SR of 65.6% in simulation, and 2.2mm with a SR of 73.2% in real data. The CMAES SRs without using ProST registration are 28.5% and 36.0% in simulation and real data, respectively. Our results suggest that the proposed ProST network learns a practical similarity function, which vastly extends the capture range of conventional intensity-based 2D/3D registration. We believe that the unique differentiable property of ProST has the potential to benefit related 3D medical imaging research applications. The source code is available at https://github.com/gaocong13/Projective-Spatial-Transformers.

Neural surface reconstruction has been shown to be powerful for recovering dense 3D surfaces via image-based neural rendering. However, current methods struggle to recover detailed structures of real-world scenes. To address the issue, we present Neuralangelo, which combines the representation power of multiresolution 3D hash grids with neural surface rendering. Two key ingredients enable our approach: (1) numerical gradients for computing higher-order derivatives as a smoothing operation and (2) coarse-to-fine optimization on the hash grids controlling different levels of details. Even without auxiliary inputs such as depth, Neuralangelo can effectively recover dense 3D surface structures from multiview images with fidelity significantly surpassing previous methods, enabling detailed large-scale scene reconstruction from RGB video captures.

Prior error management techniques often do not possess the versatility to appropriately address robot errors across tasks and scenarios. Their fundamental framework involves explicit, manual error management and implicit domain-specific information driven error management, tailoring their response for specific interaction contexts. We present a framework for approaching error-aware systems by adding implicit social signals as another information channel to create more flexibility in application. To support this notion, we introduce a novel dataset (composed of three data collections) with a focus on understanding natural facial action unit (AU) responses to robot errors during physical-based human-robot interactions---varying across task, error, people, and scenario. Analysis of the dataset reveals that, through the lens of error detection, using AUs as input into error management affords flexibility to the system and has the potential to improve error detection response rate. In addition, we provide an example real-time interactive robot error management system using the error-aware framework.

The introduction of image-guided surgical navigation (IGSN) has greatly benefited technically demanding surgical procedures by providing real-time support and guidance to the surgeon during surgery. To develop effective IGSN, a careful selection of the surgical information and the medium to present this information to the surgeon is needed. However, this is not a trivial task due to the broad array of available options. To address this problem, we have developed an open-source library that facilitates the development of multimodal navigation systems in a wide range of surgical procedures relying on medical imaging data. To provide guidance, our system calculates the minimum distance between the surgical instrument and the anatomy and then presents this information to the user through different mechanisms. The real-time performance of our approach is achieved by calculating Signed Distance Fields at initialization from segmented anatomical volumes. Using this framework, we developed a multimodal surgical navigation system to help surgeons navigate anatomical variability in a skull base surgery simulation environment. Three different feedback modalities were explored: visual, auditory, and haptic. To evaluate the proposed system, a pilot user study was conducted in which four clinicians performed mastoidectomy procedures with and without guidance. Each condition was assessed using objective performance and subjective workload metrics. This pilot user study showed improvements in procedural safety without additional time or workload. These results demonstrate our pipeline's successful use case in the context of mastoidectomy.

Telesurgery has a clear potential for providing high-quality surgery to medically underserved areas like rural areas, battlefields, and spacecraft; nevertheless, effective methods to overcome unreliable communication systems are still lacking. Furthermore, it is not well understood how users react at the moment of communication loss and also during the loss. In this paper, we aim to analyze human response by proposing a telesurgery simulation framework that models an environment incorporating local and remote sites. Furthermore, this framework generates structural data for human behavior analysis and can provide different forms of assistance during the communication failure and at the communication recovery. We investigated three different types of assistance: User-centered, Robot-centered and Hybrid. A 12-person user-study was carried out using the proposed telesurgery simulation where participants completed a peg transfer task with random communication loss. The collected data was used to analyze the human response to a communication failure. The proposed Hybrid method reduced temporal demand with no increase in completion time compared to the baseline control method where users were unable to move the input device during the communication loss. The Hybrid method also significantly reduced both the task completion time and workload compared to the other two proposed methods (User-centered and Robot-centered).

The rigid and straight nature of conventional surgical drills and screwdrivers makes it difficult to access the posterior mandible for fracture reduction without the creation of facial incisions. To assist transoral mandibular fracture reduction in hard-to-reach areas, we propose a shared-control dexterous robotic system. The end effector of this system is an articulated drilling/screwing tool to provide distal dexterity. This system uses an admittance-control-based approach to provide precision and stability during shared-control hole-drilling processes. A cadaver study showed the efficacy of the proposed system to assist plate fixation in the reduction of mandibular fractures. The proposed articulated surgical tool was capable of drilling holes in and driving screws into the mandible of a cadaver head. In addition, the shared-control robotic system ensured that the drill moved along its axial direction, leading to stable and precise hole drilling.

Temporally consistent depth estimation is crucial for online applications such as augmented reality. While stereo depth estimation has received substantial attention as a promising way to generate 3D information, there is relatively little work focused on maintaining temporal stability. Indeed, based on our analysis, current techniques still suffer from poor temporal consistency. Stabilizing depth temporally in dynamic scenes is challenging due to concurrent object and camera motion. In an online setting, this process is further aggravated because only past frames are available. We present a framework named Consistent Online Dynamic Depth (CODD) to produce temporally consistent depth estimates in dynamic scenes in an online setting. CODD augments per-frame stereo networks with novel motion and fusion networks. The motion network accounts for dynamics by predicting a per-pixel SE3 transformation and aligning the observations. The fusion network improves temporal depth consistency by aggregating the current and past estimates. We conduct extensive experiments and demonstrate quantitatively and qualitatively that CODD outperforms competing methods in terms of temporal consistency and performs on par in terms of per-frame accuracy.

Notable challenges during retinal surgery lend themselves to robotic assistance which has proven beneficial in providing a safe steady-hand manipulation. Efficient assistance from the robots heavily relies on accurate sensing of surgery states (e.g. instrument tip localization and tool-to-tissue interaction forces). Many of the existing tool tip localization methods require preoperative frame registrations or instrument calibrations. In this study using an iterative approach and by combining vision and force-based methods, we develop calibration- and registration-independent (RI) algorithms to provide online estimates of instrument stiffness (least squares and adaptive). The estimations are then combined with a state-space model based on the forward kinematics (FWK) of the Steady-Hand Eye Robot (SHER) and Fiber Brag Grating (FBG) sensor measurements. This is accomplished using a Kalman Filtering (KF) approach to improve the deflected instrument tip position estimations during robot-assisted eye surgery. The conducted experiments demonstrate that when the online RI stiffness estimations are used, the instrument tip localization results surpass those obtained from pre-operative offline calibrations for stiffness.

., surgical navigation) would benefit from a real-time method that can simultaneously track the endoscope and reconstruct the dense 3D geometry of the observed anatomy from a monocular endoscopic video. To this end, we develop a Simultaneous Localization and Mapping system by combining the learning-based appearance and optimizable geometry priors and factor graph optimization. The appearance and geometry priors are explicitly learned in an end-to-end differentiable training pipeline to master the task of pair-wise image alignment, one of the core components of the SLAM system. In our experiments, the proposed SLAM system is shown to robustly handle the challenges of texture scarceness and illumination variation that are commonly seen in endoscopy. The system generalizes well to unseen endoscopes and subjects and performs favorably compared with a state-of-the-art feature-based SLAM system. The code repository is available at https://github.com/lppllppl920/SAGE-SLAM.git.

In human-robot collaboration, robot errors are inevitable—damaging user trust, willingness to work together, and task performance. Prior work has shown that people naturally respond to robot errors socially and that in social interactions it is possible to use human responses to detect errors. However, there is little exploration in the domain of nonsocial, physical human-robot collaboration such as assembly and tool retrieval. In this work, we investigate how people's organic, social responses to robot errors may be used to enable timely automatic detection of errors in physical human-robot interactions. We conducted a data collection study to obtain facial responses to train a real-time detection algorithm and a case study to explore the generalizability of our method with different task settings and errors. Our results show that natural social responses are effective signals for timely detection and localization of robot errors even in nonsocial contexts and that our method is robust across a variety of task contexts, robot errors, and user responses. This work contributes to robust error detection without detailed task specifications.

Recent developments in data science in general and machine learning in particular have transformed the way experts envision the future of surgery. Surgical Data Science (SDS) is a new research field that aims to improve the quality of interventional healthcare through the capture, organization, analysis and modeling of data. While an increasing number of data-driven approaches and clinical applications have been studied in the fields of radiological and clinical data science, translational success stories are still lacking in surgery. In this publication, we shed light on the underlying reasons and provide a roadmap for future advances in the field. Based on an international workshop involving leading researchers in the field of SDS, we review current practice, key achievements and initiatives as well as available standards and tools for a number of topics relevant to the field, namely (1) infrastructure for data acquisition, storage and access in the presence of regulatory constraints, (2) data annotation and sharing and (3) data analytics. We further complement this technical perspective with (4) a review of currently available SDS products and the translational progress from academia and (5) a roadmap for faster clinical translation and exploitation of the full potential of SDS, based on an international multi-round Delphi process.

Surgical robots have been widely adopted with over 4000 robots being used in practice daily. However, these are telerobots that are fully controlled by skilled human surgeons. Introducing "surgeon-assist"-some forms of autonomy-has the potential to reduce tedium and increase consistency, analogous to driver-assist functions for lanekeeping, cruise control, and parking. This article examines the scientific and technical backgrounds of robotic autonomy in surgery and some ethical, social, and legal implications. We describe several autonomous surgical tasks that have been automated in laboratory settings, and research concepts and trends.

Ever since their first introduction in the late 1980s[1],[2], surgical robots have played an increasingly prominent role in medical practice[3],[4]. For example, a recent study[5]found that over 15% of all general surgery procedures in 2020 were performed robotically, compared to only 1.8% in 2012. The current worldwide robotic surgery market is estimated to be$\$ $5.3 billion and is expected to reach$\$ $19 billion by 2027, with a compound annual growth rate over 21%[6].

This article presents a dexterous robotic system for autonomous debridement of osteolytic bone lesions in confined spaces. The proposed system is distinguished from the state-of-the-art orthopedics systems because it combines a rigid-link robot with a continuum manipulator (CM) that enhances reach in difficult-to-access spaces often encountered in surgery. The CM is equipped with flexible debriding instruments and fiber Bragg grating sensors. The surgeon plans on the patient's preoperative computed tomography and the robotic system performs the task autonomously under the surgeon's supervision. An optimization-based controller generates control commands on the fly to execute the task while satisfying physical and safety constraints. The system design and controller are discussed and extensive simulation, phantom and human cadaver experiments are carried out to evaluate the performance, workspace, and dexterity in confined spaces. Mean and standard deviation of target placement are 0.5 and 0.18 mm, and the robotic system covers 91% of the workspace behind an acetabular implant in treatment of hip osteolysis, compared to the 54% that is achieved by conventional rigid tools.

Extracting geometric features from 3D models is a common first step in applications such as 3D registration, tracking, and scene flow estimation. Many hand-crafted and learning-based methods aim to produce consistent and distinguishable geometric features for 3D models with partial overlap. These methods work well in cases where the point density and scale of the overlapping 3D objects are similar, but struggle in applications where 3D data are obtained independently with unknown global scale and scene overlap. Unfortunately, instances of this resolution mismatch are common in practice, e.g., when aligning data from multiple sensors. In this work, we introduce a new normalization technique, Batch-Neighborhood Normalization, aiming to improve robustness to mean-std variation of local feature distributions that presumably can happen in samples with varying point density. We empirically demonstrate that the presented normalization method’s performance compares favorably to comparison methods in indoor and outdoor environments, and on a clinical dataset, on common point registration benchmarks in both standard and, particularly, resolution-mismatch settings. The source code and clinical dataset are available at https://github.com/lppllppl920/NeighborhoodNormalization-Pytorch.

Stereo depth estimation relies on optimal correspondence matching between pixels on epipolar lines in the left and right images to infer depth. In this work, we revisit the problem from a sequence-to-sequence correspondence perspective to replace cost volume construction with dense pixel matching using position information and attention. This approach, named STereo TRansformer (STTR), has several advantages: It 1) relaxes the limitation of a fixed disparity range, 2) identifies occluded regions and provides confidence estimates, and 3) imposes uniqueness constraints during the matching process. We report promising results on both synthetic and real-world datasets and demonstrate that STTR generalizes across different domains, even without fine-tuning.

Malaria is a worldwide scourge, and the broad deployment of an effective vaccine would improve the lives of millions of people. A vaccine based on Plasmodium falciparum (PfSPZ) sporozoites extracted from the salivary glands of infected mosquitoes shows significant promise. However, the large-scale industrial production of PfSPZ-based vaccines will benefit from automation of the key step of extracting sporozoites from mosquito salivary glands that is currently performed by manual microdissection. In this work, we demonstrate a robotic system prototype for extracting salivary glands from mosquitoes to streamline vaccine production and reduce the need for operators. In the proposed system, mosquitoes are decapitated in an automated robotic pick-place-decapitate process, then a squeezer apparatus extracts mosquito salivary glands from the body. Mosquito detection and body part localization are performed by computer vision methods. The software allows system operation in simulation and on the robotic hardware, which facilitates subsystem development and integration. Experiments show encouraging results with success rates of 93% in robotic mosquito manipulation and 87.1% in salivary gland extraction. The system has the potential to improve the efficiency of PfSPZ vaccine production with significant gains in throughput and reduction in training times for a highly deskilled initial manual step. Further, this system is expected to pave the way for a more mature future system.

Lumbar injection is an image-guided procedure performed manually for diagnosis and treatment of lower back pain and leg pain. Previously, we have developed and verified an MR-Conditional robotic solution to assisting the needle insertion process. Drawing on our clinical experiences, a virtual remote center of motion (RCM) constraint is implemented to enable our robot to mimic a clinician’s hand motion to adjust the needle tip position in situ. Force and image data are collected to study the needle behavior in gel phantoms during this motion, and a mechanics-based needle-tissue interaction model is proposed and evaluated to further examine the underlying physics. This work extends the commonly-adopted notion of an RCM for flexible needles, and introduces new motion parameters to describe the needle behavior. The model parameters can be tuned to match the experimental result to sub-millimeter accuracy, and this proposed needle manipulation method presents a safer alternative to laterally translating the needle during in situ needle adjustments.

The treatment of malaria is a global health challenge that stands to benefit from the widespread introduction of a vaccine for the disease. A method has been developed to create a live organism vaccine using the sporozoites (SPZ) of the parasite Plasmodium falciparum (Pf), which are concentrated in the salivary glands of infected mosquitoes. Current manual dissection methods to obtain these PfSPZ are not optimally efficient for large-scale vaccine production. We propose an improved dissection procedure and a mechanical fixture that increases the rate of mosquito dissection and helps to deskill this stage of the production process. We further demonstrate the automation of a key step in this production process, the picking and placing of mosquitoes from a staging apparatus into a dissection assembly. This unit test of a robotic mosquito pick-and-place system is performed using a custom-designed micro-gripper attached to a four degree of freedom (4-DOF) robot under the guidance of a computer vision system. Mosquitoes are autonomously grasped and pulled to a pair of notched dissection blades to remove the head of the mosquito, allowing access to the salivary glands. Placement into these blades is adapted based on output from computer vision to accommodate for the unique anatomy and orientation of each grasped mosquito. In this pilot test of the system on 50 mosquitoes, we demonstrate a 100% grasping accuracy and a 90% accuracy in placing the mosquito with its neck within the blade notches such that the head can be removed. This is a promising result for this difficult and non-standard pick-and-place task.

High-quality 3D reconstructions from endoscopy video play an important role in many clinical applications, including surgical navigation where they enable direct video-CT registration. While many methods exist for general multi-view 3D reconstruction, these methods often fail to deliver satisfactory performance on endoscopic video. Part of the reason is that local descriptors that establish pair-wise point correspondences, and thus drive reconstruction, struggle when confronted with the texture-scarce surface of anatomy. Learning-based dense descriptors usually have larger receptive fields enabling the encoding of global information, which can be used to disambiguate matches. In this work, we present an effective self-supervised training scheme and novel loss design for dense descriptor learning. In direct comparison to recent local and dense descriptors on an in-house sinus endoscopy dataset, we demonstrate that our proposed dense descriptor can generalize to unseen patients and scopes, thereby largely improving the performance of Structure from Motion (SfM) in terms of model density and completeness. We also evaluate our method on a public dense optical flow dataset and a small-scale SfM public dataset to further demonstrate the effectiveness and generality of our method. The source code is available at https://github.com/lppllppl920/DenseDescriptorLearning-Pytorch.

This paper presents a dynamic constraint formulation to provide protective virtual fixtures of 3D anatomical structures from polygon mesh representations. The proposed approach can anisotropically limit the tool motion of surgical robots without any assumption of the local anatomical shape close to the tool. Using a bounded search strategy and Principle Directed tree, the proposed system can run efficiently at 180 Hz for a mesh object containing 989,376 triangles and 493,460 vertices. The proposed algorithm has been validated in both simulation and skull cutting experiments. The skull cutting experiment setup uses a novel piezoelectric bone cutting tool designed for the da Vinci research kit. The result shows that the virtual fixture assisted teleoperation has statistically significant improvements in the cutting path accuracy and penetration depth control. The code has been made publicly available at https://github.com/mli0603/PolygonMeshVirtualFixture.

Robot-assisted vitreoretinal surgery can filter surgeons' hand tremors and provide safe, accurate tool manipulation. In this paper, we report the design, optimization, and evaluation of a novel tilt mechanism for a new Steady-Hand Eye Robot (SHER). The new tilt mechanism features a four-bar linkage design and has a compact structure. Its kinematic configuration is optimized to minimize the required linear range of motion (LRM) for implementing a virtual remote center-of-motion (V-RCM) while tilting a surgical tool. Due to the different optimization constraints for the robots at the left and right sides of the human head, two configurations of this tilt mechanism are proposed. Experimental results show that the optimized tilt mechanism requires a significantly smaller LRM (e.g. 5.08 mm along Z direction and 8.77 mm along Y direction for left side robot) as compared to the slider-crank tilt mechanism used in the previous SHER (32.39 mm along Z direction and 21.10 mm along Y direction). The feasibility of the proposed tilt mechanism is verified in a mock bilateral robot-assisted vitreoretinal surgery. The ergonomically acceptable robot postures needed to access the surgical field is also determined.

We present a self-supervised approach to training convolutional neural networks for dense depth estimation from monocular endoscopy data without a priori modeling of anatomy or shading. Our method only requires monocular endoscopic videos and a multi-view stereo method, e.g., structure from motion, to supervise learning in a sparse manner. Consequently, our method requires neither manual labeling nor patient computed tomography (CT) scan in the training and application phases. In a cross-patient experiment using CT scans as groundtruth, the proposed method achieved submillimeter mean residual error. In a comparison study to recent self-supervised depth estimation methods designed for natural video on in vivo sinus endoscopy data, we demonstrate that the proposed approach outperforms the previous methods by a large margin. The source code for this work is publicly available online at https://github.com/lppllppl920/EndoscopyDepthEstimation-Pytorch.

In this article, we present a novel stochastic algorithm called simultaneous sensor calibration and deformation estimation (SCADE) to address the problem of modeling deformation behavior of a generic continuum manipulator (CM) in free and obstructed environments. In SCADE, using a novel mathematical formulation, we introduce a priori model-independent filtering algorithm to fuse the continuous and inaccurate measurements of an embedded sensor (e.g., magnetic or piezoelectric sensors) with an intermittent but accurate data of an external imaging system (e.g., optical trackers or cameras). The main motivation of this article is the crucial need of obtaining an accurate shape/position estimation of a CM utilized in a surgical intervention. In these robotic procedures, the CM is typically equipped with an embedded sensing unit (ESU) while an external imaging modality (e.g., ultrasound or a fluoroscopy machine) is also available in the surgical site. The results of two different set of prior experiments in free and obstructed environments were used to evaluate the efficacy of SCADE algorithm. The experiments were performed with a CM specifically designed for orthopaedic interventions equipped with an inaccurate Fiber Bragg Grating (FBG) ESU and overhead camera. The results demonstrated the successful performance of the SCADE algorithm in simultaneous estimation of unknown deformation behavior of the utilized unmodeled CM together with realizing the time-varying drift of the poor-calibrated FBG sensing unit. Moreover, the results showed the phenomenal out-performance of the SCADE algorithm in estimation of the CM's tip position as compared to FBG-based position estimations.

In adult laparoscopy, robot-aided surgery is a reality in thousands of operating rooms worldwide, owing to the increased dexterity provided by the robotic tools. Many robots and robot control techniques have been developed to aid in more challenging scenarios, such as pediatric surgery and microsurgery. However, the prevalence of case-specific solutions, particularly those focused on non-redundant robots, reduces the reproducibility of the initial results in more challenging scenarios. In this paper, we propose a general framework for the control of surgical robotics in constrained workspaces under teleoperation, regardless of the robot geometry. Our technique is divided into a slave-side constrained optimization algorithm, which provides virtual fixtures, and with Cartesian impedance on the master side to provide force feedback. Experiments with two robotic systems, one redundant and one non-redundant, show that smooth teleoperation can be achieved in adult laparoscopy and infant surgery.

When robotic assistance is present into vitreoretinal surgery, the surgeon will experience reduced sensory input that is otherwise derived from the tool's interaction with the eye wall (sclera). We speculate that disconnecting the surgeon from this sensory input may increase the risk of injury to the eye and affect the surgeon's usual technique. On the other hand, robot autonomous motion to enhance patient safety might inhibit the surgeons tool manipulation and diminish surgeon comfort with the procedure. In this study, to investigate the parameters of patient safety and surgeon comfort in a robot-assisted eye surgery, we implemented three different approaches designed to keep the scleral force in a safe range during a synergic eye manipulation task. To assess the surgeon comfort during these procedures, the amount of interference with the surgeons usual maneuvers has been analyzed by defining quantitative comfort metrics. The first two utilized scleral force control approaches are based on an adaptive force control method in which the robot actively counteracts any excessive force on the sclera. The third control method is based on a virtual fixture approach in which a virtual wall is created for the surgeon in the unsafe directions of manipulation. The performance of the utilized approaches was evaluated in user studies with two experienced retinal surgeons and the outcomes of the procedure were assessed using the defined safety and comfort metrics. Results of these analyses indicate the significance of the opted control paradigm on the outcome of a safe and comfortable robot-assisted eye surgery.

In this paper, a novel semi-autonomous control framework is presented for enabling probe-based confocal laser endomicroscopy (pCLE) scan of the retinal tissue. With pCLE, retinal layers such as nerve fiber layer (NFL) and retinal ganglion cell (RGC) can be scanned and characterized in real-time for an improved diagnosis and surgical outcome prediction. However, the limited field of view of the pCLE system and the micron-scale optimal focus distance of the probe, which are in the order of physiological hand tremor, act as barriers to successful manual scan of retinal tissue. Therefore, a novel sensorless framework is proposed for real-time semi-autonomous endomicroscopy scanning during retinal surgery. The framework consists of the Steady-Hand Eye Robot (SHER) integrated with a pCLE system, where the motion of the probe is controlled semi-autonomously. Through a hybrid motion control strategy, the system autonomously controls the confocal probe to optimize the sharpness and quality of the pCLE images, while providing the surgeon with the ability to scan the tissue in a tremor-free manner. Effectiveness of the proposed architecture is validated through experimental evaluations as well as a user study involving 9 participants. It is shown through statistical analyses that the proposed framework can reduce the work load experienced by the users in a statistically-significant manner, while also enhancing their performance in retaining pCLE images with optimized quality.

Retinal surgery involves manipulating very delicate tissues within the confined area of eyeball. In such demanding practices, patient involuntary head movement might abruptly raise tool-to-eyeball interaction forces which would be detrimental to eye. This study is aimed at implementing different force control strategies and evaluating how they contribute to attaining sclera force safety while patient head drift is present. To simulate patient head movement, a piezoelectric-actuated linear stage is used to produce random motions in a single direction in random time intervals. Having an eye phantom attached to the linear stage then an experienced eye surgeon is asked to manipulate the eye and repeat a mock surgical task both with and without the assist of the Steady-Hand Eye Robot. For the freehand case, warning sounds were provided to the surgeon as auditory feedback to alert him about excessive slclra forces. For the robot-assisted experiments two variants of an adaptive sclera force control and a virtual fixture method were deployed to see how they can maintain eye safety under head drift circumstances. The results indicate that the developed robot control strategies are able to compensate for head drift and keep the sclera forces under safe levels as well as the free hand operation.

Tracking and controlling the shape of continuum dexterous manipulators (CDM) in constraint environments is a challenging task. The imposed constraints and interaction with unknown obstacles may conform the CDM's shape and therefore demands for shape sensing methods which do not rely on direct line of sight. To address these issues, we integrate a novel Fiber Bragg Grating (FBG) shape sensing unit into a CDM, reconstruct the shape in real-time, and develop an optimization-based control algorithm using FBG tip position feedback. The CDM is designed for less-invasive treatment of osteolysis (bone degradation). To evaluate the performance of the feedback control algorithm when the CDM interacts with obstacles, we perform a set of experiments similar to the real scenario of the CDM interaction with soft and hard lesions during the treatment of osteolysis. In addition, we propose methods for identification of the CDM collisions with soft or hard obstacles using the jacobian information. Results demonstrate successful control of the CDM tip based on the FBG feedback and indicate repeatability and robustness of the proposed method when interacting with unknown obstacles.

Functional endoscopic sinus surgery (FESS) is one of the most common outpatient surgical procedures performed in the head and neck region. It is used to treat chronic sinusitis, a disease characterized by inflammation in the nose and surrounding paranasal sinuses, affecting about 15% of the adult population. During FESS, the nasal cavity is visualized using an endoscope, and instruments are used to remove tissues that are often within a millimeter of critical anatomical structures, such as the optic nerve, carotid arteries, and nasolacrimal ducts. To maintain orientation and to minimize the risk of damage to these structures, surgeons use surgical navigation systems to visualize the 3-D position of their tools on patients' preoperative Computed Tomographies (CTs). This paper presents an image-based method for enhanced endoscopic navigation. The main contributions are: (1) a system that enables a surgeon to asynchronously register a sequence of endoscopic images to a CT scan with higher accuracy than other reported solutions using no additional hardware; (2) the ability to report the robustness of the registration; and (3) evaluation on in vivo human data. The system also enables the overlay of anatomical structures, visible, or occluded, on top of video images. The methods are validated on four different data sets using multiple evaluation metrics. First, for experiments on synthetic data, we observe a mean absolute position error of 0.21mm and a mean absolute orientation error of 2.8° compared with ground truth. Second, for phantom data, we observe a mean absolute position error of 0.97mm and a mean absolute orientation error of 3.6° compared with the same motion tracked by an electromagnetic tracker. Third, for cadaver data, we use fiducial landmarks and observe an average reprojection distance error of 0.82mm. Finally, for in vivo clinical data, we report an average ICP residual error of 0.88mm in areas that are not composed of erectile tissue and an average ICP residual error of 1.09mm in areas that are composed of erectile tissue.

Surgeons often manually palpate tissue or organs in order to find tumors or other anatomical structures. Information about organ geometry and tissue stiffness gained from palpation can also be extremely useful in robotic surgery for diagnosis, surgical guidance, and registration to other preoperative information. However, it is not always easy to obtain, even if the robot is equipped with force sensors. This paper reports our approach for concurrent estimation of stiffness and surface geometry, using a continuous motion similar to a sweeping palpation motion used by surgeons. Our method relies on force data captured by a tactile sensor rigidly attached to an end-effector probe. We use Gaussian processes to simultaneously estimate geometry and stiffness. The method is not tied to any specific robotic platform and is consistent with a variety of palpation strategies. For simplicity, we discuss the results based on two different palpation primitives. This is our first step towards developing an adaptive high fidelity model reconstruction and path optimization technique.

Lesions of the lateral skull base in the petrous apex present unique surgical challenges because of the proximity of critical structures, including the inner ear, carotid artery, jugular bulb, facial nerve, lower cranial nerves, dura and brain. Currently, there are few appropriate surgical devices that can reach and remove these lesions, each with their own disadvantages. Here we investigate the feasibility of a dexterous continuum manipulator (DCM) capable of C-& S-shaped bends enabling dissection with remote center of motion (RCM) deep to the intact inner ear. A dedicated borescope channel provides the necessary visualization, while a flexible ring curette pre-shaped with a nitinol strip is designed to work through the instrument lumen for curettage of a cystic lesion. The kinematics of the DCM with the ring curette subject to an RCM constraint are investigated to explore the boundaries of a typical cyst cavity. Experiments in the planar phantom are carried out to validate feasibility, and results show that the proposed solution is practicable, accomplishing 80% and 83% removal of cysts for two kinds of boundaries.

Suturing is a challenging and highly dexterous task in minimally invasive surgery, even with the assistance of robotic surgical systems. In this work, we propose a simple yet versatile impedance virtual fixture framework, which can be applied on the master manipulator in a tele-operated robotic surgical system. With this framework, we further develop two types of virtual fixtures that assist with the needle passing and knot tying sub-tasks in suturing. The paper also presents the results of a 14-participant user study for both needle passing and knot tying sub-tasks, showing that virtual fixture assistance for novice users increases the needle passing exit point accuracy, reduces the number of errors (suture slip) in knot tying, and simultaneously decreases the task completion time and overall operator workload.

Functional endoscopic sinus surgery (FESS) is a surgical procedure used to treat acute cases of sinusitis and other sinus diseases. FESS is fast becoming the preferred choice of treatment due to its minimally invasive nature. However, due to the limited field of view of the endoscope, surgeons rely on navigation systems to guide them within the nasal cavity. State of the art navigation systems report registration accuracy of over 1mm, which is large compared to the size of the nasal airways. We present an anatomically constrained video-CT registration algorithm that incorporates multiple video features. Our algorithm is robust in the presence of outliers. We also test our algorithm on simulated and in-vivo data, and test its accuracy against degrading initializations. These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Retinal surgery is one of the most technically challenging surgical disciplines. Many robotic systems have been developed to enhance the surgical capabilities. However, very few of them provide the surgeon the dexterity within the patient's eye to enable more flexible, more advanced surgical procedures. This paper presents a sub-millimeter intraocular dexterous robot, the Integrated Robotic Intraocular Snake (IRIS). The variable neutral-line mechanism is used to provide very high dexterity with a very small form factor. The IRIS distal dexterous unit is 0.9 mm in diameter and about 3 mm in length. It enables two rotational degrees of freedom at the distal end of the ophthalmic instruments. The analysis on contact mechanics provides a reference for the adjustment of the wire pretension. Redundant actuation is implemented by using one motor for each wire. A motion scaling transmission is developed to overcome the suboptimal resolution of the motors. A scale-up model of the IRIS is built for initial experimental evaluation. Preliminary results show that the scale-up IRIS can provide large range of motion. For given bending angle, the kinematic model can estimate the desired wire translation when the friction is not significant. The first prototype of the actual-scale IRIS is assembled and tested.

Shape sensing techniques utilizing Fiber Bragg grating (FBG) arrays can enable real-time tracking and control of dexterous continuum manipulators (DCM) used in minimally invasive surgeries. For many surgical applications, the DCM may need to operate with much larger curvatures than what current shape sensing methods can detect. This paper proposes a novel shape sensor, which can detect a radius of curvature of 15 mm for a 35 mm long DCM. For this purpose, we used FBG sensors along with nitinol wires as the supporting substrates to form a triangular cross section. For verification, we assembled the sensor inside the wall of the DCM. Experimental results indicate that the proposed sensor can detect the DCM's curvature with an average error of 3.14%.

In retinal microsurgery, membrane peeling is a standard procedure requiring the delamination of a thin fibrous membrane adherent to the retina surface by applying very small forces. Robotic devices with combined force-sensing instruments have significant potential to assist this procedure by facilitating membrane delamination through induced micro-vibrations. However, defining the optimal frequency and amplitude for generating such vibrations, and updating these parameters during the procedure is not trivial. Automatic adjustment of these parameters via an adaptive control scheme is possible only if the individual parameter effects on delamination behavior are known. This study presents an experimental exploration of how micro-vibration amplitude and frequency affect membrane peeling forces alone. Combining a micromanipulator and a force-sensing micro-forceps, several peeling experiments were done on artificial phantoms (bandages) and inner shell membrane of raw chicken eggs. In the tested range of micro-vibration frequencies (10-50 Hz) the average delamination force was minimized mostly at 30 Hz for the bandages and at 50 Hz for the egg membranes. Increasing the micro-vibration amplitude from 50 μm up to 150 μm provided further reduction in average force, thus facilitated membrane delamination.

In vitreoretinal practice, controlled tremor-free motion and limitation of applied forces to the retina are two highly desired features. This study addresses both requirements with a new integrated system: a force-sensing motorized micro-forceps combined with an active tremor-canceling handheld micromanipulator, known as Micron. The micro-forceps is a 20 Ga instrument that is mechanically decoupled from its handle and senses the transverse forces at its tip with an accuracy of 0.3 mN. Membrane peeling trials on a bandage phantom revealed a 60-95% reduction in the 2-20 Hz band in both the tip force and position spectra, while peeling forces remained below the set safety threshold.

Robotic systems have the potential to assist vitreoretinal surgeons in extremely difficult surgical tasks inside the human eye. In addition to reducing hand tremor and improving tool positioning, a robotic assistant can provide assistive motion guidance using virtual fixtures, and incorporate real-time feedback from intraocular force sensing ophthalmic instruments to present tissue manipulation forces, that are otherwise physically imperceptible to the surgeon. This paper presents the design of an FBG-based, multi-function instrument that is capable of measuring mN-level forces at the instrument tip located inside the eye, and also the sclera contact location on the instrument shaft and the corresponding contact force. The given information is used to augment cooperatively controlled robot behavior with variable admittance control. This effectively creates an adaptive remote center-of-motion (RCM) constraint to minimize eye motion, but also allows the translation of the RCM location if the instrument is not near the retina. In addition, it provides force scaling for sclera force feedback. The calibration and validation of the multi-function force sensing instrument are presented, along with demonstration and performance assessment of the variable admittance robot control on an eye phantom.

A variety of medical and surgical robot systems have been developed in academia and industry and commercial products are actively used in modern operating rooms. However, there is no safety standard that specifically governs the design of medical robot systems. Despite the availability of several safety design guidelines, the absence of a basis or foundation for safety makes it difficult to describe safety designs in a systematic manner, and to share knowledge and experiences on safety with others. In the meantime, the scale and complexity of recent medical robot systems have been increasing and this further complicates the effective representation and sharing of safety designs. As an approach to this issue, we propose the Safety Design View, a conceptual framework that can capture and describe both the design-time and run-time characteristics of safety features of medical robot systems in a systematic and structured manner. To illustrate the application of the Safety Design View, we collected a set of frequently used safety features, based on our literature review of safety in the medical robotics domain, and show how we can more effectively describe and understand safety designs of medical robot systems.

We present a telerobotics research platform that provides complete access to all levels of control via open-source electronics and software. The electronics employs an FPGA to enable a centralized computation and distributed I/O architecture in which all control computations are implemented in a familiar development environment (Linux PC) and low-latency I/O is performed over an IEEE-1394a (FireWire) bus at speeds up to 400 Mbits/sec. The mechanical components are obtained from retired first-generation da Vinci ® Surgical Systems. This system is currently installed at 11 research institutions, with additional installations underway, thereby creating a research community around a common open-source hardware and software platform.

Particles such as thrombi, bullet fragments, and shrapnel can become trapped in a person's heart after migrating through the venous system, or by direct penetration. These cardiac foreign bodies pose a serious health risk as they can interfere with cardiovascular function. Conventional treatment often requires open heart surgery, cardiopulmonary bypass, and a long incision of the heart muscle, which come with significant risk and recovery time. To circumvent these disadvantages, we propose a minimally invasive surgical approach using 3D ultrasound to guide a dexterous robotic capture device. Analysis of the foreign body trajectory indicates highly erratic motion, rendering a robotic retrieval strategy based on direct pursuit of the tracked target infeasible. To provide a relatively slow robot with the ability to retrieve such a target, we propose alternative strategies based on guiding a robot to a salient capture location, and ambushing the target upon its reappearance. In this paper, we demonstrate the use of 3D transesophageal echocardiography (TEE) in tracking a foreign body in a beating heart phantom, computing a suitable capture location, and guiding a high dexterity robot to secure the target.

We have recently developed a snake-like manipulator for use in orthopaedic environments. One example application is the treatment of osteolysis (bone degradation) due to total hip arthroplasty. Recent literature suggest constant curvature models to define manipulator configuration from string (or actuator cable) length; however, our manipulator does not conform to constant curvature bending. In this paper, we present a two-step model to predict the kinematic configuration directly from string length with no assumptions regarding constant curvature bending. We experimentally identify the model parameters and validate the model on an additional experimental data set. The results indicate our model achieved an average maximum error of 1.0 ± 0.90mm in predicting manipulator configuration compared to the ground truth over the test data set.

This paper presents an implementation and evaluation of the Robotic ENT Microsurgery System (REMS). The implementation is discussed in reference to analysis from previous work, and evaluated using a simulated surgical task designed to resemble microlaryngeal phonosurgery. Preliminary technical evaluations of resolution and accuracy are also presented. The results of the evaluations reveal that the system statistically significantly improves surgical precision (p < 0.01) with only a small increase in operating time. The force data recorded also reveals that operating force can be significantly affected by ergonomic factors, and that warnings for excessive force and workspace limits are needed.

Laser photocoagulation is currently the standard treatment for sight-threatening diseases worldwide, namely diabetic retinopathy and retinal vein occlusions. The slit lamp biomicroscope is the most commonly used device for this procedure, specially for the treatment of the eye periphery. However, only a small portion of the retina can be visualized through the biomicroscope, complicating the task of localizing and identifying surgical targets, increasing treatment duration and patient discomfort. In order to assist surgeons, we propose a method for creating intraoperative retina maps for view expansion using a slit-lamp device. Based on the mosaicking method described by Richa et al, 2012, the proposed method is a combination of direct and feature-based methods, suitable for the textured nature of the human retina. In this paper, we describe three major enhancements to the original formulation. The first is a visual tracking method using local illumination compensation to cope with the challenging visualization conditions. The second is an efficient pixel selection scheme for increased computational efficiency. The third is an entropy-based mosaic update method to dynamically improve the retina map during exploration. To evaluate the performance of the proposed method, we conducted several experiments on human subjects with a computer-assisted slit-lamp prototype. We also demonstrate the practical value of the system for photo documentation, diagnosis and intraoperative navigation.

In vitreoretinal surgery, application of excessive forces and unintentional motion due to hand-tremor can easily result in serious complications. Robotic assistance when combined with tool-to-tissue force sensing capabilities has significant potential to improve such practice. In this paper, we evaluate the membrane peeling performance of a single user for two distinct robotic systems with integrated force sensing capabilities: Micron and the Steady-Hand Robot. We show that these systems provide promising performance improvement with similar impact on peeling forces and comparable tremor cancellation trends.

Robotic assistants and smart surgical instruments have been developed to overcome many significant physiological limitations faced by vitreoretinal surgeons, one of which is lack of force perception below 7.5 mN. This paper reports the development of a new force sensor based on fiber Bragg grating (FBG) with the ability to sense forces at the tip of the surgical instrument located inside the eye and also provide information about instrument interaction with the sclera. The sclera section provides vital feedback for cooperative robot control to minimize potentially dangerous forces on the eye. Preliminary results with 2×2 degree-of-freedom (DOF) sensor and force scaling robot control demonstrate significant reduction of forces on the sclera. The design and analysis of the sensor is presented along with a simulated robot assisted retinal membrane peeling on a phantom with sclera constraints and audio feedback.

This paper reports the error specifications and error analysis for a new cooperatively controlled ENT microsurgery robot. The new robot is designed for three specific ENT surgeries, endonasal skull base surgery, transoral laryngeal surgery, and cochlear implant surgery. The design requirements for the robot resolution, accuracy, stiffness, and repeatability for these surgery types are discussed and analyzed, and the required robot parameters are calculated. The mechanical design of the robot is then analyzed and shown to fulfill these requirements in the case where it is in its home configuration. The analysis is then extended to cover the robot's whole workspace.

Methods for tracking an object have generally fallen into two groups: tracking by detection and tracking through local optimization. The advantage of detection-based tracking is its ability to deal with target appearance and disappearance, but it does not naturally take advantage of target motion continuity during detection. The advantage of local optimization is efficiency and accuracy, but it requires additional algorithms to initialize tracking when the target is lost. To bridge these two approaches, we propose a framework for unified detection and tracking as a time-series Bayesian estimation problem. The basis of our approach is to treat both detection and tracking as a sequential entropy minimization problem, where the goal is to determine the parameters describing a target in each frame. To do this we integrate the Active Testing (AT) paradigm with Bayesian filtering, and this results in a framework capable of both detecting and tracking robustly in situations where the target object enters and leaves the field of view regularly. We demonstrate our approach on a retinal tool tracking problem and show through extensive experiments that our method provides an efficient and robust tracking solution.

The safety of endoscopic skull base surgery can be enhanced by accurate navigation in preoperative computed tomography (CT) or, more recently, intraoperative cone-beam CT (CBCT). The ability to register real-time endoscopic video with CBCT offers an additional advantage by rendering information directly within the visual scene to account for intraoperative anatomical change. However, tracker localization error ( ∼ 1-2 mm ) limits the accuracy with which video and tomographic images can be registered. This paper reports the first implementation of image-based video-CBCT registration, conducts a detailed quantitation of the dependence of registration accuracy on system parameters, and demonstrates improvement in registration accuracy achieved by the image-based approach. Performance was evaluated as a function of parameters intrinsic to the image-based approach, including system geometry, CBCT image quality, and computational runtime. Overall system performance was evaluated in a cadaver study simulating transsphenoidal skull base tumor excision. Results demonstrated significant improvement in registration accuracy with a mean reprojection distance error of 1.28 mm for the image-based approach versus 1.82 mm for the conventional tracker-based method. Image-based registration was highly robust against CBCT image quality factors of noise and resolution, permitting integration with low-dose intraoperative CBCT.

A new type of deformable model is presented that merges meshes and level sets into one representation to provide interoperability between methods designed for either. This includes the ability to circumvent the CFL time step restriction for methods that require large step sizes. The key idea is to couple a constellation of disconnected triangular surface elements (springls) with a level set that tracks the moving constellation. The target application for Spring Level Sets (SpringLS) is to implement comprehensive imaging pipelines that require a mixture of deformable model representations to achieve the best performance. We demonstrate how to implement key components of a comprehensive imaging pipeline with SpringLS, including image segmentation, registration, tracking, and atlasing.

Highly accurate positioning is fundamental to the performance of vitreoretinal microsurgery. Of vitreoretinal procedures, membrane peeling is among the most prone to complications since extremely delicate manipulation of retinal tissue is required. Associated tool-to-tissue interaction forces are usually below the threshold of human perception, and the surgical tools are moved very slowly, within the 0.1-0.5 mm/s range. During the procedure, unintentional tool motion and excessive forces can easily give rise to vision loss or irreversible damage to the retina. A successful surgery includes two key features: controlled tremor-free tool motion and control of applied force. In this study, we present the potential benefits of a micro-force sensing robot in vitreoretinal surgery. Our main contribution is implementing fiber Bragg grating based force sensing in an active tremor canceling handheld micromanipulator, known as Micron, to measure tool-to-tissue interaction forces in real time. Implemented auditory sensory substitution assists in reducing and limiting forces. In order to test the functionality and performance, the force sensing Micron was evaluated in peeling experiments with adhesive bandages and with the inner shell membrane from chicken eggs. Our findings show that the combination of active tremor canceling together with auditory sensory substitution is the most promising aid that keeps peeling forces below 7 mN with a significant reduction in 2-20 Hz oscillations.

A new data structure is presented for geometrically modeling multi-objects. The model can exhibit elastic and fluid-like behavior to enable interpretability between tasks that require both deformable registration and active contour segmentation. The data structure consists of a label mask, distance field, and springls (a constellation of disconnected triangles). The representation has sub-voxel precision, is parametric, re-meshes, tracks point correspondences, and guarantees no self-intersections, air-gaps, or overlaps between adjacent structures. In this work, we show how to apply existing registration algorithms and active contour segmentation to the data structure; and as a demonstration, the data structure is used to segment cortical and subcortical structures (74 total) in the human brain.

An emerging topic is to build image segmentation systems that can segment hundreds to thousands of objects (i.e. cell segmentation\tracking, full brain parcellation, full body segmentation, etc.). Multi-object Level Set Methods (MLSM) perform this task with the benefit of sub-pixel precision. However, current implementations of MLSM are not as computationally or memory efficient as their region growing and graph cut counterparts which lack sub-pixel precision. To address this performance gap, we present a novel parallel implementation of MLSM that leverages the sparse properties of the algorithm to minimize its memory footprint for multiple objects. The new method, Multi-Object Geodesic Active Contours (MOGAC), can represent N objects with just two functions: a label mask image and unsigned distance field. The time complexity of the algorithm is shown to be O((M (power)d)/P) for M (power)d pixels and P processing units in dimension d = {2,3}, independent of the number of objects. Results are presented for 2D and 3D image segmentation problems.

A novel algorithm is presented to segment and reconstruct injected bone cement from a sparse set of X-ray images acquired at arbitrary poses. The sparse X-ray multi-view active contour (SxMAC-pronounced "smack") can 1) reconstruct objects for which the background partially occludes the object in X-ray images, 2) use X-ray images acquired on a noncircular trajectory, and 3) incorporate prior computed tomography (CT) information. The algorithm's inputs are preprocessed X-ray images, their associated pose information, and prior CT, if available. The algorithm initiates automated reconstruction using visual hull computation from a sparse number of X-ray images. It then improves the accuracy of the reconstruction by optimizing a geodesic active contour. Experiments with mathematical phantoms demonstrate improvements over a conventional silhouette based approach, and a cadaver experiment demonstrates SxMAC's ability to reconstruct high contrast bone cement that has been injected into a femur and achieve sub-millimeter accuracy with four images.

Surgeries of the skull base require accuracy to safely navigate the critical anatomy. This is particularly the case for endoscopic endonasal skull base surgery (ESBS) where the surgeons work within millimeters of neurovascular structures at the skull base. Today's navigation systems provide approximately 2 mm accuracy. Accuracy is limited by the indirect relationship of the navigation system, the image and the patient. We propose a method to directly track the position of the endoscope using video data acquired from the endoscope camera. Our method first tracks image feature points in the video and reconstructs the image feature points to produce 3D points, and then registers the reconstructed point cloud to a surface segmented from preoperative computed tomography (CT) data. After the initial registration, the system tracks image features and maintains the 2D-3D correspondence of image features and 3D locations. These data are then used to update the current camera pose. We present a method for validation of our system, which achieves submillimeter (0.70 mm mean) target registration error (TRE) results.

The 11 papers in this special issue represent different advances in interventional imaging.

Intraoperative patient registration may significantly affect the outcome of image-guided surgery (IGS). Image-based registration approaches have several advantages over the currently dominant point-based direct contact methods and are used in some industry solutions in image-guided radiation therapy with fixed X-ray gantries. However, technical challenges including geometric calibration and computational cost have precluded their use with mobile C-arms for IGS. We propose a 2D/3D registration framework for intraoperative patient registration using a conventional mobile X-ray imager combining fiducial-based C-arm tracking and graphics processing unit (GPU)-acceleration. The two-stage framework 1) acquires X-ray images and estimates relative pose between the images using a custom-made in-image fiducial, and 2) estimates the patient pose using intensity-based 2D/3D registration. Experimental validations using a publicly available gold standard dataset, a plastic bone phantom and cadaveric specimens have been conducted. The mean target registration error (mTRE) was 0.34 ± 0.04 mm (success rate: 100%, registration time: 14.2 s) for the phantom with two images 90° apart, and 0.99 ± 0.41 mm (81%, 16.3 s) for the cadaveric specimen with images 58.5° apart. The experimental results showed the feasibility of the proposed registration framework as a practical alternative for IGS routines.

A dexterous manipulator (DM) with a large open lumen is presented. The manipulator is designed for surgical applications with a preliminary focus on the removal of osteolysis formed behind the acetabular shell of primary total hip arthroplasties (THAs). The manipulator is constructed from two nested superelastic nitinol tubes enabling lengthwise channels for drive cables. Notches in the nested assembly provide reliable bending under applied cable tension producing kinematics that can be effectively modeled as a series of rigid vertebrae connected using pin joints. The manipulator is controlled in plane with two independently actuated cables in a pull-pull configuration. For the purpose of the procedure, the manipulator is mounted on a Z-θ stage adding a translational and rotational degree of freedom (DOF) along the axis of the manipulator. Preliminary experimental results demonstrate the initial modeling and control of the manipulator.

Commercial telerobotic surgery systems for soft tissue surgery are generally limited to visual imaging, though it is possible to simultaneously view picture-in-picture visualization of another workstation. However, it is not easy to correlate such information with the primary endoscopic view since it may not relate to the surface visible in the visual endoscopic images. As critical surfaces as well as surgical targets often lie subsurface, a range of techniques (e.g. ultrasound and near-infrared imaging) and registration methods have been investigated as robotic surgery gains popularity. While investigation of nerves, blood vessels, and tumors has received prior attention, we present a new prototype system for real-time multimodal image registration that focuses on the visualization of the urinary tract. By providing an accurate registration between stereo video images and a near infrared imager, we aim to enhance surgical awareness and make critical uretary tasks such as mobilization of the ureters easier. Engineering validation experiments with a prototype imager, and in-vivo experiments using a prototype Hamamatsu Photodynamic Eye (PDE) imager together with the da Vinci surgical system demonstrating feasibility are presented. These initial experiments have also shown encouraging response from the clinicians.

The goal of this paper is to introduce a direct visual tracking method based on an image similarity measure called the sum of conditional variance (SCV). The SCV was originally proposed in the medical imaging domain for registering multi-modal images. In the context of visual tracking, the SCV is invariant to non-linear illumination variations, multi-modal and computationally inexpensive. Compared to information theoretic tracking methods, it requires less iterations to converge and has a significantly larger convergence radius. The novelty in this paper is a generalization of the efficient second-order minimization formulation for tracking using the SCV, allowing us to combine the efficient second-order approximation of the Hessian with a similarity metric invariant to non-linear illumination variations. The result is a visual tracking method that copes with non-linear illumination variations without requiring the estimation of photometric correction parameters at every iteration. We demonstrate the superior performance of the proposed method through comparative studies and tracking experiments under challenging illumination conditions and rapid motions.

This paper presents a constrained optimization framework that enables the implementation of multi-robot constraints, as virtual fixtures, to assist human operators, in a teleoperated scenario. The collaborative constraints guide the motion of multiple robots such that the spatial and temporal relationships are maintained between them, while following human input motion objectives. We demonstrate this control architecture for the task of manipulating a surgical knot to a target point. The teleoperation system uses four arms from a da Vinci Surgical System® (two master manipulators and two slave manipulators), with custom electronics and software. It extends previous work, which focused on a cooperatively controlled system where the motions of two robots were directly controlled by user-applied forces. Our current system enables us to effectively evaluate the accuracy of the knot positioning task and completion time in a clinically realistic setup for Minimally Invasive Surgery

We propose a method for improving the quality of cone-beam tomographic reconstruction done with a C-arm. C-arm scans frequently suffer from incomplete information due to image truncation, limited scan length, or other limitations. Our proposed "hybrid reconstruction" method injects information from a prior anatomical model, derived from a subject-specific computed tomography (CT) or from a statistical database (atlas), where the C-arm X-ray data is missing. This significantly reduces reconstruction artifacts with little loss of true information from the X-ray projections. The methods consist of constructing anatomical models, fast rendering of digitally reconstructed radiograph (DRR) projections of the models, rigid or deformable registration of the model and the X-ray images, and fusion of the DRR and X-ray projections, all prior to a conventional filtered back-projection algorithm. Our experiments, conducted with a mobile image intensifier C-arm, demonstrate visually and quantitatively the contribution of data fusion to image quality, which we assess through comparison to a "ground truth" CT. Importantly, we show that a significantly improved reconstruction can be obtained from a C-arm scan as short as 90° by complementing the observed projections with DRRs of two prior models, namely an atlas and a preoperative same-patient CT. The hybrid reconstruction principles are applicable to other types of C-arms as well.

This paper reports the development of novel micro-force sensing tools for retinal microsurgery. Retinal microsurgery requires extremely delicate manipulation of retinal tissue, and tool-to-tissue interaction forces are frequently below human perceptual thresholds. Further, the interaction between the tool shaft and sclera makes accurate sensing of forces exerted on the retina very difficult with previously developed force sensing schemes, in which the sensor is located outside the eye. In the work reported here, we incorporate 160 µm Fiber Bragg Grating (FBG) strain sensors into the tool shaft to sense forces distal to the sclera. The sensor is applicable both with robotically manipulated and freehand tools. Preliminary results with a 1 degree-of-freedom (DOF) sensor have demonstrated 0.25 mN resolution, and work is underway to develop 2 and 3 DOF tools. The design and analysis of the force sensing tool is presented with preliminary testing data and some initial experiments using the tool with both freehand and robotic manipulation.

The impact of computer-integrated surgery (CIS) on medicine in the next 20 years will be as great as that of computer-integrated manufacturing on industrial production over the past 20 years. A novel partnership between human surgeons and machines, made possible by advances in computing and engineering technology, will overcome many of the limitations of traditional surgery.

In this work, we have extended the concept of constrained motion control of robots to surgical tasks that require multiple robots. We present virtual fixtures to guide the motion of multiple robots such that spatial and temporal relationship is maintained between them. At the same time, our algorithm keeps the surgeon in the control loop. Moreover, we show that these virtual fixtures allow bimanual tasks to be completed using input for a single robot. That is, the user requires only one hand to cooperatively control multiple robots. This reduces the cognitive load on the surgeon and makes multiple-robot setup for surgery more relevant. We demonstrate this architecture by using an example of manipulating a surgical knot to position it at a target point. Significant improvement is observed in the accuracy when bimanual virtual fixture assistance is provided. Moreover, the accuracy when using a single input from user is similar to the accuracy obtained from bimanual assistance.

Certain surgical procedures require a high degree of precise manual control within a very restricted area. Retinal surgeries are part of this group of procedures. During vitreoretinal surgery, the surgeon must visualize, using a microscope, an area spanning a few hundreds of microns in diameter and manually correct the potential pathology using direct contact, free hand techniques. In addition, the surgeon must find an effective compromise between magnification, depth perception, field of view, and clarity of view. Pre-operative images are used to locate interventional targets, and also to assess and plan the surgical procedure. This paper proposes a method of fusing information contained in pre-operative imagery, such as fundus and OCT images, with intra-operative video to increase accuracy in finding the target areas. We describe methods for maintaining, in real-time, registration with anatomical features and target areas using image processing. This registration allows us to produce information enhanced displays that ensure that the retinal surgeon is always in visual contact with his/her area of interest.

This paper describes the development and initial testing of a new and optimized version of a steady-hand manipulator for retinal microsurgery. In the steady-hand paradigm, the surgeon and the robot share control of a tool attached to the robot through a force sensor. The robot controller senses forces exerted by the operator on the tool and uses this information in various control modes to provide smooth, tremor-free, precise positional control and force scaling. The steady-hand manipulator reported here has been specifically designed with the unique constraints of retinal microsurgery in mind. In particular, the system makes use of a compact wrist design that places the bulk of the robot away from the operating field. The resulting system has high efficacy, flexibility and ergonomics while meeting the accuracy and safety requirements of microsurgery. We have now tested this robot on a biological model system and we report a protocol for reliably cannulating ~80 mum OD veins (the size of veins in the human retina) using the system

This paper describes a spatial-motion-constraints-generation approach for a human-machine collaborative surgical-assistant system from registered computer tomography models. We extend constrained optimization formulation incorporating task goals, anatomy-based constraints, "no fly zones," etc. We use a fast potential-collision-constraint-detection method based on a 3-D surface model and covariance tree data structure. These boundary constraints, along with task behaviors and joint limits, serve as constraint conditions for constrained robot control. We are able to follow a complex path inside a human skull, phantom represented by a surface model composed of 99 000 vertices and 182 000 triangles in real time. Our approach enables real-time task-based control of a surgical robot in a precise interactive minimally invasive surgery task. We illustrate our approach based on two example tasks which are analogous to the procedures in endoscopic sinus surgery, and analyze the user's performance on both teleoperation and cooperative control for one of the example tasks. The experimental results show that a robotic assistant employing our approach on spatial motion constraints can assist the user in skilled manipulation tasks, while maintaining desired properties. Our approach is equally applicable to teleoperative and cooperative controlled robots

This paper presents an approach to implement virtual fixtures for surgical robot assistants. Our approach uses a weighted, multi-objective (both linear and nonlinear) constrained optimization framework to formalize a library of virtual fixtures for task primitives. By our formulation, we provide a library of virtual fixtures on task primitives and a way to assemble multiple virtual fixture objects. We implement the constrained optimization problem with both linear and nonlinear constraints, and discuss the trade-offs between them. Moreover, we introduce the notion of soft virtual fixture mechanism for robotic surgical assistance. The soft virtual fixtures enable a surgical tool to have some resistance inside safety regions and no resistance in preferred regions. I. INTRODUCTION This paper presents an approach to implement virtual fix- tures for surgical robot assistants. Most robotic assisted sur- gical procedures are characterized by restricted access to the workspace as well as constrained manipulation of a surgical tool. In such cases, the surgeons' ability can be augmented by techniques such as virtual fixtures (VF). Virtual fixtures (1), which have been discussed previously in the literature for both telerobotic and cooperative robots, are algorithms which provide anisotropic behavior to surgeons' motion commands in addition to filtering out tremor to provide safety and precision. An important case of virtual fixtures is forbidden regions, where the surgical tool is restricted to certain regions in the workspace. Davies et al. (2) set active constraints to constrain the robot to cut the femur and tibia within a permitted region for prosthetic knee surgery. Park et al. (3) developed sensor- mediated virtual fixtures that constrain the robot's motion or create haptic feedback directing the surgeon to move the surgical instruments in a desired direction. The recent work by Bettini et al. on virtual fixtures (4) used admittance control laws to implement guidance virtual fixtures. These works are based either on a specific robot type or on a specific task. Path planning and motion control is a well discussed area with a wide variety of proposed optimality criteria (5), (6), (7), (8). Funda et al. (9) presented an optimal motion control method to control both redundant and deficient robotic systems in constrained working volumes. We extend Funda's work by applying the method to generate complicated virtual fixtures based on user input for surgical assistant robots. Typically, surgical tasks have a certain degree of uncertainty that arises from factors such as registration errors, variations in anatomy and changes during procedures. Consider an example task of placing a surgical tool at a point in space. Depending on the nature of the procedure, one can define a region and tool placement within this region that would lead to the expected outcome. This region could be on the order of a few microns for retinal vein cannulation or hundreds of microns for a biopsy procedure. Furthermore, we can define another region where the surgeon might deliberately want to place the instrument to account for some uncertainties inherent in surgical procedures. In other words, we would like to have some compliance in the virtual fixture, while maintaining a preferred motion. Therefore, we define 3 different regions: A) Preferred region: this region defines expected outcome. B) Safety region: the tool could temporarily be in this region for fulfilling some expected task. C) Forbidden region: The tool never could be here for safety purposes. The relationship of these three regions depends on the surgical task. Figure 1 shows two typical examples.

This paper provides an overview of medical robotics, from the perspective of a researcher who has been actively involved in the field for 17 years. Like all robot systems, medical robots fundamentally couple information to physical action to significantly enhance humans' ability to perform important tasks-in this case surgical interventions, rehabilitation, or simply helping handicapped people in daily living tasks. Research areas include modeling and analysis of anatomy and task environments, interface technology between the "data world" and the physical world, and study of how complex systems are put together. This paper will discuss these research areas and illustrate their interrelationship with application examples. Although the main focus will be on robotic systems for surgery, it will also discuss the relationship of these research areas to rehabilitation and assistance robots. Finally, it will include some thoughts on the factors driving the acceptance of medical robotics and of how research can be most effectively organized

This paper presents a novel method for volume rendering of unstructured grids. Previously, we introduced an algorithm for perspective-correct interpolation of barycentric coordinates and computing polynomial attenuation integrals for a projected tetrahedron using graphics hardware. Here, we enhance the algorithm by providing a simple and efficient method to compute the projected shape (silhouette) and tessellation of a tetrahedron, in perspective and orthographic projection models. Our tessellation algorithm is published here for the first time. Compared with works of other groups on rendering unstructured grids, the main contributions of this work are: 1) A new algorithm for finding the silhouette of a projected tetrahedron. 2) A method for interpolating barycentric coordinates and thickness on the faces of the tetrahedron. 3) Visualizing higher-order attenuation functions using GPU without preintegration. 4) Capability of applying shape deformations to a rendered tetrahedral mesh without significant performance loss. Our visualization model is independent of depth-sorting of the cells. We present imaging and timing results of our implementation, and an application in time-critical "2D-3D" deformable registration of anatomical models. We discuss the impact of using higher-order functions on quality and performance.

Our recent work on robotic surgical assistant systems has led us to develop a method to generate spatial motion constraints associated with complex 3D geometry for controlling a robot in a complicated working configuration. An application of specific interest is sinus surgery, in which complicated anatomic structure constrains the motion of the endoscope and other instruments, which are inserted through the nose into a sinus cavity. We have implemented this method in both hands-on cooperative operation and teleoperation control mode. We evaluate and compare user performance in these two operation modes. We show that cooperative operation is more intuitive for the user and easier to use. On the other hand, due to the robot stiffness of our current implementation, teleoperation mode shows more accurate. Both of these two robot-assisted modes significantly improve human’s performance compared to the totally freehand motion.

We describe a new method to generate virtual fixtures for surgical robot control which provide sophisticated ways to assist the surgeon. Different spatial motion constraints for human machine collaborative systems can be implemented by using this method if we know the required geometric constraints and the instantaneous kinematics of the robot. It is independent of manipulator types: teleoperative or cooperative controlled; admittance or impedance type. Our method uses weighted, linearized, multi-objective optimization framework to formalize a library of virtual fixtures for task primitives. We set the cost function based on the user's inputs, and set linearized subject function based on a combination of five basic geometric constraints. In this paper, we also illustrate the implementation for two sample tasks, which are useful for surgical applications, and provide the experimental results for these tasks.

This paper presents a novel method for computing simulated x-ray images, or DRRs (digitally reconstructed radiographs), of tetrahedral meshes with higher-order attenuation functions. DRRs are commonly used in computer assisted surgery (CAS), with the attenuation function consisting of a voxelized CT study, which is viewed from different directions. Our application of DRRs is in intra-operative "2D-3D" registration, i.e., finding the pose of the CT dataset given a small number of patient radiographs. We register 2D patient images with a statistical tetrahedral model, which encodes the CT intensity numbers as Bernstein polynomials, and includes knowledge about typical shape variation modes. The unstructured grid is more suitable for applying deformations than a rectilinear grid, and the higher-order polynomials provide a better approximation of the actual density than constant or linear models. The infra-operative environment demands a fast method for creating the DRRs, which we present here. We demonstrate this application through the creation and use of a deformable atlas of human pelvis bones. Compared with other works on rendering unstructured grids, the main contributions of this work are: 1) Simple and perspective-correct interpolation of the thickness of a tetrahedral cell. 2) Simple and perspective-correct interpolation of front and back barycentric coordinates with respect to the cell. 3) Computing line integrals of higher-order functions. 4) Capability of applying shape deformations and variations in the attenuation function without significant performance loss. The method does not depend on for pre-integration, and does not require depth-sorting of the visualized cells. We present imaging and timing results of implementing the algorithm, and discuss the impact of using higher-order functions on the quality of the result and the performance.

There has been increased interest in minimally invasive ablative treatments that typically require precise placement of the ablator tool to meet the predefined planning and lead to efficient tumor destruction. Standard ablative procedures involve free hand transcutaneous ultrasonography (TCUS) in conjunction with manual tool positioning. Unfortunately, existing TCUS systems suffer from many limitations and result in failure to identify nearly half of all treatable liver lesions. Freehand manipulation of the ultrasound (US) probe and ablator tool lacks the critical level of control, accuracy, stability, and guaranteed performance required for these procedures. Freehand US results in undefined gap distribution, anatomic deformation due to variable pressure from the sonographer's hand, and severe difficulty in maintaining optimal scanning position. In response to these limitations, we propose the use of a dual robotic arm system that manages both ultrasound manipulation and needle guidance. We report a prototype of the dual arm system and a comparative performance analysis between robotic vs. freehand systems, for both US scanning and needle placement in mechanical and animal tissue phantoms.

In Ear, Nose and Throat (ENT) surgery, the operating volume is very limited. This is especially true in sinus surgery, when the instrument passes through the nasal and sinus cavity to reach the pathological area. The nasal and sinus bones impose geometric constraints on the work volume. During the surgery, the surgeon needs to control the motion of the instrument tip to accomplish some delicate procedure; meanwhile he/she needs to avoid hitting anatomic constraints. In this paper, we present a method to assist the path following task in a constrained area. The system reads the user's force input and combines it with the planned tip-trajectory to create the tip motion constraints; meanwhile it generates the tool-shaft boundary constraints based on a 3-D geometric model. We map instrument tip motion and boundary information to joint displacements via robot kinematics, and then use a constrained quadratic optimization algorithm to compute the optimal set of corresponding joint displacements. In the preliminary study, we show that robot guidance using cooperative control and virtual fixtures derived from complex geometry can assist users in skilled manipulation tasks, while maintaining desirable properties such as collision avoidance and safety.

This work presents a design overview of a novel high DoF (degrees-of-freedom) system being developed for minimally invasive surgery of the throat. The system is designed to allow remote operation of 2-3 tools with high tip dexterity to enable suturing and soft-tissue manipulation while using the patient's mouth as the only entry port. The slave is a 34 DoF unit equipped with three snake-like distal dexterity units for surgical tool manipulation. Each of these units is a multi-backbone snakelike mechanism equipped with a detachable milli parallel manipulator allowing interchangeable tools to be used. The paper presents the outline of the kinematic analysis of the snake-like units and proposes one possible actuation redundancy resolution to allow further downsize scalability while reducing the risk of buckling of the primary backbone of the snake-like units. Finally, The work presents a first early experiment with a prototype of the snake-like unit.

We present an approach for scale recovery from monocular stereo images of an endoscopic camera with simultaneous registration to dense 3D surface models. We assume the camera motion to be unknown or at least uncertain. An example application is the registration of endoscope images to pre-operative CT scans that allows instrument navigation during surgical procedures. The application field is not restricted to the medical field. It can be extended to registration of monocular video images to laser-based surface reconstructions in, e.g., mobile navigation area or to autonomous aircraft navigation from topological surveys. A novel way for depth estimation from arbitrary camera motion is presented. In this paper, we focus on the robust initialization of the system and on the scale recovery for the reconstructed 3D point clouds with accurate registration to the candidate surfaces extracted from the CT data. We provide experimental validation of the algorithm with data obtained from our experiments with a phantom skull.

Deformable 2D-3D medical image registration is an essential technique in computer integrated surgery (CIS) to fuse 3D pre-operative data with 2D intra-operative data. Several factors may affect the accuracy of 2D-3D registration, including the number of 2D views, the angle between views, the view angle relative to anatomical objects, the co-registration error between views, the image noise, and the image distortion. In this paper, we investigate and assess the relationship between these factors and the accuracy of 2D-3D registration. We proposed a deformable 2D-3D registration method based on a statistical model. We conducted experiments using a hemi-pelvis model and simulated X-ray images. Some discussions are provided on how to improve the accuracy of 2D-3D registration based on our assessment.

This paper reports preliminary experiments with a cooperative robot system to augment single cell manipulation tasks. The JHU "Steady-Hand" robot configuration for cell manipulation is reported. Stable force control laws for the "Steady Hand" are revisited. Preliminary experiments validating stable insertion of a micropipette in a mouse embryo are detailed along with formulation of vision based tracking and augmentation. These preliminary experiments demonstrate promise of cooperative augmentation in single cell manipulation tasks.

In this paper, we present a framework of calibrating an electromagnetic tracker (Northern Digital's Aurora) using an accurate optical tracker (the Optotrak system, also from Northern Digital). First, registration methods for these two navigation systems are developed. Sub millimeter accuracy registration is achieved for both cases. We also address the latency between the two different trackers. The registration accuracy for dynamic acquired data is greatly improved after we compensate for the tracker latency. In our calibration approach, we sample the measurement field of the Aurora and compute the position and orientation error using the Optotrak measurements and previously computed registration results as "ground truth". Then we approximate the error field using Bernstein polynomials. Another comparative technique we use is to decompose the error space using KD tree, and then approximate each atomic cell with local interpolation. Experimental results show significant improvement in tracking accuracy for both position and orientation. Finally we discuss our future directions.

In this paper we present a novel technique for non-rigid medical image registration and correspondence finding based on a multiple-layer flexible mesh template matching technique. A statistical anatomical model is built in the form of a tetrahedral mesh, which incorporates both shape and density properties of the anatomical structure. After the affine transformation and global deformation of the model are computed by optimizing an energy function, a multiple-layer flexible mesh template matching is applied to find the vertex correspondence and achieve local deformation. The multiple-layer structure of the template can be used to describe different scale of anatomical features; furthermore, the template matching is flexible which makes the correspondence finding robust. A leave-one-out validation has been conducted to demonstrate the effectiveness and accuracy of our method.

This paper reports the development of a new miniature force sensor designed to measure contact forces at the tip of a microsurgical instrument in three dimensions, and its application to scaled force feedback using a cooperatively manipulated microsurgical assistant robot. The principal features of the sensor are its small size of 12.5 mm in diameter and 15 mm in height, a novel configuration of flexure beams and strain gauges in order to measure forces isotropically at the instrument tip 40 mm from the sensor body, and sub-mN three-axis force-sensing resolution.

This paper provides a broad overview of medical robot systems used in surgery. After introducing basic concepts of computer-integrated surgery, surgical CAD/CAM, and surgical assistants, it discusses some of the major design issues particular to medical robots. It then illustrates these issues and the broader themes introduced earlier with examples of current surgical CAD/CAM and surgical assistant systems. Finally, it provides a brief synopsis of current research challenges and closes with a few thoughts on the research/industry/clinician teamwork that is essential for progress in the field.

This paper presents a new approach to image-based guidance of a needle or surgical tool during percutaneous procedures. The method is based on visual servoing. It requires no prior calibration or registration. The technique provides highly precise 3D-alignment of the tool with respect to an anatomic target. By taking advantage of projective geometry and projective invariants, this can be achieved in a fixed number (12) of iterations. In addition the approach estimates the required insertion depth. Experiments include automatic 3D alignment and insertion of a needle held by a medical robot into a pig kidney under X-ray fluoroscopy.

Reports preliminary experiments with a robot system designed to cooperatively extend a human's ability to perform fine manipulation tasks requiring human judgement, sensory integration and hand-eye coordination. A completed steady-hand robot is reported. A stable force control law is reviewed. Preliminary experiments validate theoretical predictions of stable one-dimensional control of tool-tip forces in contact with both linearly and nonlinearly compliant objects. Preliminary feasibility experiments demonstrate stable one-dimensional robotic augmentation and "force scaling" of a human operator's tactile input.

We have been studying brachytherapy planning with the objective of minimizing the maximum deviation of the delivered dose from prescribed dose bounds for treatment volumes. A general framework for optimal treatment planning is presented and the minmax optimization is formulated as a linear program. Dose rate calculations are based on the dosimetry formulation of the American Association of Physicists in Medicine, Task Group 43. We apply the technique to optimal planning for intravascular brachytherapy of intimal hyperplasia using ultrasound data and 192Ir seeds. The planning includes determination of an optimal dwell-time sequence for a train of seeds that deliver radiation while stepping through the vessel lesion. The results illustrate the advantage of this strategy over the common approach of delivering radiation by positioning a single train of seeds along the whole lesion.

This paper describes an ongoing project to develop a computer-integrated system to assist surgeons in revision total hip replacement (RTHR) surgery. In RTHR surgery, a failing orthopedic hip implant, typically cemented, is replaced with a new one by removing the old implant, removing the cement and fitting a new implant into an enlarged canal broached in the femur. RTHR surgery is a difficult procedure fraught with technical challenges and a high incidence of complications. The goals of the computer-based system are the significant reduction of cement removal labor and time, the elimination of cortical wall penetration and femur fracture, the improved positioning and fit of the new implant resulting from precise, high-quality canal milling and the reduction of bone sacrificed to fit the new implant. Our starting points are the ROBODOC system for primary hip replacement surgery and the manual RTHR surgical protocol. We first discuss the main difficulties of computer-integrated RTHR surgery and identify key issues and possible solutions. We then describe possible system architectures and protocols for preoperative planning and intraoperative execution. We present a summary of methods and preliminary results in CT image metal artifact removal, interactive cement cut-volume definition and cement machining, anatomy-based registration using fluoroscopic X-ray images and clinical trials using an extended RTHR version of ROBODOC. We conclude with a summary of lessons learned and a discussion of current and future work.

We describe new methods for rigid registration of a preoperative computed tomography (CT)-scan image to a set of intraoperative X-ray fluoroscopic images, for guiding a surgical robot to its trajectory planned from CT. Our goal is to perform the registration, i.e., compute a rotation and translation of one data set with respect to the other to within a prescribed accuracy, based upon bony anatomy only, without external fiducial markers. With respect to previous approaches, the following aspects are new: 1) we correct the geometric distortion in fluoroscopic images and calibrate them directly with respect to the robot by affixing to it a new calibration device designed as a radiolucent rod with embedded metallic markers, and by moving the device along two planes, while radiographs are being acquired at regular intervals; 2) the registration uses an algorithm for computing the best transformation between a set of lines in three space, the (intraoperative) X-ray paths, and a set of points on the surface of the bone (imaged preoperatively), in a statistically robust fashion, using the Cayley parameterization of a rotation; and 3) to find corresponding sets of points to the X-ray paths on the surfaces, our new approach consists of extracting the surface apparent contours for a given viewpoint, as a set of closed three-dimensional nonplanar curves, before registering the apparent contours to X-ray paths. Aside from algorithms, there are a number of major technical difficulties associated with engineering a clinically viable system using anatomy and image-based registration. To detect and solve them, we have so far conducted two experiments with the surgical robot in an operating room (OR), using CT and fluoroscopic image data of a cadaver bone, and attempting to faithfully simulate clinical conditions. Such experiments indicate that intraoperative X-ray-based registration is a promising alternative to marker-based registration for clinical use with our proposed method.

This paper addresses the problem of optimal motion control for teleoperated surgical robots, which must maneuver in constrained workspaces, often through a narrow entry portal into the patient's body. The control problem is determining how best to use the available degrees of freedom of a surgical robot to accomplish a particular task, while respecting geometric constraints on the work volume, robot mechanism, and the specific task requirements. We present a method of formulating desired motions as sets of task goals in any number of coordinate frames (task frames) relevant to the task, optionally subject to additional linear constraints in each of the task frames. Mathematically, the kinematic control problem is posed as a constrained quadratic optimization problem and is shown to be computable in real time on a PC. We will present experimental results of the application of this control methodology to both kinematically deficient and kinematically redundant robots. Specifically, we will discuss the control issues within the context of a representative set of tasks in robot-assisted laparoscopy, which includes (but is not limited to) teleoperated navigation of a laparoscopic camera attached to a surgical robot. A system based on this control formalism has been used in preclinical in vivo studies at the Johns Hopkins University Medical Center and the early experience with the system will be summarized.

This paper describes the control and ergonomic evaluation of a ceiling mounted (or support frame suspended) 7-axis surgical robot (HISAR) for laparoscopic camera navigation. A key feature of the robot is that it incorporates a passive wrist for natural compliance with the port of entry into the patient. The use of a previously reported constrained Cartesian controller is motivated and demonstrated, and the results of successfully applying this control methodology to the manipulator are presented. The significance of the control strategy is the ease with which control of passive axes, the fulcrum constraint, and the motion inversion effect created by the fulcrum are accommodated. We also report on the results of laboratory evaluations of the arm in terms of its work volume, ergonomic factors, ease of control, and overall design within the context of laparoscopic camera control.

The authors have developed an image-directed robotic system to augment the performance of human surgeons in precise bone machining procedures in orthopaedic surgery, initially targeted at cementless total hip replacement surgery. The total system consists of an interactive CT-based presurgical planning component and a surgical system consisting of a robot, redundant motion monitoring, and man-machine interface components. In vitro experiments conducted with this system have demonstrated an order-of-magnitude improvement in implant fit and placement accuracy, compared to standard manual preparation techniques. The first generation system described in this paper was used in a successful veterinary clinical trial on 26 dogs needing hip replacement surgery. It was the basis for subsequent development of a second-generation system that is now in human clinical trials.>

The authors describe the use of force feedback in a surgical robot system (ROBODOC). The application initially being addressed is total hip replacement (THR) surgery, where the robot must prepare a cavity in the femur for an artificial implant. In this system, force feedback is used to provide safety, tactile search capabilities, and an improved man-machine interface. Output of the force sensor is monitored by a safety processor, which initiates corrective action if any of several application-defined thresholds are exceeded. The robot is able to locate objects using guarded moves and force control (ball-in-cone strategy). In addition, the force control algorithm provides an intuitive man-machine interface which allows the surgeon to guide the robot by leading its tool to the desired location. An application of force control currently under development is described, where the force feedback is used to modify the cutter feed rate (force controlled velocity).>

The authors describe a robotic surgical system that has been designed to create femoral cavities that are precisely shaped and positioned for implantation of uncemented prostheses. This robotics system creates cavities with a dimensional accuracy more than 50 times greater than broached cavities, exceeds the tolerances to which implants are manufactured, and does not produce gaps that prevent bone ingrowth. A canine study was undertaken to evaluate the prosthesis fit and placement achieved by employing a surgical robot to prepare the femur. This study compared the results achieved on 15 dogs undergoing total hip replacement with manual broaching techniques and 25 dogs undergoing robotically assisted surgery. Among the 25 dogs, which ranged in age from 2/sup 1///sub 2/ to 11 years, there were no deaths, no infections, and no intraoperative complications. Human applications of this technique are also considered.>

Three-dimensional modeling of rotations and translations in robot kinematics is most commonly performed using homogeneous transforms. An alternate approach, using quaternion-vector pairs as spatial operators, is compared with homogeneous transforms in terms of computational efficiency and storage economy. The conclusion drawn is that quaternion-vector pairs are as efficient as, more compact than, and more elegant than their matrix counterparts. A robust algorithm for converting rotational matrices into equivalent unit quaternions is described, and an efficient quaternion-based inverse kinematics solution for the Puma 560 robot arm is presented.>

This paper discusses an environment for configuration, programming, and control of robot workcells. The controller is intended to support research in automation programming and motion control, and to provide a vehicle for conveniently integrating new sensors and other devices into a workcell in a useful way. The system consists of an interactive Programming System connected through a shared memory to a multiple-processor Real Time System that performs time-critical operations. The Programming System executes programs written in an enhanced version of AML and transmits high level commands, called verbs, to the Real Time System for execution. Verbs may either be simple, consisting essentially of a process specification and termination conditions, or they may be compositions of other verbs. The processes themselves are specified in terms of lower level entities called real time application subroutines, state vector variables, and data flow graphs which describe computations to be performed in the Real Time System.

Manipulator programs in a high-level language consist of manipulation procedures and object model declarations. As higher level languages are developed, the procedures will shrink while the declarations will grow. This trend makes it desirable to develop means for automating the generation of these declarations. A system is proposed which would permit users to specify certain object models interactively, using the manipulator itself as a measuring tool in three dimensions. A preliminary version of the system has been tested.