Surgical navigation has proven to be an effective approach for providing real-time guidance and visualization of relevant information by estimating the pose of the patient’s anatomy and surgical tools. During navigated surgery, instruments are commonly equipped with fiducial markers and tracked by stationary optical tracking systems (OTS) to provide accurate navigation cues. Augmented Reality (AR) has been adopted for intuitive visual guidance, even motivating several efforts to enable surgical instrument tracking through built-in sensors on Head-Mounted Displays (HMDs). However, existing tracking methods typically require a direct line-of-sight to instruments, which is challenging to maintain in dynamic surgical environments due to frequent occlusions caused by moving medical equipment, surgical tools, and personnel. To address this challenge, this work introduces a novel framework capable of tracking surgical instruments even under occlusion by fusing different sensors in a dynamic scene graph representation. Our framework uniquely combines tracking systems with varying degrees of accuracy, providing real-time assessments of tracking reliability to the user. Unlike conventional sensor fusion approaches that are heavily dependent on specific sensor modalities, the proposed method is agnostic to tracking device modality and robust to their motion states (e.g., stationary OTS versus dynamic AR-HMD). Experimental results demonstrate that our dynamic scene graph framework successfully integrates and optimizes measurements from multiple tracking sources, significantly enhancing AR visualization consistency and accuracy with robustness under occlusion.

The segmentation of pelvic fracture fragments in CT and X-ray images is crucial for trauma diagnosis, surgical planning, and intraoperative guidance. However, accurately and efficiently delineating the bone fragments remains a significant challenge due to complex anatomy and imaging limitations. The PENGWIN challenge, organized as a MICCAI 2024 satellite event, aimed to advance automated fracture segmentation by benchmarking state-of-the-art algorithms on these complex tasks. A diverse dataset of 150 CT scans was collected from multiple clinical centers, and a large set of simulated X-ray images was generated using the DeepDRR method. Final submissions from 16 teams worldwide were evaluated under a rigorous multi-metric testing scheme. The top-performing CT algorithm achieved an average fragment-wise intersection over union (IoU) of 0.930, demonstrating satisfactory accuracy. However, in the X-ray task, the best algorithm achieved an IoU of 0.774, which is promising but not yet sufficient for intra-operative decision-making, reflecting the inherent challenges of fragment overlap in projection imaging. Beyond the quantitative evaluation, the challenge revealed methodological diversity in algorithm design. Variations in instance representation, such as primary-secondary classification versus boundary-core separation, led to differing segmentation strategies. Despite promising results, the challenge also exposed inherent uncertainties in fragment definition, particularly in cases of incomplete fractures. These findings suggest that interactive segmentation approaches, integrating human decision-making with task-relevant information, may be essential for improving model reliability and clinical applicability.

Data scarcity has long been an issue in the robot learning community. Particularly, in safety-critical domains like surgical applications, obtaining high-quality data can be especially difficult. It poses challenges to researchers seeking to exploit recent advancements in reinforcement learning and imitation learning, which have greatly improved generalizability and enabled robots to conduct tasks autonomously. We introduce dARt Vinci, a scalable data collection platform for robot learning in surgical settings. The system uses Augmented Reality (AR) hand tracking and a high-fidelity physics engine to capture subtle maneuvers in primitive surgical tasks: By eliminating the need for a physical robot setup and providing flexibility in terms of time, space, and hardware resources-such as multiview sensors and actuators-specialized simulation is a viable alternative. At the same time, AR allows the robot data collection to be more egocentric, supported by its body tracking and content overlaying capabilities. Our user study confirms the proposed system’s efficiency and usability, where we use widely-used primitive tasks for training teleoperation with da Vinci surgical robots. Data throughput improves across all tasks compared to real robot settings by 41% on average. The total experiment time is reduced by an average of 10%. The temporal demand in the task load survey is improved. These gains are statistically significant. Additionally, the collected data is over 400 times smaller in size, requiring far less storage while achieving double the frequency. The source code for this project can be accessed at https://dartvinci.finite-state.com/.

Imitation learning frameworks for robotic manipulation have drawn attention in the recent development of language model grounded robotics. However, the success of the frameworks largely depends on the coverage of the demonstration cases: When the demonstration set does not include examples of how to act in all possible situations, the action may fail and can result in cascading errors. To solve this problem, we propose a framework that uses serialized Finite State Machine (FSM) to generate demonstrations and improve the success rate in manipulation tasks requiring a long sequence of precise interactions. To validate its effectiveness, we use environmentally evolving and long-horizon puzzles that require long sequential actions. Experimental results show that our approach achieves a success rate of up to 98% in these tasks, compared to the controlled condition using existing approaches, which only had a success rate of up to 60%, and, in some tasks, almost failed completely. The source code for this project can be accessed at https://imitate.finite-state.com/.

Total Body Photography (TBP) is becoming a useful screening tool for patients at high risk for skin cancer. While much progress has been made, existing TBP systems can be further improved for automatic detection and analysis of suspicious skin lesions, which is in part related to the resolution and sharpness of acquired images. This paper proposes a novel shape-aware TBP system automatically capturing full-body images while optimizing image quality in terms of resolution and sharpness over the body surface. The system uses depth and RGB cameras mounted on a 360-degree rotary beam, along with 3D body shape estimation and an in-focus surface optimization method to select the optimal focus distance for each camera pose. This allows for optimizing the focused coverage over the complex 3D geometry of the human body given the calibrated camera poses. We evaluate the effectiveness of the system in capturing high-fidelity body images. The proposed system achieves an average resolution of 0.068 mm/pixel and 0.0566 mm/pixel with approximately 85% and 95% of surface area in-focus, evaluated on simulation data of diverse body shapes and poses as well as a real scan of a mannequin respectively. Furthermore, the proposed shape-aware focus method outperforms existing focus protocols (e.g. auto-focus). We believe the high-fidelity imaging enabled by the proposed system will improve automated skin lesion analysis for skin cancer screening.

Robotic-assisted medical systems (RAMS) have gained significant attention for their advantages in alleviating surgeons’ fatigue and improving patients’ outcomes. These systems comprise a range of human-computer interactions, including medical scene monitoring, anatomical target planning, and robot manipulation. However, despite its versatility and effectiveness, RAMS demands expertise in robotics, leading to a high learning cost for the operator. In this work, we introduce a novel framework using mixed reality technologies to ease the use of RAMS. The proposed framework achieves real-time planning and execution of medical instruments by providing 3D anatomical image overlay, human-robot collision detection, and robot programming interface. These features, integrated with an easy-to-use calibration method for head-mounted display, improve the effectiveness of human-robot interactions. To assess the feasibility of the framework, two medical applications are presented in this work: 1) coil placement during transcranial magnetic stimulation and 2) drill and injector device positioning during femoroplasty. Results from these use cases demonstrate its potential to extend to a wider range of medical scenarios.

Hand-eye calibration is the problem of solving the transformation from the end-effector of a robot to the sensor attached to it. Commonly employed techniques, such as AXXB or AXZB formulations, rely on regression methods that require collecting pose data from different robot configurations, which can produce low accuracy and repeatability. However, the derived transformation should solely depend on the geometry of the end-effector and the sensor attachment. We propose Geometry-Based End-Effector Calibration (GBEC) that enhances the repeatability and accuracy of the derived transformation compared to traditional hand-eye calibrations. To demonstrate improvements, we apply the approach to two different robot-assisted procedures: Transcranial Magnetic Stimulation (TMS) and femoroplasty. We also discuss the generalizability of GBEC for camera-in-hand and marker-in-hand sensor mounting methods. In the experiments, we perform GBEC between the robot end-effector and an optical tracker’s rigid body marker attached to the TMS coil or femoroplasty drill guide. Previous research documents low repeatability and accuracy of the conventional methods for robot-assisted TMS hand-eye calibration. Applying GBEC to repeated calibrations, we obtain transformations with standard deviations of 0.37mm, 0.65mm, and 0.40mm (translation) along x, y, and z axes of the end-effector, respectively. The tool alignment experiments after using GBEC achieve a mean accuracy around 0.2mm in Euclidean distance. When compared to some existing methods, the proposed method relies solely on the geometry of the flange and the pose of the rigid-body marker, making it independent of workspace constraints or robot accuracy, without sacrificing the orthogonality of the rotation matrix. Our results validate the accuracy and applicability of the approach, providing a new and generalizable methodology for obtaining the transformation from the end-effector to a sensor.

Continuum Dexterous Manipulators (CDMs) are well-suited tools for minimally invasive surgery due to their inherent dexterity and reachability. Nonetheless, their flexible structure and non-linear curvature pose significant challenges for shape-based feedback control. The use of Fiber Bragg Grating (FBG) sensors for shape sensing has shown great potential in estimating the CDM’s tip position and subsequently reconstructing the shape using optimization algorithms. This optimization, however, is under-constrained and may be ill-posed for complex shapes, falling into local minima. In this work, we introduce a novel method capable of directly estimating a CDM’s shape from FBG sensor wavelengths using a deep neural network. In addition, we propose the integration of uncertainty estimation to address the critical issue of uncertainty in neural network predictions. Neural network predictions are unreliable when the input sample is outside the training distribution or corrupted by noise. Recognizing such deviations is crucial when integrating neural networks within surgical robotics, as inaccurate estimations can pose serious risks to the patient. We present a robust method that not only improves the precision upon existing techniques for FBG-based shape estimation but also incorporates a mechanism to quantify the models’ confidence through uncertainty estimation. We validate the uncertainty estimation through extensive experiments, demonstrating its effectiveness and reliability on out-of-distribution (OOD) data, adding an additional layer of safety and precision to minimally invasive surgical robotics.

Realtime shape estimation of continuum objects and manipulators is essential for developing accurate planning and control paradigms. The existing methods that create dense point clouds from camera images, and/or use distinguishable markers on a deformable body have limitations in realtime tracking of large continuum objects/manipulators. The physical occlusion of markers can often compromise accurate shape estimation. We propose a robust method to estimate the shape of linear deformable objects in realtime using scattered and unordered key points. By utilizing a robust probability-based labeling algorithm, our approach identifies the true order of the detected key points and then reconstructs the shape using piecewise spline interpolation. The approach only relies on knowing the number of the key points and the interval between two neighboring points. We demonstrate the robustness of the method when key points are partially occluded. The proposed method is also integrated into a simulation in Unity for tracking the shape of a cable with a length of 1m and a radius of 5mm. The simulation results show that our proposed approach achieves an average length error of 1.07% over the continuum’s centerline and an average cross-section error of 2.11mm. The real-world experiments of tracking and estimating a heavy-load cable prove that the proposed approach is robust under occlusion and complex entanglement scenarios.

Architected materials derive performance characteristics from material properties and internal geometry. These materials are increasingly prevalent across a wide variety of domains. Many intricate feature geometries associated with architected materials can be explored using additive manufacturing (AM) processes. However, current AM methods generally cannot fabricate geometries with completely closed voids without introducing a support structure. This paper describes a new, support-free approach to AM capable of creating structures with closed voids. This work limits part geometry to three-dimensional (3D) geometries defined by a revolution about a single axis. This limitation enables planar analysis within a three-degree-of-freedom (3-DoF) task space. Part geometry in 3-DoF task space is constrained to a convex arch. Task space geometry is divided into an ordered set of sub-regions, considering feasible deposition orientations and collision constraints. The use of 3-DoF task space provides planar translation and rotation of the component during fabrication. The introduction of this rotational DoF addresses AM overhang constraints imposed by gravity. Methods for generating, ordering, and layering sub-regions suitable for printing a part with a closed hole are presented. Layers derived in the 3-DoF task space analysis are then extended to 3D deposition paths using the axis of revolution defined by the original part. The method of hole closure relies on the concept of a "keystone" which requires a 45° nozzle offset for collision-free deposition within keystone-adjacent sub-regions. The feasibility of deposition using a 45° nozzle offset is explored experimentally, and results demonstrate feasibility.

In Augmented Reality, a robust virtual-to-real calibration that aligns the virtual and real spaces is crucial to ensure accurate overlays. Inspired by popular methods in robotics, we propose establishing the virtual-to-real calibration as a hand-eye/robot-world calibration problem using an $A X=Y B$ formulation. This formulation uses both the self-localization of a head-mounted display and the tracking functionality of an external tracking system. Additional techniques are also provided to address the data synchronization issue between the two measurement systems. To further improve the results, we integrate a post-acquisition outlier filter based on the $A X-Y B$ Frobenius norm. Improvements resulting from the filter were first validated by simulation. For the assessment of the complete pipeline, both subjective evaluation based on human perception and objective evaluation based on computer vision methods were used. The results show that the proposed method is accurate and noise-resistant. With only 100 measurement samples collected, the overlay of a tracked object at the farthest distance (1300 mm) in front of the tracking system has an average rotation error of $1.77 \pm 0.54^{\circ}$ and an average translation error of $4.82 \pm 1.71 \mathrm{~mm}$.

Arthroscopy is a minimally invasive orthopedic procedure commonly used to treat joints such as the shoulder, hip, or knee. A major difficulty for surgeons in arthroscopic procedures is the simultaneous coordination of the surgical tools used for manipulation and the arthroscopic camera. To further complicate this task, the narrow space where arthroscopic procedures are performed limits the ability to move the arthroscope inside the patient’s body, restricting the field of view. In this work, to overcome these limitations, we introduce ARthroNeRF, a novel framework that combines Neural Radiance Fields (NeRF) and Augmented Reality (AR). This framework allows for the generation of synthetic viewpoints from the perspective of surgical tools without the need for an additional camera. To evaluate the feasibility of the proposed framework, we conducted a user study with 18 participants. In this study, participants were tasked to touch hidden targets assisted by a synthetic view generated from the perspective of a surgical tool. The results of this study demonstrate that ARthroNeRF provides accurate supplementary visual information and suggest that ARthroNeRF has the potential to streamline the learning process in arthroscopic surgery. In addition, we built a system capable of presenting the reconstructed scenes using the Microsoft HoloLens 2. The incorporation of AR, overlaying synthesized images alongside the original arthroscopic footage and within the surgeon’s visual field, represents a viable alternative to enhance perception in spatially constrained scenarios.

Image-based 2D/3D registration is a critical technique for fluoroscopic guided surgical interventions. Conventional intensity-based 2D/3D registration approa- ches suffer from a limited capture range due to the presence of local minima in hand-crafted image similarity functions. In this work, we aim to extend the 2D/3D registration capture range with a fully differentiable deep network framework that learns to approximate a convex-shape similarity function. The network uses a novel Projective Spatial Transformer (ProST) module that has unique differentiability with respect to 3D pose parameters, and is trained using an innovative double backward gradient-driven loss function. We compare the most popular learning-based pose regression methods in the literature and use the well-established CMAES intensity-based registration as a benchmark. We report registration pose error, target registration error (TRE) and success rate (SR) with a threshold of 10mm for mean TRE. For the pelvis anatomy, the median TRE of ProST followed by CMAES is 4.4mm with a SR of 65.6% in simulation, and 2.2mm with a SR of 73.2% in real data. The CMAES SRs without using ProST registration are 28.5% and 36.0% in simulation and real data, respectively. Our results suggest that the proposed ProST network learns a practical similarity function, which vastly extends the capture range of conventional intensity-based 2D/3D registration. We believe that the unique differentiable property of ProST has the potential to benefit related 3D medical imaging research applications. The source code is available at https://github.com/gaocong13/Projective-Spatial-Transformers.

The use of Augmented Reality (AR) for navigation purposes has shown beneficial in assisting physicians during the performance of surgical procedures. These applications commonly require knowing the pose of surgical tools and patients to provide visual information that surgeons can use during the performance of the task. Existing medical-grade tracking systems use infrared cameras placed inside the Operating Room (OR) to identify retro-reflective markers attached to objects of interest and compute their pose. Some commercially available AR Head-Mounted Displays (HMDs) use similar cameras for self-localization, hand tracking, and estimating the objects' depth. This work presents a framework that uses the built-in cameras of AR HMDs to enable accurate tracking of retro-reflective markers without the need to integrate any additional electronics into the HMD. The proposed framework can simultaneously track multiple tools without having previous knowledge of their geometry and only requires establishing a local network between the headset and a workstation. Our results show that the tracking and detection of the markers can be achieved with an accuracy of$0.09\pm 0.06\ mm$on lateral translation,$0.42 \pm 0.32\ mm$on longitudinal translation and$0.80 \pm 0.39^\circ$for rotations around the vertical axis. Furthermore, to showcase the relevance of the proposed framework, we evaluate the system's performance in the context of surgical procedures. This use case was designed to replicate the scenarios of k-wire insertions in orthopedic procedures. For evaluation, seven surgeons were provided with visual navigation and asked to perform 24 injections using the proposed framework. A second study with ten participants served to investigate the capabilities of the framework in the context of more general scenarios. Results from these studies provided comparable accuracy to those reported in the literature for AR-based navigation procedures.

This article presents a dexterous robotic system for autonomous debridement of osteolytic bone lesions in confined spaces. The proposed system is distinguished from the state-of-the-art orthopedics systems because it combines a rigid-link robot with a continuum manipulator (CM) that enhances reach in difficult-to-access spaces often encountered in surgery. The CM is equipped with flexible debriding instruments and fiber Bragg grating sensors. The surgeon plans on the patient's preoperative computed tomography and the robotic system performs the task autonomously under the surgeon's supervision. An optimization-based controller generates control commands on the fly to execute the task while satisfying physical and safety constraints. The system design and controller are discussed and extensive simulation, phantom and human cadaver experiments are carried out to evaluate the performance, workspace, and dexterity in confined spaces. Mean and standard deviation of target placement are 0.5 and 0.18 mm, and the robotic system covers 91% of the workspace behind an acetabular implant in treatment of hip osteolysis, compared to the 54% that is achieved by conventional rigid tools.

Vertebral compression fractures (VCFs), the most common fragility fractures secondary to osteoporosis, affect more than 200 million individuals worldwide. Percutaneous vertebral augmentation is an effective interventional treatment option that is routinely performed across the world. Because fluoroscopy-guided vertebral augmentation is a well-established and safe minimally invasive technique, automating its delivery is among the most important next steps. In this work, we describe the design and evaluation of a novel cannula mounted vertebral augmentation robot in a simulated X-ray environment as a first step toward autonomous vertebral augmentation. The cannula robot employs a piezo stack with inchworm control to place surgical tools within the vertebral body, while X-ray imaging verifies the robot does not interfere with imaging. Finite element analysis of the robot confirms that radiolucent materials were rigid enough to be used in the robot design as expected deformations for the cannula drive, accessory drive, and locking mechanisms$(1.299 \pm 0.034 \ um, 1.280 \pm 0.027\ um$, and$1.960 \pm 0.218\ um$, respectively) did not exceed the stroke lengths of the piezo stacks. An in silico clinical trial based on a human anatomy model suffering from VCF validates that the cannula robot does not impede visualization of the critical anatomy and tool-to-tissue positioning. Together these results demonstrate the feasibility of a cannula mounted robot for vertebral augmentation.

Craniomaxillofacial reconstruction with patientspecific customized craniofacial implants (CCIs) is most commonly performed for large-sized skeletal defects. Because the exact size of skull resection may not be known prior to the surgery, in single-stage cranioplasty, an oversized CCI is prefabricated and resized intraoperatively with a manual-cutting process provided by a surgeon. The manual resizing, however, may be inaccurate and significantly add to the operating time. This paper introduces a fast and non-contact approach for intraoperatively determining the exact contour of the skull resection and automatically resizing the implant to fit the resection area. Our approach includes four steps: First, we acquire a patient’s defect information using a handheld 3D scanner. Second, the scanned defect is aligned to the CCI by registering the scanned defect to the preoperative CT model. Third, a cutting toolpath is generated from the scanned defect model by extracting the resection contour. Lastly, a cutting robot resizes the oversized CCI to fit the resection area. To evaluate the resizing performance of our method, we generated six different resection shapes for the cutting experiments. We compared the performance of our method to the performance of surgeon’s manual resizing and an existing technique that collects the defect contour with an optical tracking system. The results show that our proposed method improves the resizing accuracy by 56% compared to the surgeon’s manual modification and 42% compared to the optical tracking method.

Image stitching is a prominent challenge in medical imaging, where the limited field-of-view captured by single images prohibits holistic analysis of patient anatomy. The barrier that prevents straight-forward mosaicing of 2D images is depth mismatch due to parallax. In this work, we leverage the Fourier slice theorem to aggregate information from multiple transmission images in parallax-free domains using fundamental principles of X-ray image formation. The details of the stitched image are subsequently restored using a novel deep learning strategy that exploits similarity measures designed around frequency, as well as dense and sparse spatial image content. Our work provides evidence that reconstruction of orthographic mosaics is possible with realistic motions of the C-arm involving both translation and rotation. We also show that these orthographic mosaics enable metric measurements of clinically relevant quantities directly on the 2D image plane.

Suboptimal interaction with patient data and challenges in mastering 3D anatomy based on ill-posed 2D interventional images are essential concerns in image-guided therapies. Augmented reality (AR) has been introduced in the operating rooms in the last decade; however, in image-guided interventions, it has often only been considered as a visualization device improving traditional workflows. As a consequence, the technology is gaining minimum maturity that it requires to redefine new procedures, user interfaces, and interactions. The main contribution of this paper is to reveal how exemplary workflows are redefined by taking full advantage of head-mounted displays when entirely co-registered with the imaging system at all times. The awareness of the system from the geometric and physical characteristics of X-ray imaging allows the exploration of different human-machine interfaces. Our system achieved an error of 4.76 ± 2.91mm for placing K-wire in a fracture management procedure, and yielded errors of 1.57 ± 1.16° and 1.46 ± 1.00° in the abduction and anteversion angles, respectively, for total hip arthroplasty (THA). We compared the results with the outcomes from baseline standard operative and non-immersive AR procedures, which had yielded errors of [4.61mm, 4.76°, 4.77°] and [5.13mm, 1.78°, 1.43°], respectively, for wire placement, and abduction and anteversion during THA. We hope that our holistic approach towards improving the interface of surgery not only augments the surgeon's capabilities but also augments the surgical team's experience in carrying out an effective intervention with reduced complications and provide novel approaches of documenting procedures for training purposes.

Flexible medical instruments, such as Continuum Dexterous Manipulators (CDM), constitute an important class of tools for minimally invasive surgery. Accurate CDM shape reconstruction during surgery is of great importance, yet a challenging task. Fiber Bragg grating (FBG) sensors have demonstrated great potential in shape sensing and consequently tip position estimation of CDMs. However, due to the limited number of sensing locations, these sensors can only accurately recover basic shapes, and become unreliable in the presence of obstacles or many inflection points such as s-bends. Optical Frequency Domain Reflectometry (OFDR), on the other hand, can achieve much higher spatial resolution, and can therefore accurately reconstruct more complex shapes. Additionally, Random Optical Gratings by Ultraviolet laser Exposure (ROGUEs) can be written in the fibers to increase signal to noise ratio of the sensors. In this comparison study, the tip position error is used as a metric to compare both FBG and OFDR shape reconstructions for a 35 mm long CDM developed for orthopedic surgeries, using a pair of stereo cameras as ground truth. Three sets of experiments were conducted to measure the accuracy of each technique in various surgical scenarios. The tip position error for the OFDR (and FBG) technique was found to be 0.32 (0.83) mm in free-bending environment, 0.41 (0.80) mm when interacting with obstacles, and 0.45 (2.27) mm in s-bending. Moreover, the maximum tip position error remains sub-millimeter for the OFDR reconstruction, while it reaches 3.40 mm for FBG reconstruction. These results propose a cost-effective, robust and more accurate alternative to FBG sensors for reconstructing complex CDM shapes.

In this article, we present a novel stochastic algorithm called simultaneous sensor calibration and deformation estimation (SCADE) to address the problem of modeling deformation behavior of a generic continuum manipulator (CM) in free and obstructed environments. In SCADE, using a novel mathematical formulation, we introduce a priori model-independent filtering algorithm to fuse the continuous and inaccurate measurements of an embedded sensor (e.g., magnetic or piezoelectric sensors) with an intermittent but accurate data of an external imaging system (e.g., optical trackers or cameras). The main motivation of this article is the crucial need of obtaining an accurate shape/position estimation of a CM utilized in a surgical intervention. In these robotic procedures, the CM is typically equipped with an embedded sensing unit (ESU) while an external imaging modality (e.g., ultrasound or a fluoroscopy machine) is also available in the surgical site. The results of two different set of prior experiments in free and obstructed environments were used to evaluate the efficacy of SCADE algorithm. The experiments were performed with a CM specifically designed for orthopaedic interventions equipped with an inaccurate Fiber Bragg Grating (FBG) ESU and overhead camera. The results demonstrated the successful performance of the SCADE algorithm in simultaneous estimation of unknown deformation behavior of the utilized unmodeled CM together with realizing the time-varying drift of the poor-calibrated FBG sensing unit. Moreover, the results showed the phenomenal out-performance of the SCADE algorithm in estimation of the CM's tip position as compared to FBG-based position estimations.

Tracking and controlling the shape of continuum dexterous manipulators (CDM) in constraint environments is a challenging task. The imposed constraints and interaction with unknown obstacles may conform the CDM's shape and therefore demands for shape sensing methods which do not rely on direct line of sight. To address these issues, we integrate a novel Fiber Bragg Grating (FBG) shape sensing unit into a CDM, reconstruct the shape in real-time, and develop an optimization-based control algorithm using FBG tip position feedback. The CDM is designed for less-invasive treatment of osteolysis (bone degradation). To evaluate the performance of the feedback control algorithm when the CDM interacts with obstacles, we perform a set of experiments similar to the real scenario of the CDM interaction with soft and hard lesions during the treatment of osteolysis. In addition, we propose methods for identification of the CDM collisions with soft or hard obstacles using the jacobian information. Results demonstrate successful control of the CDM tip based on the FBG feedback and indicate repeatability and robustness of the proposed method when interacting with unknown obstacles.

A new type of cable-driven continuum manipulator (CM) is presented, in which the stiffness of the device along its body length can be controlled using the thermomechanical properties of a phase changing alloy. The liquid phase of the alloy is used for achieving high dexterity and the solid phase for high stiffness. Joule heating and water cooling is used for transitioning the phase changing alloy between stiff and compliant states. Single-segment and two-segment working prototypes of the CM are demonstrated. The mechanical and thermodynamic features of these prototypes are discussed and their physical performance is investigated. Advantages of the presented design with phase changing alloy include: significantly improved dexterity, high payload to weight ratio, controllable stiffness, energy efficiency, and a large lumen.

This paper focuses on the design and quantitative characterization of a debriding tool integrated with a robotic system to treat osteolysis (bone degradation). Osteolysis typically occurs due to wear of the polyethylene liner of the acetabular implant after total hip replacement surgery. In less invasive treatment of osteolysis, surgeons conventionally use rigid tools to debride the lesion, however with these inflexible instruments, complex lesion shapes are not completely treatable (about 50%). To address this issue, we have developed a debriding tool that passes through the lumen of a continuum dexterous manipulator (CDM). Integration of the CDM with a robotic arm assists the surgeon to reach the desired region behind the implant. Performance of the debriding tool integrated with this system was quantitatively evaluated during a simulated robot-assisted lesion debriding scenario. Rotational speed, aspiration pressure and irrigation flow of the debriding tool, as well as the sweeping velocity of the robotic system were identified as effective parameters in this procedure. Results indicate that maximum efficiency of the tool is achievable in a particular combination of these parameters.

Lesions of the lateral skull base in the petrous apex present unique surgical challenges because of the proximity of critical structures, including the inner ear, carotid artery, jugular bulb, facial nerve, lower cranial nerves, dura and brain. Currently, there are few appropriate surgical devices that can reach and remove these lesions, each with their own disadvantages. Here we investigate the feasibility of a dexterous continuum manipulator (DCM) capable of C-& S-shaped bends enabling dissection with remote center of motion (RCM) deep to the intact inner ear. A dedicated borescope channel provides the necessary visualization, while a flexible ring curette pre-shaped with a nitinol strip is designed to work through the instrument lumen for curettage of a cystic lesion. The kinematics of the DCM with the ring curette subject to an RCM constraint are investigated to explore the boundaries of a typical cyst cavity. Experiments in the planar phantom are carried out to validate feasibility, and results show that the proposed solution is practicable, accomplishing 80% and 83% removal of cysts for two kinds of boundaries.

Shape sensing techniques utilizing Fiber Bragg grating (FBG) arrays can enable real-time tracking and control of dexterous continuum manipulators (DCM) used in minimally invasive surgeries. For many surgical applications, the DCM may need to operate with much larger curvatures than what current shape sensing methods can detect. This paper proposes a novel shape sensor, which can detect a radius of curvature of 15 mm for a 35 mm long DCM. For this purpose, we used FBG sensors along with nitinol wires as the supporting substrates to form a triangular cross section. For verification, we assembled the sensor inside the wall of the DCM. Experimental results indicate that the proposed sensor can detect the DCM's curvature with an average error of 3.14%.

We have recently developed a snake-like manipulator for use in orthopaedic environments. One example application is the treatment of osteolysis (bone degradation) due to total hip arthroplasty. Recent literature suggest constant curvature models to define manipulator configuration from string (or actuator cable) length; however, our manipulator does not conform to constant curvature bending. In this paper, we present a two-step model to predict the kinematic configuration directly from string length with no assumptions regarding constant curvature bending. We experimentally identify the model parameters and validate the model on an additional experimental data set. The results indicate our model achieved an average maximum error of 1.0 ± 0.90mm in predicting manipulator configuration compared to the ground truth over the test data set.

A new type of continuum manipulator is presented, in which the body of the device is made up of identical, repeated interlocking fibers. A working prototype is demonstrated. Basic models describing the kinematics and mechanical properties of the device are developed, and their predictions are compared with the performance of the physical prototype. Advantages of the interlocking design include improved strength due to better load distribution, controllable stiffness, and a large open lumen.

Osteolysis is a debilitating condition that can occur behind the acetabular component of total hip replacements due to wear of the polyethylene liner. Conventional treatment techniques suggest replacing the component, while less-invasive approaches attempt to access and clean the lesion through the screw holes in the component. However, current rigid tools have been shown to access at most 50% of the lesion. Using a recently developed dexterous manipulator, we have adapted a group-theoretic convolution framework to define the manipulator's workspace and its ability to fully explore a lesion. We compared this with the experimental exploration of a printed model of the lesion. This convolution approach successfully contains the experimental results and shows over 98.8% volumetric coverage of a complex lesion. The results suggest this manipulator as a possible solution to accessing much of the area unreachable to the conventional less-invasive technique.

This paper presents a method for estimating drive cable length in an underactuated, hyper-redundant, snake-like manipulator. The continuum manipulator was designed for the surgical removal of osteolysis behind total hip arthroplasties. Two independently actuated cables in a pull-pull configuration control the compliant manipulator in a single plane. Using a previously developed kinematic model, we present a method for estimating drive cable displacement for a given manipulator configuration. This calibrated function is then inverted to explore the ability to achieve local manipulator configurations from prescribed drive cable displacements without the use of continuous visual feedback. Results demonstrate an effectiveness in predicting drive cable lengths from manipulator configurations. Preliminary results also show an ability to achieve manipulator configurations from prescribed cable lengths with reasonable accuracy without continuous visual feedback.

A novel algorithm is presented to segment and reconstruct injected bone cement from a sparse set of X-ray images acquired at arbitrary poses. The sparse X-ray multi-view active contour (SxMAC-pronounced "smack") can 1) reconstruct objects for which the background partially occludes the object in X-ray images, 2) use X-ray images acquired on a noncircular trajectory, and 3) incorporate prior computed tomography (CT) information. The algorithm's inputs are preprocessed X-ray images, their associated pose information, and prior CT, if available. The algorithm initiates automated reconstruction using visual hull computation from a sparse number of X-ray images. It then improves the accuracy of the reconstruction by optimizing a geodesic active contour. Experiments with mathematical phantoms demonstrate improvements over a conventional silhouette based approach, and a cadaver experiment demonstrates SxMAC's ability to reconstruct high contrast bone cement that has been injected into a femur and achieve sub-millimeter accuracy with four images.

Intraoperative patient registration may significantly affect the outcome of image-guided surgery (IGS). Image-based registration approaches have several advantages over the currently dominant point-based direct contact methods and are used in some industry solutions in image-guided radiation therapy with fixed X-ray gantries. However, technical challenges including geometric calibration and computational cost have precluded their use with mobile C-arms for IGS. We propose a 2D/3D registration framework for intraoperative patient registration using a conventional mobile X-ray imager combining fiducial-based C-arm tracking and graphics processing unit (GPU)-acceleration. The two-stage framework 1) acquires X-ray images and estimates relative pose between the images using a custom-made in-image fiducial, and 2) estimates the patient pose using intensity-based 2D/3D registration. Experimental validations using a publicly available gold standard dataset, a plastic bone phantom and cadaveric specimens have been conducted. The mean target registration error (mTRE) was 0.34 ± 0.04 mm (success rate: 100%, registration time: 14.2 s) for the phantom with two images 90° apart, and 0.99 ± 0.41 mm (81%, 16.3 s) for the cadaveric specimen with images 58.5° apart. The experimental results showed the feasibility of the proposed registration framework as a practical alternative for IGS routines.

A dexterous manipulator (DM) with a large open lumen is presented. The manipulator is designed for surgical applications with a preliminary focus on the removal of osteolysis formed behind the acetabular shell of primary total hip arthroplasties (THAs). The manipulator is constructed from two nested superelastic nitinol tubes enabling lengthwise channels for drive cables. Notches in the nested assembly provide reliable bending under applied cable tension producing kinematics that can be effectively modeled as a series of rigid vertebrae connected using pin joints. The manipulator is controlled in plane with two independently actuated cables in a pull-pull configuration. For the purpose of the procedure, the manipulator is mounted on a Z-θ stage adding a translational and rotational degree of freedom (DOF) along the axis of the manipulator. Preliminary experimental results demonstrate the initial modeling and control of the manipulator.

A new self-reconfigurable robot is presented. The robot is a hybrid chain/lattice design with several novel features. An active mechanical docking mechanism provides inter-module connection, along with optical and electrical interface. The docking mechanisms function additionally as driven wheels. Internal slip rings provide unlimited rotary motion to the wheels, allowing the modules to move independently by driving on flat surfaces, or in assemblies negotiating more complex terrain. Modules in the system are mechanically homogeneous, with three identical docking mechanisms within a module. Each mechanical dock is driven by a high torque actuator to enable movement of large segments within a multi-module structure, as well as low-speed driving. Preliminary experimental results demonstrate locomotion, mechanical docking, and lifting of a single module.

A problem associated with motion planning for the assembly of individual modules in a new self-reconfigurable modular robotic system is presented. Modules of the system are independently mobile and can be driven on flat surfaces in a similar fashion to the classic kinematic cart. This problem differs from most nonholonomic steering problems because of an added constraint on one of the internal states. The constraint properly aligns the docking mechanism, allowing modules to connect with one another along wheel surfaces. This paper presents an initial method for generating trajectories and control inputs that allow module assembly. It also provides an iterative method for locally optimizing a nominal control function using weighted perturbation functions, while preserving the final pose and internal states.