In this paper, we validate the effectiveness of the optimal planning algorithms we have developed for devising surgical plans for Percutaneous Nephrolithotomy (PCNL) using patient-specific Concentric-Tube Robots (CTRs). To do so, we built a life-sized phantom model of the right hemithorax, replicating the anatomy of a patient who suffered from kidney stone and underwent conventional PCNL. Two-dimensional CT scans of the phantom model and its 3D reconstruction enabled the creation of a surgical plan using our planning algorithms based on a puncture into the mid-pole of the kidney. This was compared with two other percutaneous tracts involving punctures into the lower and upper calyces for comparison. The optimal mid-pole plan achieved 84% stone coverage, significantly outperforming the lower pole (58 %) and upper pole (45 %) plans. These results validate the effectiveness of the algorithms and align with simulation-based findings from previous studies, which reported an average volume coverage of 81.6±19.6 % in clinical cases.

Catheter-based cardiac ablation is the preferred method of treating atrial fibrillation. Conventionally, the catheter is navigated in the heart using X-ray fluoroscopy imaging and an electroanatomical map. Although successful, these imaging modalities do not provide real-time feedback on the quality of lesions created, which in turn could lead to recurrence of arrhythmia. Intracardiac echo (ICE) catheter provides real-time imaging within the heart to visualize both the ablation catheter and lesions created. However, manipulating the ablation and ICE catheters simultaneously is tedious and time consuming. As a first step towards developing a robotic ICE catheter that can autonomously follow the ablation catheter and monitor the lesions, we have developed a dynamic model for the ICE catheter. The model is based on the Cosserat theory for flexible rods that relies on strain parametrization. The model also accounts for frictional forces between the catheter sheath and tendons, external loads and fluid forces acting on the catheter. A good nominal model for describing the catheter dynamics is essential to develop a robust control scheme for the robotic ICE catheter. The parameters of the ICE catheter are estimated using weight release, tendon-driven actuation and fluid flow experiments. To the best of our knowledge, this is the first dynamic model for the ICE catheter that accurately reflects the dynamics of the catheter under pulsatile fluid flow within a heart phantom.

This paper proposes a novel method for removing image feature mismatches in real-time that can handle both rigid and smooth deforming environments. Image distortion, parallax and object deformation may cause the pixel coordinates of feature matches to have non-rigid deformations, which cannot be represented using a single analytical rigid transformation. To solve this problem, we propose an algorithm based on the re-weighting and 1-point RANSAC strategy (R1P-RNSC), which operates under the assumption that a non-rigid deformation can be approximately represented by multiple rigid transformations. R1P-RNSC is fast but suffers from the drawback that local smoothing information cannot be considered, thus limiting its accuracy. To solve this problem, we propose a non-parametric algorithm based on the expectation-maximization algorithm and the dual quaternion-based representation (EMDQ). EMDQ generates dense and smooth deformation fields by interpolating among the feature matches, simultaneously removing mismatches that are inconsistent with the deformation field. It relies on the rigid transformations obtained by R1P-RNSC to improve its accuracy. The experimental results demonstrate that EMDQ has superior accuracy compared to other state-of-the-art mismatch removal methods. The ability to build correspondences for all image pixels using the dense deformation field is another contribution of this paper.

Multimodal image registration (MIR) is a fundamental procedure in many image-guided therapies. Recently, unsupervised learning-based methods have demonstrated promising performance over accuracy and efficiency in deformable image registration. However, the estimated deformation fields of the existing methods fully rely on the to-be-registered image pair. It is difficult for the networks to be aware of the mismatched boundaries, resulting in unsatisfactory organ boundary alignment. In this paper, we propose a novel multimodal registration framework, which elegantly leverages the deformation fields estimated from both: (i) the original to-be-registered image pair, (ii) their corresponding gradient intensity maps, and adaptively fuses them with the proposed gated fusion module. With the help of auxiliary gradient-space guidance, the network can concentrate more on the spatial relationship of the organ boundary. Experimental results on two clinically acquired CT-MRI datasets demonstrate the effectiveness of our proposed approach.

The ability to extend the field of view of laparoscopy images can help the surgeons to obtain a better understanding of the anatomical context. However, due to tissue deformation, complex camera motion and significant three-dimensional (3D) anatomical surface, image pixels may have non-rigid deformation and traditional mosaicking methods cannot work robustly for laparoscopy images in real-time. To solve this problem, a novel two-dimensional (2D) non-rigid simultaneous localization and mapping (SLAM) system is proposed in this paper, which is able to compensate for the deformation of pixels and perform image mosaicking in real-time. The key algorithm of this 2D non-rigid SLAM system is the expectation maximization and dual quaternion (EMDQ) algorithm, which can generate smooth and dense deformation field from sparse and noisy image feature matches in real-time. An uncertainty-based loop closing method has been proposed to reduce the accumulative errors. To achieve real-time performance, both CPU and GPU parallel computation technologies are used for dense mosaicking of all pixels. Experimental results on in vivo and synthetic data demonstrate the feasibility and accuracy of our non-rigid mosaicking method.

We propose an approach to reconstruct dense three-dimensional (3D) model of tissue surface from stereo optical videos in real-time, the basic idea of which is to first extract 3D information from video frames by using stereo matching, and then to mosaic the reconstructed 3D models. To handle the common low-texture regions on tissue surfaces, we propose effective post-processing steps for the local stereo matching method to enlarge the radius of constraint, which include outliers removal, hole filling, and smoothing. Since the tissue models obtained by stereo matching are limited to the field of view of the imaging modality, we propose a model mosaicking method by using a novel feature-based simultaneously localization and mapping (SLAM) method to align the models. Low-texture regions and the varying illumination condition may lead to a large percentage of feature matching outliers. To solve this problem, we propose several algorithms to improve the robustness of the SLAM, which mainly include 1) a histogram voting-based method to roughly select possible inliers from the feature matching results; 2) a novel 1-point RANSAC-based [Formula: see text] algorithm called as DynamicR1PP [Formula: see text] to track the camera motion; and 3) a GPU-based iterative closest points (ICP) and bundle adjustment (BA) method to refine the camera motion estimation results. Experimental results on ex- and in vivo data showed that the reconstructed 3D models have high-resolution texture with an accuracy error of less than 2 mm. Most algorithms are highly parallelized for GPU computation, and the average runtime for processing one key frame is 76.3 ms on stereo images with 960×540 resolution.

The ability to handle outliers is essential for performing the perspective- n-point (P nP) approach in practical applications, but conventional RANSAC+P3P or P4P methods have high time complexities. We propose a fast P nP solution named R1PP nP to handle outliers by utilizing a soft re-weighting mechanism and the 1-point RANSAC scheme. We first present a P nP algorithm, which serves as the core of R1PP nP, for solving the P nP problem in outlier-free situations. The core algorithm is an optimal process minimizing an objective function conducted with a random control point. Then, to reduce the impact of outliers, we propose a reprojection error-based re-weighting method and integrate it into the core algorithm. Finally, we employ the 1-point RANSAC scheme to try different control points. Experiments with synthetic and real-world data demonstrate that R1PP nP is faster than RANSAC+P3P or P4P methods especially when the percentage of outliers is large, and is accurate. Besides, comparisons with outlier-free synthetic data show that R1PP nP is among the most accurate and fast P nP solutions, which usually serve as the final refinement step of RANSAC+P3P or P4P. Compared with REPP nP, which is the state-of-the-art P nP algorithm with an explicit outliers-handling mechanism, R1PP nP is slower but does not suffer from the percentage of outliers limitation as REPP nP.

This paper describes some novel research results on bilateral teleoperation of a flexible catheter in a constrained environment. The dynamics of the catheter in a constrained environment is highly nonlinear and is affected by a large number of factors such as friction, flexibility of the catheter, force of insertion, shape and size of the catheter, etc. As a result, the distal end of the catheter almost never follows the actuated end of the catheter which could cause fatigue to the clinician/user attempting to insert the catheter in a blood vessel. In addition, there is a time delay of up to 0.8 second before the distal end of the catheter advances after the near end is actuated. This delay is sufficient to cause instability in the control loop. We have developed a teleoperation framework using wave variables to perform robot- assisted catheter insertion in a constrained environment while reflecting the forces at the tip of the catheter to the clinician. Experimental results showing the effect of different factors such as the velocity and stroke length of insertion on the delays introduced in control loop and errors between the master and slave position are included. To the best of our knowledge, this work is among the first to attempt to provide an understanding of the effects of various factors on the flexing of the catheter. In addition, master-slave insertion of a catheter instrumented with shape memory alloys (SMA) has also been performed and the results are included in this paper.

Interventional cardiologists are at great risk from radiation exposure due to lengthy procedures performed under X-ray radiations. Angioplasty is one such procedure wherein the clinician guides a catheter into the femoral artery under X-rays and the procedure often extends to over 50 min. A clinician performs several hundred such procedures over his/her lifetime, leading to an accumulation of the total radiation he/she is exposed to. In this paper, we investigate autonomous robot-assisted insertion of an active catheter instrumented with shape memory alloy (SMA) actuators using image guidance. The tip of the active catheter is tracked in real time to provide information on the location of the catheter that determines the optimal stroke length of insertion for the robot and the necessary bending angle for the active catheter. The catheter is autonomously guided from the point of entry to the site of plaque buildup, thereby shielding the clinician from harmful radiation due to the X-rays used for imaging and providing a more ergonomic approach for catheter insertion. Experimental results are given to illustrate the robot-assisted catheter insertion procedure using image guidance.

In this paper, we investigate autonomous robot- assisted insertion of an active catheter instrumented with shape memory alloy (SMA) actuators using image guidance. An Augmented Hybrid Impedance Control (AHIC) algorithm is implemented on a Mitsubishi robot (PA 10-7C) to insert the active catheter. The robot is constrained to move in Cartesian space along a pre-defined trajectory while controlling the force of insertion. A closed-loop control scheme has been developed to accurately control the bending in the active catheter. The tip of the active catheter is tracked in real-time to provide information on the path of the catheter and for determining the future course of insertion. The catheter is autonomously guided from the point of entry to the site of plaque buildup, thereby shielding the surgeon from the harmful radiation due to the X-rays used for imaging, providing a more ergonomic approach for catheter insertion. Experimental results are given to illustrate the robot-assisted catheter insertion procedure.

In the recent years, there has been considerable interest in developing active catheters instrumented with shape memory alloy (SMA) actuators to perform angioplasty. The problem, however, with SMA actuators is the extremely non-linear response of the strain with input current. Therefore, control of such an active catheter inside the body is inaccurate and which could lead to unforeseen damage. In this paper, a novel model for SMAs and a robust Hinfinloop-shaping controller has been implemented to guarantee robust performance of the active catheter. A closed-loop control algorithm has been implemented to precisely orient the catheter in space based on feedback from a 5-DOF magnetic sensor. The active catheter tracks a reference trajectory generated either by a haptic device under the control of a user or from an image processing algorithm which determines the angle of a branch in the vasculature.

Angioplasty is a minimally invasive procedure wherein a catheter (a thermoplastic hollow wire) is inserted into the femoral artery and guided till it reaches a blockage in the blood vessel. There could be some potential complications arising from the conventional way of performing angioplasty, e.g., damage to the blood vessel due to excessive force of insertion and exposure of clinicians to harmful radiations and/or high levels of noise from an MRI machine. In this paper, we investigate the use of a robot manipulator (Mitsubishi PA 10-7C) to aid in the insertion of a catheter into a blood vessel. The robot controls the insertion force while the surgeon can remotely operate the robot from a safe and comfortable environment. The paper describes a hybrid impedance control scheme implemented on the Mitsubishi robot to perform simultaneous force/position control. The robot is used in experiments to insert a catheter into a test-bed by controlling the force of insertion and preventing the catheter from buckling or "bunching up". Experimental results for the insertion algorithms are shown