In this paper, we propose a probabilistic motion model for tendon actuated continuum robots that experience actuation transmission non-linearities due to cable slack and cable-sheath friction. The model is based on a Lie group formulation of the robot’s end-effector pose that incorporates a new simple backlash model. Bayesian parameter estimation is then employed to learn a probability distribution over the model’s parameters. This allows the uncertainty over the parameters to be propagated in the prediction of the end-effector’s trajectory. The model’s predictive capabilities are compared against the static Cosserat-rod-based model and the Kirchhoff model in simulation and are validated with experiments on a robotized medical endoscope.

In application domains such as surgical robotics, fully autonomous control remains a long-term ambition and the systems are mostly teleoperated. In this article, the presence of an operator in-the-loop is exploited to perform the online registration of an initially inaccurate haptic guidance and the calibration of robot kinematic models using operator’s intention instead of relying on exteroceptive sensors. This is used to improve online haptic guidance in the context of shared control, or to progress toward automatic task completion after an initial learning phase. The method presented in this article is based on an optimization in the task space to minimize the errors between the executed and desired trajectories, both estimated from models. This approach is particularly relevant when the execution of a planned task would suffer from errors that exteroceptive measurements could not fully correct, because of sensor inaccuracy or unavailability. A user study realized for a drawing task is detailed to illustrate that initially inaccurate task registration and robot models can be corrected from user inputs only. The results show that the proposed algorithm can learn the correct models, which in turns significantly improves the quality of the haptic guidance and decreases path deviations during the teleoperated task.

Cardiovascular disease treatment involves the challenging task of navigating guidewires and catheters through the vascular anatomy. This often results in prolonged procedures where both the patient and clinician are subjected to X-ray radiation. As a potential solution, Deep Reinforcement Learning methods have demonstrated potential in learning this task, paving the way for automated catheter navigation during robotic interventions. However, current works show a limited ability to generalize to unseen and/or deforming anatomies.In this paper, we extend our previous reinforcement learning approach in two main areas: we improve the training strategy to learn a control of the device even when the vascular anatomy is deforming and we propose a method to estimate the motion of the anatomy from single view fluoroscopy images. The combination of these two contributions makes it possible to automatically navigate across a moving vascular anatomy under fluoroscopic imaging, even without injecting a contrast agent. We validate our method on two scenarios: a simulated beating heart and a liver subjected to breathing motion. Our approach leads to an average success rate of 95% in reaching random targets within these anatomies. Our framework is also computationally efficient, enabling the training of our controller to be completed in about 6 hours.

Optical Coherence Tomography (OCT) is increasingly used in endoluminal procedures since it provides high-speed and high resolution imaging. Distortion and instability of images obtained with a proximal scanning endoscopic OCT system are significant due to the motor rotation irregularity, the friction between the rotating probe and outer sheath and synchronization issues. On-line compensation of artefacts is essential to ensure image quality suitable for real-time assistance during diagnosis or minimally invasive treatment. In this paper, we propose a new online correction method to tackle both B-scan distortion, video stream shaking and drift problem of endoscopic OCT linked to A-line level image shifting. The proposed computational approach for OCT scanning video correction integrates a Convolutional Neural Network (CNN) to improve the estimation of azimuthal shifting of each A-line. To suppress the accumulative error of integral estimation we also introduce another CNN branch to estimate a dynamic overall orientation angle. We train the network with semi-synthetic OCT videos by intentionally adding rotational distortion into real OCT scanning images. The results show that networks trained on this semi-synthetic data generalize to stabilize real OCT videos, and the algorithm efficacy is demonstrated on both ex vivo and in vivo data, where strong scanning artifacts are successfully corrected.

Haptic guidance can improve accuracy and dexterity during the teleoperation of a robot, but only if the model of the task used to provide the assistance is accurate. In medical robotics, the registration of a task from pre-operative planning from medical images to the robot’s task-space can be erroneous. Additionally, the deformability of the environment can require online correction of a planned task. Therefore, we propose a method to update the geometry and the registration of a pathfollowing task online. This model is simultaneously used to physically guide the user during the teleoperation. Experimental results obtained on a haptic interface show the validity of the approach for a simulated 2D task.

Non-linearities in cable transmissions are important limitations for the accurate control of flexible instruments used in medical endoscopic systems. Hysteresis effects greatly impact the accuracy of conventional kinematic models. This is especially critical for implementing automatic motions in flexible medical robotic systems. In this paper, we propose a method for improving open-loop accuracy of flexible instruments by implementing a Position Inverse Kinematic Model which is able to take into account hysteresis effects. In order to avoid complex physical modeling, the method relies on the off-line learning of the behavior of the instruments. Basic knowledge of the kinematic is also incorporated in the learning process in order to make it fast. The validity of the approach is demonstrated by the execution of 2D and 3D trajectories with the instruments of the STRAS medical robot. The accuracy is shown to be significantly improved with respect to other learning-based methods.

Automatic robot / Computed Tomography (CT) scanner registration is an important feature for robot-assisted percutaneous needle placement under CT-scanner. This registration can be done using 3D images, but for fast, low X-ray radiation it is interesting to be able to perform the registration with a single slice. In this paper, a new marker is proposed, which allows to estimate the pose of a device using a single slice. This marker, called ZCM, consists of three circles or ellipses arranged in a Z-shape configuration. It is shown that it provides a larger workspace (i.e. a larger set of visible configurations) for pose measurement than the standard Brown-Roberts-Wells, while maintaining a good accuracy. A closed-form method is proposed for solving the pose estimation with this marker. Simulations and experimental tests using a mock-up patient-mounted robot are presented and confirm the theoretical analysis.

The spatial configuration of actuated flexible instruments is fundamental for control applications in robotic noscar surgery. In these operations, the instruments are inserted in the channels of a flexible guide equipped with an endoscopic camera. In this paper we propose to estimate the position of the instruments of the Anubis platform (Karl Storz) using the endoscopic images provided by the embedded camera. In this system flexible instruments have 3-DOF (translation and rotation in the channel and deflection). Application of standard approaches for 3D location with such system do not provide good accuracy because of uncertainties on several model parameters. To cope with these uncertainties, supervised learning methods has been explored. With the help of colored visual markers attached to the instruments, the proposed approach consists of an image segmentation stage followed by a position estimation stage. Firstly, the markers are segmented in the images using an AdaBoost classifier manually trained on in-vivo images. Subsequently, the resulting blobs are used as input data of an approximation function trained using ground truth information provided by a magnetic sensor. A comparison with two other model-based methods showed the potentialities of such an approach on real devices.

In this paper we present a new robotic system, called STRAS, designed for endoluminal and transluminal surgery. The system is based on the Anubis®platform - a manual flexible system designed by Karl Storz for transluminal operations. Our new robot is a modular robotic system, compatible with the medical environment, allowing an easy setup in the operating room. It provides up to 10 Degrees of Freedom (DoFs), enabling the 3D positioning of an endoscopic camera, positioning of two instruments and offering grasper opening / closing functionalities. The paper presents for the first time the mechanical, electrical and control design of STRAS. An initial characterization of the system shows that, because of complex mechanical interactions, the kinematic modeling is not sufficient for cartesian control. However, first experiments, demonstrate that the robotic system can be telemanipulated using joint control and that the robotic system enables a single user to perform complex tasks with the underlying flexible system.

Stras is a flexible robotic system based on the Anubis® platform from Karl Storz and is aimed for intralu-minal and transluminal procedures. It is composed of three cable-driven sub-systems, one endoscope and two insertable instruments. The bending instruments have three degrees of freedom and can be teleoperated by the user via two commercial master interfaces (Omega.7, Force Dimension). In this paper we investigate several ways to map the motions from the master side to the instruments, from joint per joint control to cartesian control. We describe these mappings and compare them in elementary tasks in an attempt to analyze how non-linearities affect the accuracy of control. Results show that joint control and pseudo-cartesian control provide equivalent accuracy but with different difficulties for the user.

The use of flexible cable-driven systems is common in medicine (endoscope, catheter...). Their flexiblity allows surgeons to reach internal organs through sinuous and constrained ways. Unfortunately these systems are subject to backlash due to their internal mechanism. These non linearities raise many difficulties when robotizing and controlling such systems. In this article we propose an approach to improve the cartesian control of a four ways flexible endoscopic system with strong and unknown backlash-like non linearities. The method is based on an automatic off-line hystereses learning. We show that, despite coupling between degrees of freedom, it is possible to extract information from the hystereses which allow to improve cartesian control. Experiments on a real endoscopic system show the validity and the interest of the approach.

Flexible endoscopes are used in many diagnostic exams, like gastroscopies or colonoscopies, as well as for small surgical procedures. Recently, they have also been used for endoscopic surgical procedures through natural orifices [natural orifice transluminal endoscopic surgery (NOTES)] and for single-port-access abdominal surgery (SPA). Indeed, flexible endoscopes allow access of operating areas that are not easily reachable with only one small external or internal incision. However, their manipulation is complex, especially for surgical interventions. This study proposes to motorize the flexible endoscope and to partly robotize its movements in order to help the physicians during such interventions. The paper explains how the robotized endoscope can be used to automatically track an area of interest despite breathing motion. The system uses visual servoing and repetitive control strategies and allows stabilization of the endoscopic view. All required parameters are automatically estimated. In vivo experiments show the validity of the proposed solution for the improvement of the manipulation of flexible endoscopes.

The use of flexible endoscopes in new surgical procedures such as NOTES, i.e. Natural Orifice Transluminal Endoscopic Surgery, raises many problems. Indeed, the movements of conventional flexible endoscopes are limited and surgeons can only perform basic tasks with these systems. In order to enhance endoscope possibilities and workspace, we are currently developing a robotized system. The prototype we propose is based on the combination of several flexible endoscopes. The kinematic model of the system is detailed in order to develop a method of control. The paper presents the implementation of a control strategy using an external sensor which allows to deal with non linearities induced by the cable mechanism of the endoscopes.

Estimating insertion points of surgical instruments for minimally invasive surgery is a necessary step to be able to control surgical instruments using endoscopic images. In this paper, we propose an analysis of possible methods which use image information only. Mathematical properties are detailed together with statistical properties obtained by simulations. Then a specific method is chosen to estimate the insertion point for bi-modal surgery (laparoscopy and flexible endoscopy). In vitro experiments show the accuracy of the approach and how it is possible to track the motion of the insertion point in the case of physiological motions.

Flexible endoscopes are used in many surgical procedures and diagnostic exams. They have also been used recently for new surgical procedures using natural orifices called NOTES. While these procedures are really promising for the patients, they are really awkward for the surgeons. In order to assist the surgeon, physiological motion cancellation has been successfully applied on a robotized endoscope in L. Ott, et al., (May 19-23, 2008), by using a prototype repetitive controller (PRC) and a repetitive generalized predictive controller (R-GPC). Both controllers showed to be powerful tools to cancel periodic disturbances but with poor transient response to non-periodic disturbances. Contrary to the R-GPC, the PRC is unsuitable for handling non-periodic reference changes. We propose in this paper, as a first improvement, a model-based control scheme using the PRC which allows to decouple the reference tracking from the periodic output disturbance rejection. The response to nonperiodic disturbance is also improved by this technique but a repetition appears caused by the repetitive controller. As a second improvement, a switching control scheme is proposed to avoid the repetition.

Flexible endoscopes are used in many surgical procedures and diagnostic exams, like in gastroscopy or coloscopy. They have also been used recently for new surgical procedures using natural orifices called NOTES. While these procedures are very promising for the patients, they are quite awkward for the surgeons. The flexible endoscope allows the access to operating areas which are not easily reachable, with small or no incisions; but the manipulation of the system is complex. In order to help the practicians during NOTES or classical interventions with flexible endoscopes, we propose to motorize the system so as to partially robotize the movements. This paper presents the problems in the use of the flexible endoscope and explains how the system can be used to stabilize the endoscope on an area of interest despite physiological motions and therefore to improve the manipulation of the system.

Robot-assisted surgical procedures require to generate paths for surgical instruments during a planning step and then to follow automatically the computed trajectories as close as possible. Most of the existing procedures use external localization devices to control the robot. However, these devices are not adequate for minimally invasive interventions such as laparoscopic surgery because this kind of surgery involves large lever effects which make high accuracy unreachable. In this paper, we propose a path following method based on the use of the endoscopic camera and instruments with markers. It mainly stands on an image-based visual servoing scheme and on the estimation of the 3D position of the incision point in the abdominal wall. We show that this method can lead to precise tracking despite the errors on the position of the incision point. The proposed method is finally demonstrated in vitro in an automatic suturing intervention.

The work presented in this paper addresses the problem of the stitching task in laparoscopic surgery using a circular needle and a conventional 4 DOFs needle-holder. This task is particularly difficult for the surgeons because of the kinematics constraints introduced by the trocar. So as to assist the surgeons, we propose to compute possible pathes for the needle through the tissues, which limits as much as possible tissues deformations while driving the needle towards the desired target. The article proposes a kinematic analysis and a geometric modeling of the stitching task. Based on this, some simple but useful information can be obtained to help the surgeon. The description of the task with well-chosen state variables allows to simply express the path planning problem. Conditions for the existence of acceptable pathes are given and a method to compute possible pathes is presented. Resulting pathes are shown to be satisfactory even under awkward configurations.

In this paper, the detection and localization of grey regions in color images is addressed. This work has been developed in the scope of the robotized laparoscopic surgery, specifically for surgical procedures occurring inside the abdominal cavity. Since very few works have been already published about that purpose, some existing algorithms have been selected and brought together to achieve a robust color segmentation, as fast as possible. The foreseen application is a good training ground to evaluate these algorithms since main difficulties came from the complexity of the scene, the moving background due to breathing motion, the high surface reflectance, the non-uniform and time-varying lighting conditions. Nevertheless, to achieve the image segmentation suitable for robot control, we propose a new approach, without markers, based on a recursive thresholding of the histogram of a new purity color attribute and region growing. The main contribution of this work is threefold and consists in: the definition of a new color purity component, a selection of reliable, fast and robust existing video processings for the above-mentioned application areas, improving some existing video processings to enhance color properties either to homogenize regions and to emphasize the saturation feature of chromatic pixels. The usefulness of the proposed set of sequential processings has been successfully validated with image sequences of an endoscope to efficiently extracting boundaries of a cylindrical needle-holder with a sampling rate of 5 Hz.