Navigation inside luminal organs is an arduous task that requires non-intuitive coordination between the movement of the operator's hand and the information obtained from the endoscopic video. The development of tools to automate certain tasks could alleviate the physical and mental load of doctors during interventions allowing them to focus on diagnosis and decision-making tasks. In this paper we present a synergic solution for intraluminal navigation consisting of a 3D printed endoscopic soft robot that can move safely inside luminal structures. Visual servoing based on Convolutional Neural Networks (CNNs) is used to achieve the autonomous navigation task. The CNN is trained with phantoms and in-vivo data to segment the lumen and a model-less approach is presented to control the movement in constrained environments. The proposed robot is validated in anatomical phantoms in different path configurations. We analyze the movement of the robot using different metrics such as task completion time smoothness error in the steady-state mean and maximum error. We show that our method is suitable to navigate safely in hollow environments and conditions which are different than the ones the network was originally trained on.

Optical Coherence Tomography (OCT) is increasingly used in endoluminal procedures since it provides high-speed and high resolution imaging. Distortion and instability of images obtained with a proximal scanning endoscopic OCT system are significant due to the motor rotation irregularity, the friction between the rotating probe and outer sheath and synchronization issues. On-line compensation of artefacts is essential to ensure image quality suitable for real-time assistance during diagnosis or minimally invasive treatment. In this paper, we propose a new online correction method to tackle both B-scan distortion, video stream shaking and drift problem of endoscopic OCT linked to A-line level image shifting. The proposed computational approach for OCT scanning video correction integrates a Convolutional Neural Network (CNN) to improve the estimation of azimuthal shifting of each A-line. To suppress the accumulative error of integral estimation we also introduce another CNN branch to estimate a dynamic overall orientation angle. We train the network with semi-synthetic OCT videos by intentionally adding rotational distortion into real OCT scanning images. The results show that networks trained on this semi-synthetic data generalize to stabilize real OCT videos, and the algorithm efficacy is demonstrated on both ex vivo and in vivo data, where strong scanning artifacts are successfully corrected.

The trend of achieving minimal invasiveness in surgeries and recent technological advances in robotics have resulted in the emergence of flexible surgical robots. Such flexible robots can reach a surgical site via narrow and tortuous pathways, extending the reach of robotic surgery and potentially reducing the incision size. This review covers the key technical issues associated with flexible surgical robotics and introduces emerging flexible surgical robot systems organized according to their target applications in the endoluminal surgical field. Furthermore, the challenges and recent advancements in manipulator design, modeling, and control as well as the shape and force sensing of flexible robots are presented as key technical issues. Furthermore, the technical features and clinical values of emerging flexible surgical robot systems are introduced with their medical applications.

U reteroscopy is becoming the first surgical treatment option for the majority of urinary affections. This procedure is performed using an endoscope which provides the surgeon with the visual information necessary to navigate inside the urinary tract. Having in mind the development of surgical assistance systems, that could enhance the performance of surgeon, the task of lumen segmentation is a fundamental part since this is the visual reference which marks the path that the endoscope should follow. This is something that has not been analyzed in ureteroscopy data before. However, this task presents several challenges given the image quality and the conditions itself of ureteroscopy procedures. In this paper, we study the implementation of a Deep Neural Network which exploits the advantage of residual units in an architecture based on U-Net. For the training of these networks, we analyze the use of two different color spaces: gray-scale and RGB data images. We found that training on gray-scale images gives the best results obtaining mean values of Dice Score, Precision, and Recall of 0.73, 0.58, and 0.92 respectively. The results obtained shows that the use of residual U-Net could be a suitable model for further development for a computer-aided system for navigation and guidance through the urinary system.

We present a new quasi-affine reconstruction of a scene and its application to camera self-calibration. We refer to this reconstruction as QUARCH (QUasi-Affine Reconstruction with respect to Camera centers and the Hodographs of horopters). A QUARCH can be obtained by solving a semidefinite programming problem when, (i) the images have been captured by a moving camera with constant intrinsic parameters, and (ii) a vague knowledge of the relative orientation (under or over 120 degrees) between camera pairs is available. The resulting reconstruction comes close enough to an affine one allowing thus an easy upgrade of the QUARCH to its affine and metric counterparts. We also present a constrained Levenberg-Marquardt method for nonlinear optimization subject to Linear Matrix Inequality (LMI) constraints so as to ensure that the QUARCH LMIs are satisfied during optimization. Experiments with synthetic and real data show the benefits of QUARCH in reliably obtaining a metric reconstruction.

Non-linearities in cable transmissions are important limitations for the accurate control of flexible instruments used in medical endoscopic systems. Hysteresis effects greatly impact the accuracy of conventional kinematic models. This is especially critical for implementing automatic motions in flexible medical robotic systems. In this paper, we propose a method for improving open-loop accuracy of flexible instruments by implementing a Position Inverse Kinematic Model which is able to take into account hysteresis effects. In order to avoid complex physical modeling, the method relies on the off-line learning of the behavior of the instruments. Basic knowledge of the kinematic is also incorporated in the learning process in order to make it fast. The validity of the approach is demonstrated by the execution of 2D and 3D trajectories with the instruments of the STRAS medical robot. The accuracy is shown to be significantly improved with respect to other learning-based methods.

This paper addresses the problem of estimating the Euclidean transformation relating two rigidly attached reference frames from motion measurements: a problem that is commonly referred to as the Hand-Eye calibration. The motion measurements are often affected by synchronization issues and hardware malfunctions. When using pose sensors, such as Electromagnetic or Inertial Measurement Units, the measurements are often less reliable than those obtained by a traditional robot-link and camera pair. Corrupt measurements, whether due to disturbances, large synchronization mismatches or malfunctions, are considered outliers that ought to be filtered out of the estimation process. While this may be achieved non-deterministically by using Random Sample Consensus, we propose a deterministic, robust and accurate method for solving the Hand-Eye calibration problem despite the presence of large amounts of outliers and high levels of measurement noise. The proposed method is based on a reformulation of this estimation problem as a rank-constrained semi-definite programming problem allowing for robustness to be enforced via an iteratively re-weighted optimization approach.

This paper introduces a new approach for the control of a robotic system interacting with deformable structures. The method is applied to needle insertion procedures, which are among the least invasive surgical approaches to access deep internal structures with sometimes poor access conditions. Yet, during the insertion both tissues and needles deform resulting in a displacement of targets identified at the planning step and significantly raising the technical difficulty of these approaches. Robotic assistance may offer new possibilities to enforce the accuracy of the needle's positioning, but the deformation of tissues remains an open problem. In this paper, we propose a numerical approach where finite-element (FE) models are used in a close-control robotic loop. We introduce a complete forward simulation of deformable structures (needle and environment) and constraint-based interaction models allowing for the simulation of needle insertion and complex nonlinear phenomena (friction, puncture, and insertion) at a high frequency. For the control, we numerically derive the so-called Jacobian of the Simulation using an inverse method. The most original aspect of this paper lies in the fact that inverse steps are performed in constraints space, allowing this way for fast estimation of the Jacobian (i.e., between 40 and 100 Hz). The method is validated both numerically and experimentally using a flexible needle inserted inside a deformable foam. We show that the robot is able to follow a given trajectory, defined during the planning step, taking into account any occurring deformation of both the needle and the foam during the insertion; without any need for tracking the needle neither the target nor the trajectory.

This paper introduces an Augmented Reality (AR) system for open liver surgery. Although open surgery remains the gold-standard for the treatment of complex tumors and central lesions, technological issues actually prevent using AR with sufficient accuracy for clinical use. We propose a markers-based method allowing for the tracking and the deformation of a preoperative model in real-time during the surgery. Markers are manually placed on the surface of the organ after opening the abdominal cavity, and tracked in real-time by a set of infrared cameras. Our framework is composed of both a nonrigid initial registration method, providing an estimation of the location of the markers in the preoperative model, and a realtime tracking algorithm to deform the model during the surgery (even for large deformation or partial occlusion of the organ). The method is validated on both synthetic and ex-vivo samples; in addition, we demonstrate its applicability in the operating room during a liver resection surgery on a human patient. Preliminary studies provided promising results to improve the location of tumors, and to help surgeons into planning the ideal resection intraoperatively.

The effective design of virtual reality (VR) simulators requires a deeper understanding of VR mediated human actions such as hand movements, with specifically tailored experiments testing how different design parameters affect performance. The present experiment investigates the time and precision of hand (index finger) movements under varying conditions of structural complexity and image size in VR without tactile feed-back from object to hand/finger. 18 right-handed subjects followed a complex and a simple physiological structure of small, medium and large size in VR, with the index finger of one of their two hands, from right to left, and from left to right. The results show that subjects performed best with small-size-simple structures and large-size-complex structures in VR. Movement execution was generally faster and more precise on simple structures. Performance was less precise when the dominant hand was used to follow the complex structures and small object size in VR. It is concluded that both size and structural complexity critically influence task execution in VR when no tactile feed-back from object to finger is generated. Individual learning curves should be monitored from the beginning of the training as suggested by the individual speed-precision analyses.

Minimally-invasive (MI) procedures are becoming more popular and frequent due to their benefits such as reduced patient trauma and hospitalization time. However, several common types of MI interventions are performed under X-ray guidance, which exposes both patients and staff to harmful ionizing radiation. Radiation exposure has therefore become a major concern for the medical community. Yet, few efforts to actively reduce it by exploiting the robotic capabilities of the devices present in the surgical suite have been performed. The propagation of radiation highly depends on the X-ray source positioning. Hence, we propose an approach to optimize the imaging device's pose in order to reduce the exposure to radiation of both patient and staff, while preserving the visibility of the targeted anatomical structure in the acquired image. Our method is based on the optimization of a cost function, which takes the current context and device parameters into account to compute the overall radiation exposure. It relies on GPU-accelerated Monte Carlo methods to simulate radiation propagation and performs the optimization in quasi real-time. When evaluated on a set of standard imaging configurations, our approach is able to recommend a device's pose in a few seconds, for which the delivered dose is reduced. Such an approach can contribute to lower the probability of appearance and severity of long-term negative effects due to radiation exposure and improve overall radiation safety.

Many approaches have been proposed for human pose estimation in single and multi-view RGB images. However, some environments, such as the operating room, are still very challenging for state-of-the-art RGB methods. In this paper, we propose an approach for multi-view 3D human pose estimation from RGB-D images and demonstrate the benefits of using the additional depth channel for pose refinement beyond its use for the generation of improved features. The proposed method permits the joint detection and estimation of the poses without knowing a priori the number of persons present in the scene. We evaluate this approach on a novel multi-view RGB-D dataset acquired during live surgeries and annotated with ground truth 3D poses.

Surgical workflow recognition has numerous potential medical applications, such as the automatic indexing of surgical video databases and the optimization of real-time operating room scheduling, among others. As a result, surgical phase recognition has been studied in the context of several kinds of surgeries, such as cataract, neurological, and laparoscopic surgeries. In the literature, two types of features are typically used to perform this task: visual features and tool usage signals. However, the used visual features are mostly handcrafted. Furthermore, the tool usage signals are usually collected via a manual annotation process or by using additional equipment. In this paper, we propose a novel method for phase recognition that uses a convolutional neural network (CNN) to automatically learn features from cholecystectomy videos and that relies uniquely on visual information. In previous studies, it has been shown that the tool usage signals can provide valuable information in performing the phase recognition task. Thus, we present a novel CNN architecture, called EndoNet, that is designed to carry out the phase recognition and tool presence detection tasks in a multi-task manner. To the best of our knowledge, this is the first work proposing to use a CNN for multiple recognition tasks on laparoscopic videos. Experimental comparisons to other methods show that EndoNet yields state-of-the-art results for both tasks.

This paper introduces a new method for automatic robotic needle steering in deformable tissues. The main contribution relies on the use of an inverse Finite Element (FE) simulation to control an articulated robot interacting with deformable structures. In this work we consider a flexible needle, embedded in the end effector of a 6 arm Mitsubishi RV1A robot, and its insertion into a silicone phantom. Given a trajectory on the rest configuration of the silicone phantom, our method provides in real-time the displacements of the articulated robot which guarantee the permanence of the needle within the predefined path, taking into account any undergoing deformation on both the needle and the trajectory itself. A forward simulation combines i) a kinematic model of the robot, ii) FE models of the needle and phantom gel iii) an interaction model allowing the simulation of friction and puncture force. A Newton-type method is then used to provide the displacement of the robot to minimize the distance between the needle's tip and the desired trajectory. We validate our approach with a simulation in which a virtual robot can successfully perform the insertion while both the needle and the trajectory undergo significant deformations.

In this paper we present a new robotic system, called STRAS, designed for endoluminal and transluminal surgery. The system is based on the Anubis®platform - a manual flexible system designed by Karl Storz for transluminal operations. Our new robot is a modular robotic system, compatible with the medical environment, allowing an easy setup in the operating room. It provides up to 10 Degrees of Freedom (DoFs), enabling the 3D positioning of an endoscopic camera, positioning of two instruments and offering grasper opening / closing functionalities. The paper presents for the first time the mechanical, electrical and control design of STRAS. An initial characterization of the system shows that, because of complex mechanical interactions, the kinematic modeling is not sufficient for cartesian control. However, first experiments, demonstrate that the robotic system can be telemanipulated using joint control and that the robotic system enables a single user to perform complex tasks with the underlying flexible system.

Stras is a flexible robotic system based on the Anubis® platform from Karl Storz and is aimed for intralu-minal and transluminal procedures. It is composed of three cable-driven sub-systems, one endoscope and two insertable instruments. The bending instruments have three degrees of freedom and can be teleoperated by the user via two commercial master interfaces (Omega.7, Force Dimension). In this paper we investigate several ways to map the motions from the master side to the instruments, from joint per joint control to cartesian control. We describe these mappings and compare them in elementary tasks in an attempt to analyze how non-linearities affect the accuracy of control. Results show that joint control and pseudo-cartesian control provide equivalent accuracy but with different difficulties for the user.

In this paper, we focus on the technology issues to be solved to develop a cable-driven robot compatible with Magnetic Resonance Imaging constraints. This study is based on the design of a new compact cable-driven manipulator with remote actuators, initially developed for prostate interventions. One of the originalities of the system is to use an instrumented structure to evaluate the cable tensions and lengths in order to perform an adequate control. The sensors assessment has been experimentally achieved and the necessity to introduce a new control strategy using the developed sensors has been demonstrated.

The use of flexible cable-driven systems is common in medicine (endoscope, catheter...). Their flexiblity allows surgeons to reach internal organs through sinuous and constrained ways. Unfortunately these systems are subject to backlash due to their internal mechanism. These non linearities raise many difficulties when robotizing and controlling such systems. In this article we propose an approach to improve the cartesian control of a four ways flexible endoscopic system with strong and unknown backlash-like non linearities. The method is based on an automatic off-line hystereses learning. We show that, despite coupling between degrees of freedom, it is possible to extract information from the hystereses which allow to improve cartesian control. Experiments on a real endoscopic system show the validity and the interest of the approach.

Low-cost piezoresistive sensors can be of great interest in robotic applications due not only to their advantageous cost but to their dimension as well, which enables an advanced mechanical integration. In this paper, a comparison of two commercial piezoresistive sensors based on different technologies is performed in the case of a medical robotics application. The existence of significant nonlinearities in their dynamic behavior is demonstrated, and a nonlinear modeling is proposed. A compensation scheme is developed for the sensor with the largest nonlinearities before discussing the selection of a sensor for dynamic applications. It is shown that force control is achievable with these kinds of sensors, in spite of their drawbacks. Experiments with both types of sensors are presented, including force control with a medical robot.

Flexible endoscopes are used in many diagnostic exams, like gastroscopies or colonoscopies, as well as for small surgical procedures. Recently, they have also been used for endoscopic surgical procedures through natural orifices [natural orifice transluminal endoscopic surgery (NOTES)] and for single-port-access abdominal surgery (SPA). Indeed, flexible endoscopes allow access of operating areas that are not easily reachable with only one small external or internal incision. However, their manipulation is complex, especially for surgical interventions. This study proposes to motorize the flexible endoscope and to partly robotize its movements in order to help the physicians during such interventions. The paper explains how the robotized endoscope can be used to automatically track an area of interest despite breathing motion. The system uses visual servoing and repetitive control strategies and allows stabilization of the endoscopic view. All required parameters are automatically estimated. In vivo experiments show the validity of the proposed solution for the improvement of the manipulation of flexible endoscopes.

The use of flexible endoscopes in new surgical procedures such as NOTES, i.e. Natural Orifice Transluminal Endoscopic Surgery, raises many problems. Indeed, the movements of conventional flexible endoscopes are limited and surgeons can only perform basic tasks with these systems. In order to enhance endoscope possibilities and workspace, we are currently developing a robotized system. The prototype we propose is based on the combination of several flexible endoscopes. The kinematic model of the system is detailed in order to develop a method of control. The paper presents the implementation of a control strategy using an external sensor which allows to deal with non linearities induced by the cable mechanism of the endoscopes.

Estimating insertion points of surgical instruments for minimally invasive surgery is a necessary step to be able to control surgical instruments using endoscopic images. In this paper, we propose an analysis of possible methods which use image information only. Mathematical properties are detailed together with statistical properties obtained by simulations. Then a specific method is chosen to estimate the insertion point for bi-modal surgery (laparoscopy and flexible endoscopy). In vitro experiments show the accuracy of the approach and how it is possible to track the motion of the insertion point in the case of physiological motions.

Flexible endoscopes are used in many surgical procedures and diagnostic exams. They have also been used recently for new surgical procedures using natural orifices called NOTES. While these procedures are really promising for the patients, they are really awkward for the surgeons. In order to assist the surgeon, physiological motion cancellation has been successfully applied on a robotized endoscope in L. Ott, et al., (May 19-23, 2008), by using a prototype repetitive controller (PRC) and a repetitive generalized predictive controller (R-GPC). Both controllers showed to be powerful tools to cancel periodic disturbances but with poor transient response to non-periodic disturbances. Contrary to the R-GPC, the PRC is unsuitable for handling non-periodic reference changes. We propose in this paper, as a first improvement, a model-based control scheme using the PRC which allows to decouple the reference tracking from the periodic output disturbance rejection. The response to nonperiodic disturbance is also improved by this technique but a repetition appears caused by the repetitive controller. As a second improvement, a switching control scheme is proposed to avoid the repetition.

In this paper, the design of a novel MRI-compatible torque sensor is proposed. The sensor is based on an optical measurement, with a particular architecture to provide a very low cross-sensitivity. Its structure allows to simultaneously tune the sensor stiffness and the measurement amplitude of the optical sensor. To do so, kinematic singularities of parallel structures are used as well as the principles of dissociated metrology structure. The design of the sensor is first introduced by a kinematic analysis, before detailing its behavior using Finite Element Analysis. A prototype is finally introduced with first experimental results.

In this paper, nonlinear modeling of low cost force sensors is considered for force control applications in robotic or biomechanical applications. Commercial force sensors are often expensive, with a limited use in severe conditions such as the presence of a strong magnetic field. On the contrary, thin film piezoresistive sensors such as the Tekscan Flexiforce and the Interlink FSR sensors are of low cost and can be considered in such an environment. Only a few information is however available on their dynamic properties. We therefore provide an experimental study of their dynamic behavior, showing nonlinear properties. Identification is then achieved, and a compensation model is proposed. A force control experiment is finally presented to evaluate the compensation scheme.

Flexible endoscopes are used in many surgical procedures and diagnostic exams, like in gastroscopy or coloscopy. They have also been used recently for new surgical procedures using natural orifices called NOTES. While these procedures are very promising for the patients, they are quite awkward for the surgeons. The flexible endoscope allows the access to operating areas which are not easily reachable, with small or no incisions; but the manipulation of the system is complex. In order to help the practicians during NOTES or classical interventions with flexible endoscopes, we propose to motorize the system so as to partially robotize the movements. This paper presents the problems in the use of the flexible endoscope and explains how the system can be used to stabilize the endoscope on an area of interest despite physiological motions and therefore to improve the manipulation of the system.

This paper addresses the influence of human operators on telemanipulators bilateral control. A method to deal with the user variability is proposed, based on the auto-tuning of the force controller at the master side. First, the approach is evaluated experimentally with a group of users in the context of nonlinear interactions with soft tissues, showing very good tracking performances, for both force and position. Second, a robustness analysis is presented that allows to prove the stability of the system for large variations of the behavior of both the environment and the user. This analysis is based on an original uncertain model of the user.

The commercial development of haptic devices is very promising. Existing systems are often 6-degree-of-freedom mechanisms equipped with a stylus that acts as a tool. They exhibit force feedback for 3 or 6 degrees of freedom of their end-effector, depending on whether only forces or both forces and torques are rendered. Some planar devices are also used but oddly, one-degree-of-freedom linear haptic devices are quite rare. This lack can probably be explained by the necessary mechanical transformations that are required to achieve linear motions with rotary motors. In this paper, we review several possible structures and present the design of a new one-degree-of-freedom linear haptic device with a limited number of joints and a compact design, compatible with rotary actuation. We evaluate this device in the telemanipulation context.

This paper deals with the visual servoing of textured surfaces inside the human abdomen with a laparoscope for the robot-assisted minimally invasive surgery (MIS). The well-known image-based visual servoing (IBVS) is one of the most common approaches used for model-based servoing. When no CAD model is available, the efficient second-order minimization (ESM) tracking developed by Malis (2002, 2004) for grey-level images is one of the powerful recent techniques which is extended here to color images so as to handle occluded parts of the region of interest (ROI). Firstly, the ROI is splitted into small areas and a histogram-based color feature comparison of image areas is presented. For each frame and for each area, a metric based on the Bhattacharyya criterion is used to select the contributing areas for the computation of the planar homography between views. Secondly, since for any MIS technique, the endoscopic lens is passing through an insertion point on the abdominal wall, a specific control strategy is developed to perform the ESM tracking with a 4-DOF surgical robot. The method presented in this paper has been validated with several video sequences. Experimental results show that the tracking method is efficient even with more than 75 % of the tracked ROI occluded. Finally, the model-free visual servoing has been performed with the AESOP surgical robot and a training box. Even if the convergence rate is a little bit slow, the desired region is always reached.

In this paper, we address the problem of the pose recovery of a straight homogeneous circular cylinder (SHCC) from its apparent contour in a single 2-D image. In many real-world situations, one may encounter cylindrical objects especially in man-made structures. Object models based on surfaces of revolution, in particular cylinders, are suitable for many fields, like the automatic assembly, the human motion capture and also in medical image-based robotic guidance. To model the geometry of a SHCC, we first introduce a singular matrix which represents this degenerate quadric and we show it can be expressed with the Plucker coordinates of its symmetry axis. We demonstrate that the perspective projection of a SHCC is related to the pose parameters and for this model-based pose estimation problem, we present a degenerate conic-based fitting method which has some connections with the estimation of the fundamental matrix. We provide a closed-form solution for the pose determination (4 dof). Compared to earlier works in this field, the proposed approach exhibits some geometric properties of SHCCs, it can deal with partial occlusions of apparent contours and it provides an efficient direct pose solution for the symmetry axis. Simulated data and real images are used to validate the fitting and pose computation. Finally, to highlight the effectiveness of the proposed method to deal with apparent contours in a poor structured environment, we apply this work to the localization of instruments used in laparoscopic surgery.

Medical robotics is a field where dedicated mechanisms have an increasing importance. The strong operating room constraints, both medical and practical, lead to heavily customized solutions. In this paper, we consider the practical problems that must be solved to build a robotic system dedicated to medical interventions under CT-scan guidance. This compact robotic assistant is placed on the patient what raises new generic robotic design problems. Practical solutions together with the evaluation of a prototype are presented in this paper

Robot-assisted surgical procedures require to generate paths for surgical instruments during a planning step and then to follow automatically the computed trajectories as close as possible. Most of the existing procedures use external localization devices to control the robot. However, these devices are not adequate for minimally invasive interventions such as laparoscopic surgery because this kind of surgery involves large lever effects which make high accuracy unreachable. In this paper, we propose a path following method based on the use of the endoscopic camera and instruments with markers. It mainly stands on an image-based visual servoing scheme and on the estimation of the 3D position of the incision point in the abdominal wall. We show that this method can lead to precise tracking despite the errors on the position of the incision point. The proposed method is finally demonstrated in vitro in an automatic suturing intervention.

Robotic technologies provide new ways to compensate quasi-periodic biological motion, enabling higher surgical accuracy without invasive measures such as cardiopulmonary bypass. This paper describes current research in robotic compensation of hand tremor, respiratory motion, and heartbeat during surgical procedures. An analysis of each physiological motion pattern is provided, as well as a description of novel compensation techniques

In this paper is considered fast visual servoing of a two-links arm including flexibilities for compensation of heart movements in surgery. The flexibilities are globally identified with no additional sensor than the external-500 Hz camera used in the visual servoing loop. The obtained model is valid around the working position of the arm and can be easily modified for new positions of the camera. Two control strategies to handle flexibilities and achieve high bandwidth are implemented: Generalized Predictive Control (GPC) and H∞ control. Simulation and experimental results are given allowing to compare their efficiencies.

In this paper, we present a robot positioning task with respect to a static target using visual servoing and optimization techniques. The vision system is uncalibrated. The displacements of the robot are generated in real time in order to minimize an objective function using a simplex method and a Newton-like method. Our method allows for the inclusion of constraints in the image as well as in joint space. On-line image Jacobian estimation runs at the same time, and is used to accelerate convergence near the target. We successfully validate this method with simulations under the graphic library OpenGL, and practical experiment with industrial manipulators.

The work presented in this paper addresses the problem of the stitching task in laparoscopic surgery using a circular needle and a conventional 4 DOFs needle-holder. This task is particularly difficult for the surgeons because of the kinematics constraints introduced by the trocar. So as to assist the surgeons, we propose to compute possible pathes for the needle through the tissues, which limits as much as possible tissues deformations while driving the needle towards the desired target. The article proposes a kinematic analysis and a geometric modeling of the stitching task. Based on this, some simple but useful information can be obtained to help the surgeon. The description of the task with well-chosen state variables allows to simply express the path planning problem. Conditions for the existence of acceptable pathes are given and a method to compute possible pathes is presented. Resulting pathes are shown to be satisfactory even under awkward configurations.

This work presents a predictive-control approach to active mechanical filtering of complex, periodic motions of organs induced by respiration or heart beating in robotized surgery. Two different predictive-control schemes are proposed for the compensation of respiratory motions or cardiac motions. For respiratory motions, the periodic property of the disturbance has been included into the input-output model of the controlled system so as to have the robotic system learn and anticipate perturbation motions. A new cost function is proposed for the unconstrained generalized predictive controller (GPC), where reference tracking is decoupled from the rejection of predictable periodic motions. Cardiac motions are more complex, since they are the combination of two periodic nonharmonic components. An adaptive disturbance predictor is proposed which outputs future predicted disturbance values. These predicted values are used to anticipate the disturbance by using the predictive feature of a regular GPC. Experimental results are presented on a laboratory testbed and in vivo on pigs. They demonstrate the effectiveness of the two proposed methods to compensate complex physiological motion.

This work presents first in-vivo results of beating heart tracking with a surgical robot arm in off-pump cardiac surgery. The tracking is performed in a 2D visual servoing scheme using a 500 frame per second video camera. Heart motion is measured by means of active optical markers that are put onto the heart surface. Amplitude of the motion is evaluated along the two axis of the image reference frame. This is a complex and fast motion that mainly reflects the influence of both the respiratory motion and the electro-mechanical activity of the myocardium. A model predictive controller is setup to track the two degrees of freedom of the observed motion by computing velocities for two of the robot joints. The servoing scheme takes advantage of the ability of predictive control to anticipate over future references provided they are known or they can be predicted. An adaptive observer is defined along with a simple cardiac model to estimate the two components of the heart motion. The predictions are then fed into the controller references and it is shown that the tracking behaviour is greatly improved.

In this paper, the detection and localization of grey regions in color images is addressed. This work has been developed in the scope of the robotized laparoscopic surgery, specifically for surgical procedures occurring inside the abdominal cavity. Since very few works have been already published about that purpose, some existing algorithms have been selected and brought together to achieve a robust color segmentation, as fast as possible. The foreseen application is a good training ground to evaluate these algorithms since main difficulties came from the complexity of the scene, the moving background due to breathing motion, the high surface reflectance, the non-uniform and time-varying lighting conditions. Nevertheless, to achieve the image segmentation suitable for robot control, we propose a new approach, without markers, based on a recursive thresholding of the histogram of a new purity color attribute and region growing. The main contribution of this work is threefold and consists in: the definition of a new color purity component, a selection of reliable, fast and robust existing video processings for the above-mentioned application areas, improving some existing video processings to enhance color properties either to homogenize regions and to emphasize the saturation feature of chromatic pixels. The usefulness of the proposed set of sequential processings has been successfully validated with image sequences of an endoscope to efficiently extracting boundaries of a cylindrical needle-holder with a sampling rate of 5 Hz.

In this paper, we address the problem of precisely recovering the 3D (three-dimensional) pose of 3D shape fiducials from images obtained by means of an uncalibrated computed tomography (CT) imaging device. The main goal in this work is to model and estimate the geometric transformation relating line fiducials to their projections in cross-sectional images. To do so, we propose techniques, which solve the points to lines correspondence using closed-form and numerical algorithms. A geometric transformation with eight degrees of freedom (rotation, translation and anisotropic scaling) is used to model both a rigid-body transformation and a scaling transformation accounting for CT scan intrinsic parameters. Furthermore, an estimation of error bounds in space is given when image data are affected by noise. Real experiments show that the proposed method provides good results on a set of CT images from many viewpoints.

Periodic deformations of organs and soft tissues are complex, repetitive disturbances for surgeons manipulating robotic interfaces in computer-assisted surgery. They are due to respiratory movements or heart beats, and they have to be manually compensated for by the surgeon whenever accurate gestures are needed, as it is the case in cardiac or robotized laparoscopic surgery. This work presents a repetitive model predictive control scheme for the cancellation of fast periodic motions by a robot arm, which is controlled by visual servoing at 500 Hz by means of a high-speed camera. The problem we address is to keep a constant distance in the camera images from a surgical tool's tip to the organ surface. Contributions of the control input to reference tracking and to the fast-disturbance rejection are split and computed separately to ensure that the surgeon's interaction on the robot bas no influence on the cancellation performance. The system is tested in a laboratory experiment with an experimental surgical arm and in in vivo conditions on a living pig with a standard surgical robot. Results show the effectiveness and the potential of the proposed control scheme.

This paper presents a robotic vision system that automatically retrieves and positions surgical instruments during robotized laparoscopic surgical operations. The instrument is mounted on the end-effector of a surgical robot which is controlled by visual servoing. The goal of the automated task is to safely bring the instrument at a desired three-dimensional location from an unknown or hidden position. Light-emitting diodes are attached on the tip of the instrument, and a specific instrument holder fitted with optical fibers is used to project laser dots on the surface of the organs. These optical markers are detected in the endoscopic image and allow localizing the instrument with respect to the scene. The instrument is recovered and centered in the image plane by means of a visual servoing algorithm using feature errors in the image. With this system, the surgeon can specify a desired relative position between the instrument and the pointed organ. The relationship between the velocity screw of the surgical instrument and the velocity of the markers in the image is estimated online and, for safety reasons, a multistages servoing scheme is proposed. Our approach has been successfully validated in a real surgical environment by performing experiments on living tissues in the surgical training room of the Institut de Recherche sur les Cancers de l'Appareil Digestif (IRCAD), Strasbourg, France.

This paper presents a robotic vision system that automatically retrieves and positions surgical instruments in robotized laparoscopic surgery. The surgical instrument is mounted on the end-effector of a surgical robot which can be controlled by automatic visual feedback. The goal of the automated task is to bring the instrument at a desired location from an unknown or hidden position. To achieve this task, a special instrument-holder is designed with optical fibers and collimators. This instrument-holder projects laser dot patterns onto the organ surface which are seen in the endoscopic images. Then, the instrument is retrieved and centered in the image plane using a visual servoing algorithm. With this system, the surgeon can also specify a desired position for the instrument in the image. Our approach is successfully validated in a real surgical environment by performing experiments on living animals in the surgical training room of IRCAD.

In this paper, we present a new solution to laparoscopic manipulation based on force feedback control. This method allows to both explicitly control the forces applied to the patient through the trocar, and to precisely control the position of the surgical instrument. It does not require any geometrical model of the operative environment, nor any fine robot base placement prior to the instrument insertion. Different control strategies, involving different kinds of sensory equipments are proposed. They are experimentally validated on a laboratory apparatus.

In this paper, we address the problem of controlling the motion of a surgical instrument close to an unknown organ surface by visual servoing in the context of robotized laparoscopic surgery. To achieve this goal, a visual servoing algorithm is developed that combines feature errors in the image and errors in depth measurements. The relationship between the velocity screw of the surgical instrument, the depth and the motion field is defined and a two-stage servoing scheme is proposed. In order to measure the orientation and the depth of the instrument with respect to the organ, a laser dot pattern is projected on the organ surface and optical markers are stuck on the instrument. Our work has been successfully validated with a surgical robot by performing experiments on living tissues in the surgical training room of IRCAD.

In this paper, we present a robot positioning task with respect to a static target using visual servoing and optimization techniques. The vision system is uncalibrated and the kinematic model of the robot may be totally unknown. The displacements of the robot are generated in real time in order to minimize an objective function. The objective function includes the quadratic error between the current target image and the desired target image. A simplex method is used to minimize the objective function. Our method allows the system to include constraints in the image as well as in joint-space. We successfully validate this method, with simulations under the graphic library QpenGL and with experiments on a 6-DOF industrial manipulator.

This paper presents the visual servoing of a six degrees of freedom (6-DOF) manipulator for unknown three-dimensional profile following. The profile has an unknown curvature, but its cross section is known. The visual servoing keeps the transformation between a cross section of the profile and the camera constant with respect to 6 DOE The position of the profile with respect to only five degrees of freedom can be measured with the camera since the image does not provide position information along the profile. The kinematic model of the robot is used to reconstruct the displacement along the profile, i.e., the sixth degree of freedom, and allows to control the profile-following velocity. Experiments show good accuracy for positioning at a sampling rate of 50 Hz. Two control strategies are tested: proportional-integral control and generalized predictive control (GPC). The visual servoing exhibits better accuracy with the GPC in simulations and in real experiments on a 6-DOF manipulator due to the predictive property of the algorithm.

This paper presents a novel approach to model and control high speed 6 DOF visual servo loops. The modelling and control strategy takes into account the dynamics of a velocity controlled 6 DOF manipulator as well as a simplified model of the camera and acquisition system in order to increase the bandwidth of the servo loop. multi-input multi-output (MIMO) generalized predictive control (GPC) is used to optimally control the visual loop with respect to the proposed dynamic model. The visual sensor used in the experiments is a high speed camera that allows to acquire 120 noninterlaced images per second. With this camera, we achieve a sampling frequency of 120 Hz for the visual loop. Experimental results on a 6 DOF industrial robot are presented that validate the proposed model. They show drastic improvement of the loop performance with respect to more classical control strategies like PID type control.

This paper presents the visual servoing of a 6-DOF manipulator for profile following. The profile has an unknown curvature, but its cross-section is known. The visual servoing keeps the transformation constant between a cross-section of the profile and the camera with respect to 6 degrees of freedom. The position of the profile with respect to only five of these degrees of freedom can be measured with the camera, since the image gives no position information along the profile. The kinematic model of the robot is used to reconstruct the displacement along the profile and allow us to control the profile following velocity. Experiments show good accuracy for positioning at a sampling rate of 20 ms. Two control strategies are tested: PI control and generalized predictive control (GPC). The visual servoing exhibits better accuracy with the GPC in simulations and in real experiments.

Presents a way to model visual servoing in the case of a 6 DOF industrial manipulator. The manipulator with its actuators (DC motors), their current feedback loops and their velocity control loops, are modeled as a "virtual Cartesian device". The system with its visual feedback loop is decoupled to form 6 independent loops. Then, 6 generalized predictive controllers (GPC) are implemented online to take into account the dynamics of the manipulator. Simulations and experimental results show a drastic improvement in performance for a 6 DOF industrial manipulator in an eye-in-hand configuration compared to standard approaches neglecting these dynamics.

In this paper, the design of a position tracking control scheme for an industrial robot manipulator is considered. The design is based on the normalized left coprime factors robust stabilization approach. By coupling this H/sub /spl infin// synthesis to a sequential quadratic programming optimization procedure, a systematic design method is developed optimizing the performance of the controller. Further, the role of the nominal model for the design is explored. The control scheme is successfully implemented and tested on a real industrial robot manipulator. The computed controller worked immediately in the first experiment without any further tuning and showed excellent robust performance.