In this paper we propose a visual servoing approach that controls the deformation of a suspended tether cable subject to gravity from visual data provided by a RGB-D camera. The cable shape is modelled with a parabolic curve together with the orientation of the plane containing the tether. The visual features considered are the parabolic coefficients and the yaw angle of that plane. We derive the analytical expression of the interaction matrix that relates the variation of the visual features to the velocities of the cable extremities. Singularities are demonstrated to occur if and only if the cable is taut horizontally or vertically. An image processing algorithm is also developed to extract in real-time the current features fitting the parabola to the cable from the observed point cloud. Simulations and experimental results demonstrate the efficiency of our visual servoing approach to deform the tether cable toward a desired shape configuration.

In this paper, we propose a physics-based robot controller to deform a soft object toward a desired 3D shape using a limited number of handling points. For this purpose, the shape of the deformable object is represented using Fourier descriptors. We derive the analytical relation that provides the variation of the Fourier coefficients as a function of the movements of the handling points by considering a mass-spring model (MSM). A control law is then designed from this relation. Since the MSM provides an approximation of the object behavior, which in practice can lead to a drift between the object and its model, an online realignment of the model with the real object is performed by tracking its surface from data provided by a remote RGB-D camera. Simulation results validate the approach for the case where many points interact on a 2D soft object while experimental results obtained with two robotic arms demonstrate the autonomous shaping of a 3D soft object.

In this paper, we present a complete pipeline for positioning a feature point of a soft object to a desired 3D position, by acting on a different manipulation point using a robotic manipulator. For that purpose, the analytic relation between the feature point displacement and the robot motion is derived using a coarse mass-spring model (MSM), while taking into consideration the propagation delay introduced by a MSM. From this modeling step, a novel closed-loop controller is designed for performing the positioning task. To get rid of the model approximations, the object is tracked in real-time using a RGB-D sensor, thus allowing to correct on-line any drift between the object and its model. Our model-based and vision-based controller is validated in real experiments for two different soft objects and the results show promising performance in terms of accuracy, efficiency and robustness.

In this paper, we propose a framework for tracking the deformation of soft objects using a RGB-D camera by utilizing the physically-based model of the considered object. A coarse, 3D template of the object being tracked is the only prior information required by the proposed method. The proposed approach does not rely on the accurate knowledge of the material properties of the object being tracked. In this paper, we integrate computer vision based tracking methodology with physical model based deformation representation without requiring expensive numerical optimization for minimizing nonlinear error terms. The proposed approach enables deformation tracking by joint minimization of a geometric error and a direct photometric intensity error while utilizing co-rotational Finite Element Method (FEM) as the underlying deformation model. The proposed method has been validated both on synthetic data (with groundtruth) and real data.

In this paper, we propose the ADVISEd (Active Deformation through VIsual SErvoing) method, a novel model-free deformation servoing method able to deform a soft object towards a desired shape. ADVISEd relies on an online estimation of the deformation Jacobian that relates the motion of the robot end-effector to the deformation behavior of the object. The estimation is based on a weighted least-squares minimization with a sliding window. The robustness of the method to observation noise is ensured using an eigenvalue-based confidence criterion. The ADVISEd method is validated through comparisons with a model-based and a model-free state-of-the-art methods. Two experimental setups are proposed to compare the methods, one to perform a marker-based active shaping task and one to perform several marker-less active shaping and shape preservation tasks. Experiments showed that our approach can interactively control the deformations of an object in different tasks while ensuring better robustness to external perturbations than the state-of-the-art methods.

In this paper, we propose a novel method to simultaneously track the deformation of soft objects and estimate their elasticity parameters. The tracking of the deformable object is performed by combining the visual information captured by a RGB-D sensor with interactive Finite Element Method simulations of the object. The visual information is more particularly used to distort the simulated object. In parallel, the elasticity parameter estimation minimizes the error between the tracked object and a simulated object deformed by the forces that are measured using a force sensor. Once the elasticity parameters are estimated, our tracking algorithm can be used to estimate the deformation forces applied to an object without the use of a force sensor. We validated our method on several soft objects with different shape complexities. Our evaluations show the ability of our method to estimate the elasticity parameters as well as its use to estimate the forces applied to a deformable object without any force sensor. These results open novel perspectives to better track and control deformable objects during robotic manipulations.

This paper presents a framework for accurately tracking objects of complex shapes with joint minimization of geometric and photometric parameters using a coarse 3D object model with the RGB-D cameras. Tracking with coarse 3D model is remarkably useful for industrial applications. A technique is proposed that uses a combination of point-to-plane distance minimization and photometric error minimization to track objects accurately. The concept of `keyframes' are used in this system of object tracking for minimizing drift. The proposed approach is validated on both simulated and real data. Experimental results show that our approach is more accurate than existing state-of-the-art approaches, especially when dealing with low-textured objects with multiple coplanar faces.

Needle insertion procedures can greatly benefit from robotic systems to improve their accuracy and success rate. However, a fully automated system is usually not desirable and the clinicians need to be included in the control loop. In this paper we present a teleoperation framework for beveledtip flexible needle steering that enables the user to directly and intuitively control the trajectory of the needle tip via a haptic interface. The 6 degrees of freedom of the needle base are used to perform several automatic safety and targeting tasks in addition to the one controlled by the user. Real-time visual feedback is provided by a 3D ultrasound probe and used to track the 3D location of the needle and of a spherical target. Several haptic force feedback are compared as well as two different levels of mix between automated and user-controlled tasks. A validation of the framework is conducted in gelatin phantom and a mean targeting accuracy of 2.5 mm is achieved. The results show that providing an adequate haptic guidance to the user can reduce the risks of damage to the tissues while still letting the surgeon in control of the tip trajectory.

In minimally-invasive procedures like biopsy, the physician has to insert a needle into the tissues of a patient to reach a target. Currently, this task is mostly performed manually and under visual guidance. However, manual needle insertion can result in a large final positioning error of the tip that might lead to misdiagnosis and inadequate treatment. A way to solve this limitation is to use shared control; a gesture-assistance paradigm that combines the cognitive skills of the operator with the precision, stamina and repeatability of a robotic or haptic device. In this paper, we propose to assist the physician with a haptic device that holds the needle and generates mechanical guides during the phase of manual needle pre-positioning. In the latter, the physician has to place the tip of the needle on a planned entry point, with a pre-defined angle of incidence. From this pre-operative information and also from intra-operative measurements, we propose to generate haptic cues, known as virtual fixtures, to guide the physician towards the desired position and orientation of the needle. It takes the form of five haptic guides, each one implementing virtual fixtures. We conducted a user study where those guides were compared to the unassisted reference gesture. The most constraining guide, in terms of assisted degrees of freedom, was highlighted as the one that provides the best results in terms of performance and user experience.

A method to accurately track deformable objects using a RGB-D camera with the help of a coarse object model is presented in this paper. The deformation model is based on corotational FEM formulation. The physical model of the object does not need to be exact, nor do we require the precise physical properties for accurately tracking the object. The position of the vertices of the surface mesh of the tracked object is deformed using a set of virtual forces. A point-to-plane distance based geometric error between the pointcloud and the mesh is minimized with respect to these virtual forces. The point of application of force is determined by analysis of the error obtained from rigid tracking, which is done in parallel with the non-rigid tracking. This architecture also enables the overall system to be realtime. The proposed approach is evaluated on a synthetic data with ground-truth for deformation at every frame, as well as on real data.

In this paper we present a novel approach to track and explore stiff tissues within 3-D ultrasound volumes acquired by a medical 3-D ultrasound probe mounted on a six degrees of freedom robotic arm. Autonomous palpation and on-line elastography process are implemented to estimate the elastic property of the tissues (strain) in a volume of interest (VoI) indicated by the user. The compression motion, required for the elastography, is performed by controlling the force applied by the ultrasound probe to the tissues. A visual servoing control for centering a rigid tissue (target) inside the field of view (FoV) of the ultrasound probe is established to always maintain the target visible. Additionally, rotations around the contact point between the tissue and the ultrasound probe are teleoperated through a haptic device handled by the user in order to allow exploration of the target surrounding areas. Results show a stable system that can be used in the future for diagnosis of diseases or tumor location.

This paper presents a robot-assisted system to obtain elastic information of a moving tissue using a 2-D ultrasound probe actuated by a 6 degrees of freedom robotic arm. The proposed method combines ultrasound image-based visual servoing, force control and non-rigid motion estimation. We present how the motion estimation, while the force control and visual sevoing are enabled, is useful to compute the strain map of the tissue. Ex-vivo experiments performed on a moving abdominal phantom demonstrate the efficiency and robustness of this methodology to compute the strain information of a tissue in motion.

We present a control framework for optimizing image quality during robotic ultrasound acquisitions. The quality of the ultrasound signal across the field of view is represented by a confidence map that is computed online from the B-mode frames, following a model of sound propagation. Moments extracted from this confidence map are used to design a control law for optimizing imaging quality, based on the task function approach. The proposed confidence control is combined with force and position control to build two illustrative applications. First, we use force control and confidence control in order to maintain a correct pressure and a good orientation of the probe during teleoperation. Thus, control is shared between a human operator and the robot. Then, we add an automatic positioning task, so that the quality is optimized while maintaining a target in the image center. We show experimentally that confidence-driven control can effectively optimize the acoustic window in real time. In addition, we show that it can improve the tracking robustness, by preventing the target from being shadowed. Finally, we present the results of experiments performed on a human volunteer.

We propose a control framework to optimize the quality of robotic ultrasound imaging while tracking an anatomical target. We use a multitask approach to control the in-plane motion of a convex probe mounted on the end-effector of a robotic arm, based not only on the position of the target in the image, but also on features extracted from an ultrasound confidence map. The resulting control law therefore guarantees a good image quality, while keeping the target aligned with the central ultrasound scan-line. Potential applications of the proposed approach are, for example, teleoperated ultrasound examination, motion compensation for ultrasound-guided interventions, or automatic ultrasound acquisition. We demonstrate our approach with experiments on an ultrasound examination training phantom in motion.

Works on robotic needle steering often consider either tip-based control of flexible beveled tip needles or control of the bending of symmetric tip needles. In this paper a control law for needle steering which uses both direct manipulation of the needle base and the control of the tip deflection, is proposed. A 3D model of a beveled tip needle is used to determine the base motion that is needed to obtain the desired tip motion. The needle-tissue interaction model uses local virtual springs placed along the needle shaft and takes into account the interaction forces at the bevel. Online estimation and update of this model parameters is performed via visual feedback. The low level controller uses the task function framework to allow control of the tip velocity. Additionally duty cycling method is used if a reduction of the natural deflection of the needle tip is needed. Finally an experimental targeting task in a gelatin phantom is presented for preliminary validation of the framework. The method provided sub-millimeter accuracy on a target that would be unreachable using only tip-based control.

This paper deals with the development of a new 6 degrees-of-freedom (DOF) vision-based controller for robot-assisted medical applications. The main objective is to use visual information extracted from Ultrasounds (US) images to control an US probe held by a robot. Instead of the conventional use of visual features e.g., geometric features (points, lines, moments, etc.) in the visual control law design, the described method uses the shearlet coefficients. More precisely, the time-variation of the coarsest level of shearlet decomposition coefficients are linked to the US probe (respectively to the robot) spatial velocity and then related to the task-function control law. The proposed control law was experimentally tested and validated using a realistic abdominal phantom. The obtained results demonstrated promising performances in terms of accuracy, repeatability, robustness and convergence behavior.

With the increasing number of clinical interventions using needle shaped tools, robotic control of needle insertion procedures has been an active research field for many years. In this work we propose a 3D model of a flexible needle that takes into account tissue deformations in order to predict the needle shape and trajectory when it is inserted using a robotic arm. To account for tissue displacements, we designed a method based on visual feedback that updates the interaction model between the needle and the tissue using an unscented Kalman filter. Results obtained from several needle insertions in a soft tissue phantom showed that the method gives good performance in terms of needle trajectory prediction. This model was also considered in a closed-loop control approach to allow automatic reaching of a target.

This paper deals with the development of a vision-based controller for robot-assisted medical applications. It concerns the use of shearlet coefficients in case of ultrasounds (US) images as visual signal inputs and the design of the associated interaction matrix. The proposed controller was validated in both simulation and on an experimental test bench which consists of a robotic arm holding an US probe in contact with a realistic abdominal phantom. Also, the proposed control scheme was compared to the photometry-based visual servoing approach in order to evaluate its efficiency in different conditions of use (nominal and unfavorable conditions).

The precise location of tumors is an important step in surgical planning that can be obtained from mechanical properties of soft tissues. In this paper we propose a robotic-assisted palpation system that automatically moves an ultrasound probe to optimize the elastography process and improve the resulting elastogram. The main contribution of this work is the use of the elastography modality directly as input of the robot controller. Force measures are also considered in the probe control in order to automatically induce soft tissue deformation needed for real-time elastography imaging process. Moreover, an automatic exploration process is implemented to orient the probe to reach different views of a soft tissue target of interest. This allows to improve the elastogram quality of the element of interest by fusing the information observed from different positions.

This paper presents a new image-based visual servoing approach to control a robotic system equipped with an ultrasound (US) imaging device. The presented method allows an automatic positioning of the probe with respect to an object of interest. Moments-based image features are computed from three orthogonal US images to servo in-plane and out-of-plane motions of the system. An efficient segmentation method, based on graph-cut strategy, is proposed to extract the object contour in each image plane. Simulation results demonstrate that this approach improves upon techniques based on a single 2-D US image in terms of probe positioning. Our method was also validated from robotic experiments performed on a US phantom with the use of a motorized 3-D probe that provides the three US images.

We present a method for the three-dimensional (3D) steering of a flexible needle under 3D ultrasound guidance. The proposed solution is based on a duty-cycling visual servoing strategy we designed in a previous work, and on a new needle tracking algorithm for 3D ultrasound. The flexible needle modeled as a polynomial curve is tracked during automatic insertion using particle filtering. This new tracking algorithm enables real-time closed-loop needle control with 3D ultrasound feedback. Experimental results of a targeting task demonstrate the robustness of the proposed tracking algorithm and the feasibility of 3D ultrasound-guided needle steering.

In this paper we propose a new ultrasound-based visual servoing framework, for the optimization of the positioning of an ultrasound probe manipulated by a robotic arm, in order to improve the quality of the acquired ultrasound images. To this end, we use the recent framework of ultrasound confidence map, which aims at estimating the perpixel quality of the ultrasound signal based on a model of sound propagation in soft tissues. More specifically, we treat the ultrasound confidence maps as a new modality to design a visual servoing control law for image quality optimization. The proposed framework aims at improving ultrasound imaging techniques, such as robotic tele-echography, target tracking or volume reconstruction. Here we illustrate our approach with the application of robotic tele-echography. Experiments are performed on both an ultrasound examination training phantom and ex vivo tissue samples.

In this paper, we present a novel approach for tracking a deformable anatomical target within 3D ultrasound volumes. Our method is able to estimate deformations caused by the physiological motions of the patient. The displacements of moving structures are estimated from an intensity-based approach combined with a physically-based model and has therefore the advantage to be less sensitive to the image noise. Furthermore, our method does not use any fiducial marker and has real-time capabilities. The accuracy of our method is evaluated on real data acquired from an organic phantom. The validation is performed on different types of motions comprising rigid and non-rigid motions. Thus, our approach opens novel possibilities for computer-assisted interventions where deformable organs are involved.

This paper presents a three dimensional ultrasound (3DUS)-based visual servoing technique for intraoperative tracking of the motion of both surgical instruments and tissue targets. In the proposed approach, visual servoing techniques are used to control the position of a virtual ultrasound probe so as to keep a target centered within the virtual probe's field-of-view. Multiple virtual probes can be servoed in parallel to provide simultaneous tracking of instruments and tissue. The technique is developed in the context of robotic beating-heart intracardiac surgery in which the goal of tracking is to both provide guidance to the operator as well as to provide the means to automate the surgical procedure. To deal with the low signal-to-noise ratio (SNR) of the 3DUS volumes, an intensity-based method is proposed that requires no primitive extraction or image segmentation since it directly utilizes the image intensity information as a visual feature. This approach is computationally efficient and can be applied to a wide range of tissue types and medical instruments. This paper presents the first validation of these techniques through offline robot and tissue tracking using actual in vivo cardiac volume sequences from a robotic beating-heart surgery.

In case of motor impairments, steering a wheelchair can become a hazardous task. Typically, along corridors, joystick jerks induced by uncontrolled motions are source of wall collisions. This paper describes a vision based assistance solution for safe indoor semi-autonomous navigation purposes. To this aim, the control process is based on a visual servoing process designed for wall avoidance purposes. As the patient manually drives the wheelchair, a virtual guide is defined to progressively activate an automatic trajectory correction. The proposed solution does not require any knowledge of the environment. Experiments have been conducted over corridors that present different configurations and illumination conditions. Results demonstrate the ability of the system to smoothly and adaptively assist people during their motions.

In this paper, we present a method to localize and track manually inserted needles in real-time using a three-dimensional ultrasound probe mounted on a robotized arm. The system tracks the needle using online image processing. We first propose a new algorithm capable of robustly detecting a needle from the moment it is inserted, without any a priori information on the insertion direction. By combining the random sample consensus (RANSAC) algorithm with Kalman filtering in closed loop, we achieve robust real-time tracking of the needle. In addition, we propose a control scheme to automatically guide the ultrasonic probe in order to keep the needle within the field of view, while aligning its axis with the ultrasound beam. This method will ease the insertion of the needle by the operator, and allow the development of autonomous needle insertion by medical robots.

In this paper, we present an ultrasound (US) visual servoing to control a robotic system equipped with a US probe. To avoid the difficult and time-consuming image segmentation process, we develop a new approach taking as visual input directly the intensity of the image pixels. The analytic form of the interaction matrix that relates the variation of the intensity features to the motion of the probe is established and used to control the six degrees of freedom (dof) of the robotic system. Our approach is applied with a 2-D and a 3-D US probe, and the results that are obtained with both sensors are compared in simulation. The 2-D probe shows good performances for tracking tasks and the 3-D one, which ensures a larger domain of convergence, is more particularly used for positioning tasks. The intensity-based approach is validated through experimental results performed with a realistic abdominal phantom and with animal soft tissue.

This paper presents a multi-task control method to maintain the visibility of an anatomic element of interest while the doctor tele-operates a 2D ultrasound probe held by a medical robot. The prior task consists in automatically maintaining several visual constraints that guarantee an inter-section between the ultrasound image plane and the anatomic object of interest and the second task allows the medical expert to manually apply probe motion through tele-operation. Unlike classical visual servoing technique which continually regulate the current visual features to desired values, our control approach gradually activates the regulation of one or several ultrasound visual features that go close to fixed limits in such a way to keep them in a safe domain. The main advantage of this approach is to give to the clinician the control of all the degrees of freedom of the probe to examine the patient while automatically preserving the visibility of the element of interest if required. Both simulations and experiments performed on an abdominal phantom demonstrate the efficiency of the visibility assistance task.

This paper presents an optimal parametric active contour method based on Fourier descriptors to detect and track a contour from a sequence of images in real-time. The advantage of this approach is that not only convex but also concave contours can be considered. We also propose a method to initialize the active contour using only three user's clicks in the first image of the sequence and to automatically re-initialize it when possible topological changes are detected during the tracking. Moreover, the algorithm is implemented on GPU to ensure the real-time tracking performance.

This paper presents a new image-based approach to the control of a robotic system equipped with an ultrasound imaging device. For diagnostic applications, the proposed method makes it possible to position an ultrasound probe on a desired organ section and to track it by compensating for rigid motions of the organ. Both in-plane and out-of-plane motions of the probe are controlled by the proposed method. The main contribution of this work is the direct use of the ultrasound image as visual feature which spares any segmentation or image processing time consuming step. Simulation and robotic experiments are performed on a realistic abdominal phantom and validate this ultrasound intensity-based visual servoing approach.

This paper presents a visual-servoing method for compensating motion of soft tissue structures using 4D ultra-sound. The motion of soft tissue structures caused by physiological and external motion makes it difficult to investigate them for diagnostic and therapeutic purposes. The main goal is to track non-rigidly moving soft tissue structures and compensate the motion in order to keep a lesion on its target position during a treatment. We define a 3D non-rigid motion model by extending the Thin-Plate Spline (TPS) algorithm. The motion parameters are estimated with intensity-value changes of a points set in a tracking soft tissue structure. Finally, the global rigid motion is compensated with a 6-DOF robot according to the motion parameters of the tracking structure. Simulation experiments are performed with recorded 3D US images of in-vivo soft tissue structures and validate the effectiveness of the non-rigid motion tracking method. Robotic experiments demonstrated the success of our method with a deformable phantom.

Application of the compliance to increase the robustness of the bipedal walker locomotion is the main target of this paper. The control of bipedal robots with point feet is one of the most challenging problems in the domain of hybrid dynamical systems. In order to make an exponentially stable periodic walking for an Acrobot as a planar biped with only one degree of freedom, a controller is designed based on the hybrid zero dynamics analysis. Then, through appropriate alterations in robot structure, the controller complexities are reduced which result in fewer parameters for tuning. Improvement of the robot structure via compliance insertion not only compensates the lower degree of freedom of the control design process, but also expands the domain of stability of the closed-loop system. In this regard, a nonlinear damper is inserted between two legs of Acrobot to enable it to walk on a wider range of slopes. The main controller is designed for walking on a flat terrain and the compliance of the damper empowers it to make stable walking on slopes up to 17°. The simulation results confirmed the efficiency of the proposed approach.

This paper presents a new image-based approach to control a robotic system equipped with an ultrasound imaging device. Moments based image features are extracted from three orthogonal ultrasound images to servo in-plane and out-of-plane motions of the system. Experimental results demonstrate that this approach improves upon techniques based on a single 2D US image in term of probe positioning. The second contribution of this paper is to use this method to perform a multimodal registration task by formulating it as a virtual visual servoing problem. Multimodal registration experiments performed with an ultrasound phantom containing an egg-shaped object provide a first experimental validation of the proposed method.

This paper presents a visual-servoing method that is based on 2-D ultrasound (US) images. The main goal is to guide a robot actuating a 2-D US probe in order to reach a desired cross-section image of an object of interest. The method we propose allows the control of both in-plane and out-of-plane probe motions. Its feedback visual features are combinations of moments extracted from the observed image. The exact analytical form of the interaction matrix that relates the image-moments time variation to the probe velocity is developed, and six independent visual features are proposed to control the six degrees of freedom of the robot. In order to endow the system with the capability of automatically interacting with objects of unknown shape, a model-free visual servoing is developed. For that, we propose an efficient online estimation method to identify the parameters involved in the interaction matrix. Results obtained in both simulations and experiments validate the methods presented in this paper and show their robustness to different errors and perturbations, especially those inherent to the noisy US images.

This paper presents a new model-free visual servoing that is able to servo a robotized 2D ultrasound probe that interacts with a soft tissue object. It makes direct use of the B-mode ultrasound images in order to reach a desired one. This approach does not require the 3D model of the object nor its location in the 3D space. The visual features are based on image moments. The exact analytical form of the interaction matrix relating the image moments variation to the probe velocity is modelled. To perform model-free servoing, the approach combines the image points coordinates with the probe pose to estimate efficiently 3D parameters required in the control law. The approach is validated with simulation and experimental results showing its robustness to different errors and perturbations.

A new visual servoing method based on B-mode ultrasound images is proposed to automatically control the motion of a 2D ultrasound probe held by a medical robot in order to reach a desired B-scan image of an object of interest. In this approach, combinations of image moments extracted from the current observed object cross-section are used as feedback visual features. The analytical form of the interaction matrix, relating the time variation of these visual features to the probe velocity, is derived and used in the control law. Simulations performed with a static ultrasound volume containing an egg-shaped object, and in-vitro experiments using a robotized ultrasound probe that interacts with a rabbit heart immersed in water, show the validity of this new approach and its robustness with respect to modeling and measurements errors.

This paper presents a new visual servoing method that is able to stabilize a moving area of soft tissue within an ultrasound B-mode imaging plane. The approach consists of moving the probe in order to minimize the relative position between a target imaging plane and the ultrasound plane observed by the probe of the moving tissue target. The problem is decoupled into motion out-of-plane and motion within plane. For the former, a new original method based on the speckle information contained in the images is developed. For the latter, an image region tracker is used to provide the in-plane motion. A visual servoing control scheme is then developed to perform the tracking robotic task. The method is validated on simulated motions of a probe on a static ultrasound volume acquired from a phantom.

Robotized ultrasound examination offers on one hand more accuracy for the medical diagnostic and on the other hand a suitable assistance to radiologists. In the literature, very few researches deal with ultrasound image-based control of a medical robot holding the ultrasound probe. This paper presents a new visual servoing approach allowing the automatic guidance of the ultrasound probe. This technique is based on the modeling of the coupling between the ultrasound probe and egg shaped objects. The robotic task consists in reaching a desired image from an arbitrary initial position. The aim of this task is to assist the radiologist diagnostic by automatically moving the ultrasound probe in order to find the ultrasound image view of a tumour cross-section registered in a previous date. This work presents simulation results of the proposed ultrasound vision-based control scheme

Three-dimensional free-hand ultrasound imaging consists of capturing a set of ultrasound images with a 2D ultrasound system and their respective locations in order to position them in a 3D reference frame. Usually the clinician performs the acquisition manually through the use of an optical or magnetic localization system attached to the ultrasound probe. To assist the clinician, we propose to use a robotic system to automatically move the ultrasound probe and measure its position. As for manual 3D ultrasound imaging, it is crucial to know precisely the spatial calibration parameters of the ultrasound system in order to perform accurate 3D imaging. Therefore, we propose to automate the spatial calibration procedure. A robotic task is developed to automatically position the ultrasound image on the intersection point of a cross-wire phantom used for spatial calibration. To perform this task, a new visual servoing technique based on 2D ultrasound images is used to control automatically the motion of the ultrasound probe held by a medical robot

A new visual servoing technique based on 2D ultrasound image is proposed in order to control the motion of an ultrasound probe held by a medical robot. In opposition to a standard camera which provides a projection of the 3D scene to a 2D image, ultrasound information is strictly in the observation plan of the probe and consequently visual servoing techniques have to be adapted. In this paper the coupling between the ultrasound probe and a motionless crossed string phantom used for probe calibration is modeled. Then a robotic task is developed which consists to position the ultrasound image on the intersection point of the crossed string phantom while moving the probe to different orientations. The goal of this task is to optimize the procedure of spatial parameters calibration of 3D ultrasound systems.

The objective of this project, named MESIA (Multifunctional Micro-Endoscope Head System for Industrial Applications) is to develop an integrated multisensor system and to create intelligent sensor fusion for detection of micro-cracks. This paper is focused on the modeling of a 2-DOF pan-tilt platform actuated by Ni-Ti shape memory alloy (SMA) wires and antagonistic mechanical springs in order to control visually the CCD camera motion. A Preisach model is used and experimentally identified. The derived model is then exploited to design a position controller which compensates for the hysteretic nonlinearity. Finally, by controlling the catheter head orientation through a head motion tracker, in-pipe visual tracking has been experimentally verified from a remote site.

This paper presents a robotic vision system that automatically retrieves and positions surgical instruments during robotized laparoscopic surgical operations. The instrument is mounted on the end-effector of a surgical robot which is controlled by visual servoing. The goal of the automated task is to safely bring the instrument at a desired three-dimensional location from an unknown or hidden position. Light-emitting diodes are attached on the tip of the instrument, and a specific instrument holder fitted with optical fibers is used to project laser dots on the surface of the organs. These optical markers are detected in the endoscopic image and allow localizing the instrument with respect to the scene. The instrument is recovered and centered in the image plane by means of a visual servoing algorithm using feature errors in the image. With this system, the surgeon can specify a desired relative position between the instrument and the pointed organ. The relationship between the velocity screw of the surgical instrument and the velocity of the markers in the image is estimated online and, for safety reasons, a multistages servoing scheme is proposed. Our approach has been successfully validated in a real surgical environment by performing experiments on living tissues in the surgical training room of the Institut de Recherche sur les Cancers de l'Appareil Digestif (IRCAD), Strasbourg, France.

This paper presents a robotic vision system that automatically retrieves and positions surgical instruments in robotized laparoscopic surgery. The surgical instrument is mounted on the end-effector of a surgical robot which can be controlled by automatic visual feedback. The goal of the automated task is to bring the instrument at a desired location from an unknown or hidden position. To achieve this task, a special instrument-holder is designed with optical fibers and collimators. This instrument-holder projects laser dot patterns onto the organ surface which are seen in the endoscopic images. Then, the instrument is retrieved and centered in the image plane using a visual servoing algorithm. With this system, the surgeon can also specify a desired position for the instrument in the image. Our approach is successfully validated in a real surgical environment by performing experiments on living animals in the surgical training room of IRCAD.

In this paper, we present a new solution to laparoscopic manipulation based on force feedback control. This method allows to both explicitly control the forces applied to the patient through the trocar, and to precisely control the position of the surgical instrument. It does not require any geometrical model of the operative environment, nor any fine robot base placement prior to the instrument insertion. Different control strategies, involving different kinds of sensory equipments are proposed. They are experimentally validated on a laboratory apparatus.

In this paper, we address the problem of controlling the motion of a surgical instrument close to an unknown organ surface by visual servoing in the context of robotized laparoscopic surgery. To achieve this goal, a visual servoing algorithm is developed that combines feature errors in the image and errors in depth measurements. The relationship between the velocity screw of the surgical instrument, the depth and the motion field is defined and a two-stage servoing scheme is proposed. In order to measure the orientation and the depth of the instrument with respect to the organ, a laser dot pattern is projected on the organ surface and optical markers are stuck on the instrument. Our work has been successfully validated with a surgical robot by performing experiments on living tissues in the surgical training room of IRCAD.