Contemporary endoscopic simultaneous localization and mapping (SLAM) methods accurately compute endoscope poses; however, they only provide a sparse 3-D reconstruction that poorly describes the surgical scene. We propose a novel dense SLAM method whose qualities are: 1) monocular, requiring only RGB images of a handheld monocular endoscope; 2) fast, providing endoscope positional tracking and 3-D scene reconstruction, running in parallel threads; 3) dense, yielding an accurate dense reconstruction; 4) robust, to the severe illumination changes, poor texture and small deformations that are typical in endoscopy; and 5) self-contained, without needing any fiducials nor external tracking devices and, therefore, it can be smoothly integrated into the surgical workflow. It works as follows. First, accurate cluster frame poses are estimated using the sparse SLAM feature matches. The system segments clusters of video frames according to parallax criteria. Next, dense matches between cluster frames are computed in parallel by a variational approach that combines zero mean normalized cross correlation and a gradient Huber norm regularizer. This combination copes with challenging lighting and textures at an affordable time budget on a modern GPU. It can outperform pure stereo reconstructions, because the frames cluster can provide larger parallax from the endoscope's motion. We provide an extensive experimental validation on real sequences of the porcine abdominal cavity, both in-vivo and ex-vivo. We also show a qualitative evaluation on human liver. In addition, we show a comparison with the other dense SLAM methods showing the performance gain in terms of accuracy, density, and computation time.

• We review the state of the art about laparoscopic augmented reality with 270+ recent references. • We highlight that most methods consist in a static registration followed by augmentation updates. • We propose a clear classification of all methods and weigh pros and cons for each of them. • This survey is comprehensive and self-contained thanks to various backgrounds on related topics. This article establishes a comprehensive review of all the different methods proposed by the literature concerning augmented reality in intra-abdominal minimally invasive surgery (also known as laparoscopic surgery). A solid background of surgical augmented reality is first provided in order to support the survey. Then, the various methods of laparoscopic augmented reality as well as their key tasks are categorized in order to better grasp the current landscape of the field. Finally, the various issues gathered from these reviewed approaches are organized in order to outline the remaining challenges of augmented reality in laparoscopic surgery .

The use of augmented reality in minimally invasive surgery has been the subject of much research for more than a decade. The endoscopic view of the surgical scene is typically augmented with a 3D model extracted from a preoperative acquisition. However, the organs of interest often present major changes in shape and location because of the pneumoperitoneum and patient displacement. There have been numerous attempts to compensate for this distortion between the pre- and intraoperative states. Some have attempted to recover the visible surface of the organ through image analysis and register it to the preoperative data, but this has proven insufficiently robust and may be problematic with large organs. A second approach is to introduce an intraoperative 3D imaging system as a transition. Hybrid operating rooms are becoming more and more popular, so this seems to be a viable solution, but current techniques require yet another external and constraining piece of apparatus such as an optical tracking system to determine the relationship between the intraoperative images and the endoscopic view. In this article, we propose a new approach to automatically register the reconstruction from an intraoperative CT acquisition with the static endoscopic view, by locating the endoscope tip in the volume data. We first describe our method to localize the endoscope orientation in the intraoperative image using standard image processing algorithms. Secondly, we highlight that the axis of the endoscope needs a specific calibration process to ensure proper registration accuracy. In the last section, we present quantitative and qualitative results proving the feasibility and the clinical potential of our approach.

The spatial configuration of actuated flexible instruments is fundamental for control applications in robotic noscar surgery. In these operations, the instruments are inserted in the channels of a flexible guide equipped with an endoscopic camera. In this paper we propose to estimate the position of the instruments of the Anubis platform (Karl Storz) using the endoscopic images provided by the embedded camera. In this system flexible instruments have 3-DOF (translation and rotation in the channel and deflection). Application of standard approaches for 3D location with such system do not provide good accuracy because of uncertainties on several model parameters. To cope with these uncertainties, supervised learning methods has been explored. With the help of colored visual markers attached to the instruments, the proposed approach consists of an image segmentation stage followed by a position estimation stage. Firstly, the markers are segmented in the images using an AdaBoost classifier manually trained on in-vivo images. Subsequently, the resulting blobs are used as input data of an approximation function trained using ground truth information provided by a magnetic sensor. A comparison with two other model-based methods showed the potentialities of such an approach on real devices.

In this paper, we present a new stereo visual servoing (SVS) with decoupling properties. The proposed approach is not related to the position-based nor image-based categories as the state vectors are built from the two-view affine geometry and the visual measurements. The Plücker coordinates are used to model many types of visual features like 3-D lines but also point pairs in the same framework. With such a representation, a SVS is carried out with a very few number of features, provided the Fundamental matrix and the infinite homography are both available. This choice is also motivated so as to better reject outliers, to ensure the epipolar constraint and to provide efficient decoupled rotational and translational kinematic control laws with closed-form interaction matrices. Several tasks functions have been designed with low-dimensional error vectors which make the proof of stability and convergence study possible.

Structured light based patterns provide a means to capture the state of an object shape. However it may be inefficient when the object is freely moving, when its surface contains high curvature parts or in out of depth of field situations. For image-based robotic guidance in unstructured and dynamic environment, only one shot is required for capturing the shape of a moving region-of-interest. Then robust patterns and real-time capabilities must be targeted. To this end, we have developed a novel technique for the generation of coded patterns directly driven by the Hamming distance. The counterpart is the big amount of codes the coding/decoding algorithms have to face with a high desired Hamming distance. We show that the mean Hamming distance is a useful criterion for driving the patterns generation process and we give a way to predict its value. Furthermore, to ensure local uniqueness of codewords with consideration of many incomplete ones, the Perfect Map theory is involved. Then, we describe a pseudorandom/exhaustive algorithm to build patterns with more than 200×200 features, in a very short time, thanks to a splitting strategy which performs the Hamming tests in the codeword space instead of the pattern array. This leads to a significant reduction of the computational complexity and it may be applied to other purposes. Finally, real-time reconstructions from single images are reported and results are compared to the best known which are outperformed in many cases.

In this paper, we address a robust matching technique based on coded structured lighting (SL) to achieve real-time 3-D reconstructions. To that purpose, most existing approaches involve color or grey levels coding but they are well-known to be sensitive to spectral properties and texture of the viewed surfaces. Therefore, the overall robustness of the proposed technique comes first from the geometrical features used in conjunction with the SL neighbourhood scheme to carry out pattern coding. Second, a desired minimum Hamming distance between features' codewords drives the pattern design. This is suited for autonomous navigation in unknown environments as this parameter enables codewords correction capabilities during the decoding stage in real-time. Furthermore, we take advantage of the known epipolar geometry by including projective invariants between corresponding epipolar lines directly in the pattern components. Thus, the final pattern displays relocated and reoriented (cuneiform) features along the epipolar lines. This grouping also contributes to reduce the search space more and results in a significantly less constrained coding. Finally, we show how this coding and this non-grid based pattern offer efficient and fast correction of mislabeled features due to blurring, spectral harmful effects and surfaces discontinuities prior to the 3-D reconstruction of real scenes.

Solving visual features' correspondence and dealing with missing data are two factors of limitations for points registration techniques. To tackle this problem, we conceived a pattern, primarily designed for structured lighting vision systems, which may also be used for camera calibration purposes. The pattern design previously presented provides a huge of benefits. Among them, we firstly present a new calibration technique of a structured lighting system and secondly an automatic distortion compensation based on a printed pattern. These two well-known issues are very useful in 3D vision-based metrology with range data, for instance for model-based visual robot control, especially when the model is incrementally built with a real-time 3D reconstruction of moving surfaces. Perhaps, one of the most significant profit with a high Hamming distance pattern is the ability to reliably decode its projected individual elements even if several of items are missing, as it greatly extends the range of measurements volume. A technique which solves the distortion parameters by means of a robust M-estimator algorithm is presented. It uses a printed pattern and it allows the distortion be corrected with a single view and without the computation of other (intrinsic/extrinsic) parameters, even in presence of occlusions. Experimental results, in one hand by means of a printed pattern for the distortion compensation of a rigid endoscope and on the other hand by means of a projected pattern for the calibration of the structured lighting system, show very good performance for the 3-D reconstruction.

In this paper we present a new monochromatic pattern for a robust structured light coding based on the spatial neighborhood scheme using the M-array approach. The proposed pattern is robust as it allows a high error rate characterized by an average Hamming distance higher than 6. We tackle the design problem with the definition of a small set of symbols associated to simple geometrical primitives. One of these primitives embeds the local orientation of the pattern. This is helpful while performing the search for the relevant neighborhood during the decoding process. The aim of this work is to use this pattern for the real-time 3-D reconstruction of dynamic scenes, particularly in endoscopic surgery, with fast and reliable detection and decoding stages. Ongoing results are presented to assess both the capabilities of the proposed pattern and the reliable decoding algorithm with projections onto simple 3-D scenes and onto internal structures of a pig abdomen.

In this paper we present a new pattern for the robust coding of the structured light based on the spatial neighborhood scheme and by means of the M-array approach. The pattern is robust as it allows a high error rate characterized by a Hamming distance higher than 3. We tackle the design problem with the definition of a small set of symbols associated to geometrical features with simple shapes. We benefit from the stripe, as it is one of the considered primitives, to embed the local orientation of the pattern. This is helpful while performing the search for the relevant neighborhood during the decoding process. The aim of this work is to use this pattern for the 3 - D reconstruction of dynamic scenes with fast and reliable detection and decoding stages. Ongoing results are presented to assess both the capabilities of the proposed pattern and the decoding algorithm with projections onto simple 3 - D scenes.

This paper deals with the visual servoing of textured surfaces inside the human abdomen with a laparoscope for the robot-assisted minimally invasive surgery (MIS). The well-known image-based visual servoing (IBVS) is one of the most common approaches used for model-based servoing. When no CAD model is available, the efficient second-order minimization (ESM) tracking developed by Malis (2002, 2004) for grey-level images is one of the powerful recent techniques which is extended here to color images so as to handle occluded parts of the region of interest (ROI). Firstly, the ROI is splitted into small areas and a histogram-based color feature comparison of image areas is presented. For each frame and for each area, a metric based on the Bhattacharyya criterion is used to select the contributing areas for the computation of the planar homography between views. Secondly, since for any MIS technique, the endoscopic lens is passing through an insertion point on the abdominal wall, a specific control strategy is developed to perform the ESM tracking with a 4-DOF surgical robot. The method presented in this paper has been validated with several video sequences. Experimental results show that the tracking method is efficient even with more than 75 % of the tracked ROI occluded. Finally, the model-free visual servoing has been performed with the AESOP surgical robot and a training box. Even if the convergence rate is a little bit slow, the desired region is always reached.

In this paper, we address the problem of the pose recovery of a straight homogeneous circular cylinder (SHCC) from its apparent contour in a single 2-D image. In many real-world situations, one may encounter cylindrical objects especially in man-made structures. Object models based on surfaces of revolution, in particular cylinders, are suitable for many fields, like the automatic assembly, the human motion capture and also in medical image-based robotic guidance. To model the geometry of a SHCC, we first introduce a singular matrix which represents this degenerate quadric and we show it can be expressed with the Plucker coordinates of its symmetry axis. We demonstrate that the perspective projection of a SHCC is related to the pose parameters and for this model-based pose estimation problem, we present a degenerate conic-based fitting method which has some connections with the estimation of the fundamental matrix. We provide a closed-form solution for the pose determination (4 dof). Compared to earlier works in this field, the proposed approach exhibits some geometric properties of SHCCs, it can deal with partial occlusions of apparent contours and it provides an efficient direct pose solution for the symmetry axis. Simulated data and real images are used to validate the fitting and pose computation. Finally, to highlight the effectiveness of the proposed method to deal with apparent contours in a poor structured environment, we apply this work to the localization of instruments used in laparoscopic surgery.

Robot-assisted surgical procedures require to generate paths for surgical instruments during a planning step and then to follow automatically the computed trajectories as close as possible. Most of the existing procedures use external localization devices to control the robot. However, these devices are not adequate for minimally invasive interventions such as laparoscopic surgery because this kind of surgery involves large lever effects which make high accuracy unreachable. In this paper, we propose a path following method based on the use of the endoscopic camera and instruments with markers. It mainly stands on an image-based visual servoing scheme and on the estimation of the 3D position of the incision point in the abdominal wall. We show that this method can lead to precise tracking despite the errors on the position of the incision point. The proposed method is finally demonstrated in vitro in an automatic suturing intervention.

The work presented in this paper addresses the problem of the stitching task in laparoscopic surgery using a circular needle and a conventional 4 DOFs needle-holder. This task is particularly difficult for the surgeons because of the kinematics constraints introduced by the trocar. So as to assist the surgeons, we propose to compute possible pathes for the needle through the tissues, which limits as much as possible tissues deformations while driving the needle towards the desired target. The article proposes a kinematic analysis and a geometric modeling of the stitching task. Based on this, some simple but useful information can be obtained to help the surgeon. The description of the task with well-chosen state variables allows to simply express the path planning problem. Conditions for the existence of acceptable pathes are given and a method to compute possible pathes is presented. Resulting pathes are shown to be satisfactory even under awkward configurations.

In this paper, the detection and localization of grey regions in color images is addressed. This work has been developed in the scope of the robotized laparoscopic surgery, specifically for surgical procedures occurring inside the abdominal cavity. Since very few works have been already published about that purpose, some existing algorithms have been selected and brought together to achieve a robust color segmentation, as fast as possible. The foreseen application is a good training ground to evaluate these algorithms since main difficulties came from the complexity of the scene, the moving background due to breathing motion, the high surface reflectance, the non-uniform and time-varying lighting conditions. Nevertheless, to achieve the image segmentation suitable for robot control, we propose a new approach, without markers, based on a recursive thresholding of the histogram of a new purity color attribute and region growing. The main contribution of this work is threefold and consists in: the definition of a new color purity component, a selection of reliable, fast and robust existing video processings for the above-mentioned application areas, improving some existing video processings to enhance color properties either to homogenize regions and to emphasize the saturation feature of chromatic pixels. The usefulness of the proposed set of sequential processings has been successfully validated with image sequences of an endoscope to efficiently extracting boundaries of a cylindrical needle-holder with a sampling rate of 5 Hz.

In this paper, we address the problem of precisely recovering the 3D (three-dimensional) pose of 3D shape fiducials from images obtained by means of an uncalibrated computed tomography (CT) imaging device. The main goal in this work is to model and estimate the geometric transformation relating line fiducials to their projections in cross-sectional images. To do so, we propose techniques, which solve the points to lines correspondence using closed-form and numerical algorithms. A geometric transformation with eight degrees of freedom (rotation, translation and anisotropic scaling) is used to model both a rigid-body transformation and a scaling transformation accounting for CT scan intrinsic parameters. Furthermore, an estimation of error bounds in space is given when image data are affected by noise. Real experiments show that the proposed method provides good results on a set of CT images from many viewpoints.

This paper presents a robotic vision system that automatically retrieves and positions surgical instruments during robotized laparoscopic surgical operations. The instrument is mounted on the end-effector of a surgical robot which is controlled by visual servoing. The goal of the automated task is to safely bring the instrument at a desired three-dimensional location from an unknown or hidden position. Light-emitting diodes are attached on the tip of the instrument, and a specific instrument holder fitted with optical fibers is used to project laser dots on the surface of the organs. These optical markers are detected in the endoscopic image and allow localizing the instrument with respect to the scene. The instrument is recovered and centered in the image plane by means of a visual servoing algorithm using feature errors in the image. With this system, the surgeon can specify a desired relative position between the instrument and the pointed organ. The relationship between the velocity screw of the surgical instrument and the velocity of the markers in the image is estimated online and, for safety reasons, a multistages servoing scheme is proposed. Our approach has been successfully validated in a real surgical environment by performing experiments on living tissues in the surgical training room of the Institut de Recherche sur les Cancers de l'Appareil Digestif (IRCAD), Strasbourg, France.

This paper presents a robotic vision system that automatically retrieves and positions surgical instruments in robotized laparoscopic surgery. The surgical instrument is mounted on the end-effector of a surgical robot which can be controlled by automatic visual feedback. The goal of the automated task is to bring the instrument at a desired location from an unknown or hidden position. To achieve this task, a special instrument-holder is designed with optical fibers and collimators. This instrument-holder projects laser dot patterns onto the organ surface which are seen in the endoscopic images. Then, the instrument is retrieved and centered in the image plane using a visual servoing algorithm. With this system, the surgeon can also specify a desired position for the instrument in the image. Our approach is successfully validated in a real surgical environment by performing experiments on living animals in the surgical training room of IRCAD.

In this paper, we address the problem of controlling the motion of a surgical instrument close to an unknown organ surface by visual servoing in the context of robotized laparoscopic surgery. To achieve this goal, a visual servoing algorithm is developed that combines feature errors in the image and errors in depth measurements. The relationship between the velocity screw of the surgical instrument, the depth and the motion field is defined and a two-stage servoing scheme is proposed. In order to measure the orientation and the depth of the instrument with respect to the organ, a laser dot pattern is projected on the organ surface and optical markers are stuck on the instrument. Our work has been successfully validated with a surgical robot by performing experiments on living tissues in the surgical training room of IRCAD.

In this paper, we present a model-based vision system using a CCD camera mounted on the end-effector of an industrial robot. We describe an autonomous monocular vision system which is used as a feedback signal to control the position and orientation of the manipulator with respect to a dynamic object part of a 3D scene. Without any previous knowledge of the part's placement or motion, this visual feedback would allow a robot to track a pattern in real time. Applications for such a vision-guided robot include 3D object recognition, by extracting 2D image features and rebuilding 3D features. This is performed via rigidity constraint and search comparison in model database geometric inference. Kinematic redundancy is used for general solution of joint velocity represented by the generalized inverse of Jacobian matrix. The problem of redundancy utilization can generally be formulated in the framework of tasks with the order of priority. We present a visual servoing scheme using the task function approach in term of 6 degrees of freedom of robot's gripper regulation. Redundancy theory allows to control position and orientation of the end-effector of an industrial robot in less than 80 ms of cycle time.>