Colorectal cancer is one of the leading causes of cancer-related deaths worldwide, although it can be effectively treated if detected early. Teleoperated flexible endoscopes are an emerging technology to promote participation in these preventive screenings. Real-time pose estimation is therefore essential to enable feedback to the robotic endoscope's control system. Vision-based endoscope localization approaches are a promising avenue, since they do not require extra sensors on board the endoscopes. In this work, we compare several state-of-the-art algorithms for computing the image motion (optical flow), which is then used with a supervised learning strategy to provide an accurate estimate of the 6 degree of freedom endoscope motion. The method is validated using a robotically actuated endoscope in a human colon simulator, and represents a preliminary effort towards testing with clinical video data.

Augmented-Reality (AR) displays increase surgeon's visual awareness of high-risk surgical targets (e.g., the location of a tumor) by accurately overlaying pre-operative radiological 3-D model onto the intra-operative laparoscopic video. Existing AR systems lack in accuracy and robustness against frequent illumination changes, camera motions, and organ occlusions, which rapidly cause the loss of image (anchor) points, and thus the loss of the AR display after a few seconds. In this paper, we present a new AR system, which represents the first step toward long term and accurate augmented surgical display. Our system leverages feature matching to automatically recover the overlay by predicting the image locations of a high number of anchor points that were lost after a sudden image change. Additionally, a weighted sliding-window least-squares approach is also used to increase the accuracy of the AR display over time. The effectiveness of the proposed system in maintaining a long term, stable, and accurate augmentation has been tested over a set of real partial-nephrectomy laparascopic monocular videos from a DaVinci surgical robot.

This paper proposes a novel active target tracking strategy for a team of cooperating quadrotors equipped with 3-D range-finding sensors. The work builds upon previous research of the authors, and adopts a realistic nonlinear dynamic model for the quadrotors. A hierarchical controller is designed for the generation and tracking of the desired optimal trajectories of the aerial vehicles, and a discrete-time Kalman filter is used for fusing their local estimates of the target position. Under suitable conditions, it is shown that the cost function for the D-optimality criterion that the quadrotors aim at collaboratively reduce, possesses a single global minimum and no local minima. Numerical simulations and real-world experiments show the effectiveness of the proposed control strategy.

The ability to find image similarities between two distinct endoscopic views is known as feature matching, and is essential in many robotic-assisted minimally-invasive surgery (MIS) applications. Differently from feature-tracking methods, feature matching does not make any restrictive assumption about the chronological order between the two images or about the organ motion, but first obtains a set of appearance-based image matches, and subsequently removes possible outliers based on geometric constraints. As a consequence, feature-matching algorithms can be used to recover the position of any image feature after unexpected camera events, such as complete occlusions, sudden endoscopic-camera retraction, or strong illumination changes. We introduce the hierarchical multi-affine (HMA) algorithm, which improves over existing feature-matching methods because of the larger number of image correspondences, the increased speed, and the higher accuracy and robustness. We tested HMA over a large (and annotated) dataset with more than 100 MIS image pairs obtained from real interventions, and containing many of the aforementioned sudden events. In all of these cases, HMA outperforms the existing state-of-the-art methods in terms of speed, accuracy, and robustness. In addition, HMA and the image database are made freely available on the internet.

The ability to find similar features between two distinct views of the same scene (feature matching) is essential in robotics and computer vision. We are here interested in the robotic-assisted minimally-invasive surgical scenario, for which feature matching can be used to recover tracked features after prolonged occlusions, strong illumination changes, image clutter, or fast camera motion. In this paper we introduce the Hierarchical Multi-Affine (HMA) feature-matching algorithm, which improves over the existing methods by recovering a larger number of image correspondences, at an increased speed and with a higher accuracy and robustness. Extensive experimental results are presented that compare HMA against existing methods, over a large surgical-image dataset and over several types of detected features.

We present our novel Adaptive Multi-Affine (AMA) feature-matching algorithm that finds correspondences between two views of the same non-planar object. The proposed method only uses monocular images to robustly match clusters of 2-D features according to their relative position on the object surface; finally, AMA adaptively finds the best number of clusters that maximizes the number of matching features. We use AMA to recover a feature tracker from failure (e.g., loss of points due to occlusions or deformations), by robustly matching the features in the images before and after such events. This is paramount in Augmented-Reality (AR) systems for Minimally-Invasive Surgery (MIS) to cope for frequent occlusions and organ deformations that can cause the tracked image-points to drastically reduce (or even disappear) in the current video. We validated our approach on a large set of MIS videos of partial-nephrectomy surgery; AMA achieves an increased number of matches, as well as a reduced feature-matching error when compared to state-of-the-art method.