Minimally invasive surgery involves using flexible medical instruments such as endoscopes and catheters. Magnetically actuated catheters can provide improved steering precision over conventional catheters. However, besides the actuation method, an accurate tip position is required for precise control of the medical instruments. In this study, the tip position obtained from transverse 2D ultrasound images and multicore optical shape sensors are combined using a robust sensor fusion algorithm. The tip position is tracked in the ultrasound images using a template-based tracker and a convolutional neural network based tracker, respectively. Experimental results for a rhombus path are presented, where data obtained from both tracking sources are fused using Luenberger and Kalman state estimators. The mean and standard deviation of the Euclidean error for the Luenberger observer is 0.2 ± 0.11 [mm] whereas for the Kalman filter it is 0.18 ± 0.13 [mm], respectively.

Minimally invasive surgery can benefit greatly from utilizing micro-agents. These miniaturized agents need to be clearly visualized and precisely controlled to ensure the success of the surgery. Since medical imaging modalities suffer from low acquisition rate, multi-rate sampling methods can be used to estimate the intersample states of micro-agents. Hence, the sampling rate of the controller can be virtually increased even if the position data is acquired using a slow medical imaging modality. This study presents multi-rate Luenberger and Kalman state estimators for visual tracking of micro-agents. The micro-agents are tracked using sum of squared differences and normalized cross correlation based visual tracking. Further, the outputs of the two methods are merged to minimize the tracking error and prevent tracking failures. During the experiments, the micro-agents with different geometrical shapes and sizes are imaged using a 2D ultrasound machine and a microscope, and manipulated using electromagnetic coils. The multi-rate state estimation accuracy is measured using a high speed camera. The precision of the tracking and multi-rate state estimation are verified experimentally under challenging conditions. For this purpose, an elliptical shaped magnetic micro-agent with a length of 48 pixels is used. Maximum absolute error in x and y axes are 2.273 and 2.432 pixels for an 8-fold increase of the sample rate (25 frames per second), respectively. During the experiments, it was observed that the micro-agents could be tracked more reliably using normalized cross correlation based visual tracking and inters ample states could be estimated more accurately using Kalman state estimator. Experimental results show that the proposed method could be used to track micro-agents in medical imaging modalities and estimate system states at intermediate time instants in real-time.

Percutaneous needle procedures are mostly carried out with the guidance of 2D ultrasound (US) imaging. US images are inherently noisy and their resolutions are low. Hence, target tracking can be challenging. Image based tracking methods can be used to track the needle and the target. This paper proposes visual tracking of multiple moving points, such as biopsy needles and targets, in 2D US images using normalized cross correlation and mutual information similarity functions. Both moving and deformable targets can be tracked. An affine motion model is used for small and moving target tracking and a thin plate spline motion model is used for deformable target tracking. During the tracking, needle and target template images are updated with a template update strategy. Also, tracking outputs of normalized cross correlation and mutual information are fused using the Kalman filter to reduce the tracking error. During the experiments, needle is inserted using a needle insertion robot. 2D US probe is attached to a robotic arm's end effector to servo the probe along the needle insertion path. Proposed needle and target tracking methods were tested with phantoms. Accuracies of the needle tip and moving target tracking methods were measured using an optical tracking system. Experimental results showed that the proposed tracking method could be used to simultaneously track the needle tip and the targets in real-time in 2D US guided percutaneous needle procedures.

Ultrasound (US) is one the most commonly used medical imaging techniques in percutaneous needle procedures. However, US images are inherently noisy and contain excessive number of artifacts. Hence, it is not easy to track the needle tip in the US images during the needle insertions. At this point, image based visual tracking techniques can be used for needle tip tracking. This paper presents a method for visual tracking of biopsy needles in 2D US images using sum of squared differences and sum of conditional variances. Second order Gauss-Newton optimization is used to decrease processing time and make the tracking more robust. The needle template images used in the method are updated with a strategy to prevent needle loss and detection failures during tracking. The paper also explains how to identify needle losses during tracking and how to recover the needle position without using a needle localization algorithm. We demonstrate the precision of the visual needle tip tracking method with experiments under challenging tracking conditions.

In the percutaneous needle procedures using ultrasound (US) imaging, the needle should be detected precisely to avoid damage to the tissue and to get the samples from the appropriate site. Excessive artifacts and low resolution of the US images make it difficult to detect the needle and its tip. It is possible to enhance the needle image using image processing; and this work proposes a novel needle detection method in 2D US images based on the Gabor filter. This method enhances the needle outline while suppressing the other structures in the image. First, the needle insertion angle is estimated and then the needle trajectory is found with the RANSAC line estimator. The experiments with three different phantoms showed that the algorithm is robust and could work in percutaneous needle procedures using US images.