2D/3D image registration to align a 3D volume and 2D X-ray images is a challenging problem due to its ill-posed nature and various artifacts presented in 2D X-ray images. In this paper, we propose a multi-agent system with an auto attention mechanism for robust and efficient 2D/3D image registration. Specifically, an individual agent is trained with dilated Fully Convolutional Network (FCN) to perform registration in a Markov Decision Process (MDP) by observing a local region, and the final action is then taken based on the proposals from multiple agents and weighted by their corresponding confidence levels. The contributions of this paper are threefold. First, we formulate 2D/3D registration as a MDP with observations, actions, and rewards properly defined with respect to X-ray imaging systems. Second, to handle various artifacts in 2D X-ray images, multiple local agents are employed efficiently via FCN-based structures, and an auto attention mechanism is proposed to favor the proposals from regions with more reliable visual cues. Third, a dilated FCN-based training mechanism is proposed to significantly reduce the Degree of Freedom in the simulation of registration environment, and drastically improve training efficiency by an order of magnitude compared to standard CNN-based training method. We demonstrate that the proposed method achieves high robustness on both spine cone beam Computed Tomography data with a low signal-to-noise ratio and data from minimally invasive spine surgery where severe image artifacts and occlusions are presented due to metal screws and guide wires, outperforming other state-of-the-art methods (single agent-based and optimization-based) by a large margin.

Cryo-balloon catheters have attracted an increasing amount of interest in the medical community as they can reduce patient risk during left atrial pulmonary vein ablation procedures. As cryo-balloon catheters are not equipped with electrodes, they cannot be localized automatically by electro-anatomical mapping systems. As a consequence, X-ray fluoroscopy has remained an important means for guidance during the procedure. Most recently, image guidance methods for fluoroscopy-based procedures have been proposed, but they provide only limited support for cryo-balloon catheters and require significant user interaction. To improve this situation, we propose a novel method for automatic cryo-balloon catheter detection in fluoroscopic images by detecting the cryo-balloon catheter's built-in X-ray marker. Our approach is based on a blob detection algorithm to find possible X-ray marker candidates. Several of these candidates are then excluded using prior knowledge. For the remaining candidates, several catheter specific features are introduced. They are processed using a machine learning approach to arrive at the final X-ray marker position. Our method was evaluated on 75 biplane fluoroscopy images from 40 patients, from two sites, acquired with a biplane angiography system. The method yielded a success rate of 99.0% in plane A and 90.6% in plane B, respectively. The detection achieved an accuracy of 1.00 mm±0.82 mm in plane A and 1.13 mm±0.24 mm in plane B. The localization in 3-D was associated with an average error of 0.36 mm±0.86 mm.

The presence of robots not only in industrial settings but also in operating rooms is increasing with advancements in technology. They are not only huge machines carrying out tedious and arduous tasks while being caged behind fences, but also work in collaboration, close proximity of, and in cooperation with humans. Image guided minimally invasive approaches to accurately navigate a robot to precise anatomical locations can be followed only once the robot is registered to the medical imaging system. In this paper, we present an approach to register a robotic system to a medical imaging system without the use of any tracking device or continuous X-ray imaging. The system consists of a lightweight robot equipped with internal torque sensors. The registration is realized by a series of landmark transformations. The robot is manually moved and guided to distinct landmarks whose positions are known in the coordinate system of the imaging system. By driving the robot to these distinct landmarks, their position relative to the coordinate system of the robotic system is also known. By comparing the respective positions of the landmarks in the two coordinate systems, both systems are registered. With 8 screws on the operating table used as landmarks for this registration, we obtain a median error of [6.9, 20.6, 18.6] mm and a standard deviation of [2.9, 2.6, 2.2] mm when compared to an X-ray based ground truth.

Magnetic Resonance Imaging (MRI) has potential to be a superior medical imaging modality for guiding and monitoring prostatic interventions. The strong magnetic field prevents the use of conventional mechatronics and the confined physical space makes it extremely challenging to access the patient. We have designed a robotic assistant system that overcomes these difficulties and promises safe and reliable intra-prostatic needle placement inside closed high-field MRI scanners. The robot performs needle insertion under real-time 3T MR image guidance; workspace requirements, MR compatibility, and workflow have been evaluated on phantoms. The paper explains the robot mechanism and controller design and presents results of preliminary evaluation of the system.

Prostate cancer biopsy is a routine medical procedure, yet conventional techniques suffer from low sensitivity attributed to suboptimal image guidance and needle placement. Targeting small lesions and foci (5 mm in diameter) is particularly prone to errors. We developed an integrated system to perform robot-assisted transperineal needle insertions into the prostate, under Magnetic Resonance Imaging (MRI) guidance. The system provides arbitrary needle trajectories and allows for simultaneous surveillance and correction of the needle path, based on intra-operative MRI. System functionality and data transfer and processing tests were conducted. Five lesions embedded in the gel phantom were targeted successfully, while communication delays (due to higher image frame rates) had no adverse affect on robot-software communication. The system was sufficiently resistant to high network loads and performed with an acceptable transfer rate.