Deep Learning-based 2D/3D registration enables fast, robust, and accurate X-ray to CT image fusion when large annotated paired datasets are available for training. However, the need for paired CT volume and X-ray images with ground truth registration limits the applicability in interventional scenarios. An alternative is to use simulated X-ray projections from CT volumes, thus removing the need for paired annotated datasets. Deep Neural Networks trained exclusively on simulated X-ray projections can perform significantly worse on real X-ray images due to the domain gap. We propose a self-supervised 2D/3D registration framework combining simulated training with unsupervised feature and pixel space domain adaptation to overcome the domain gap and eliminate the need for paired annotated datasets. Our framework achieves a registration accuracy of 1.83 ± 1.16 mm with a high success ratio of 90.1% on real X-ray images showing a 23.9% increase in success ratio compared to reference annotation-free algorithms.

Minimally invasive procedures rely on image guidance for navigation at the operation site to avoid large surgical incisions. X-ray images are often used for guidance, but important structures may be not well visible. These structures can be overlaid from pre-operative 3-D images and accurate alignment can be established using 2-D/3-D registration. Registration based on the point-to-plane correspondence model was recently proposed and shown to achieve state-of-the-art performance. However, registration may still fail in challenging cases due to a large portion of outliers. In this paper, we describe a learning-based correspondence weighting scheme to improve the registration performance. By learning an attention model, inlier correspondences get higher attention in the motion estimation while the outlier correspondences are suppressed. Instead of using per-correspondence labels, our objective function allows to train the model directly by minimizing the registration error. We demonstrate a highly increased robustness, e.g. increasing the success rate from 84.9% to 97.0% for spine registration. In contrast to previously proposed learning-based methods, we also achieve a high accuracy of around 0.5mm mean re-projection distance. In addition, our method requires a relatively small amount of training data, is able to learn from simulated data, and generalizes to images with additional structures which are not present during training. Furthermore, a single model can be trained for both, different views and different anatomical structures.

In minimally invasive procedures, the clinician relies on image guidance to observe and navigate the operation site. In order to show structures which are not visible in the live X-ray images, such as vessels or planning annotations, X-ray images can be augmented with pre-operatively acquired images. Accurate image alignment is needed and can be provided by 2-D/3-D registration. In this paper, a multi-view registration method based on the point-to-plane correspondence model is proposed. The correspondence model is extended to be independent of the used camera coordinates and different multi-view registration schemes are introduced and compared. Evaluation is performed for a wide range of clinically relevant registration scenarios. We show for different applications that registration using correspondences from both views simultaneously provides accurate and robust registration, while the performance of the other schemes varies considerably. Our method also outperforms the state-of-the-art method for cerebral angiography registration, achieving a capture range of 18 mm and an accuracy of 0.22±0.07 mm. Furthermore, investigations on the minimum angle between the views are performed in order to provide accurate and robust registration, while minimizing the obstruction to the clinical workflow. We show that small angles around 30° are sufficient to provide reliable registration results.

In image-guided interventional procedures, live 2-D X-ray images can be augmented with preoperative 3-D computed tomography or MRI images to provide planning landmarks and enhanced spatial perception. An accurate alignment between the 3-D and 2-D images is a prerequisite for fusion applications. This paper presents a dynamic rigid 2-D/3-D registration framework, which measures the local 3-D-to-2-D misalignment and efficiently constrains the update of both planar and non-planar 3-D rigid transformations using a novel point-to-plane correspondence model. In the simulation evaluation, the proposed method achieved a mean 3-D accuracy of 0.07 mm for the head phantom and 0.05 mm for the thorax phantom using single-view X-ray images. In the evaluation on dynamic motion compensation, our method significantly increases the accuracy comparing with the baseline method. The proposed method is also evaluated on a publicly-available clinical angiogram data set with "gold-standard" registrations. The proposed method achieved a mean 3-D accuracy below 0.8 mm and a mean 2-D accuracy below 0.3 mm using single-view X-ray images. It outperformed the state-of-the-art methods in both accuracy and robustness in single-view registration. The proposed method is intuitive, generic, and suitable for both initial and dynamic registration scenarios.

Geometry and physics of absorption imaging impose certain constraints on X-ray projections. Recently, the Epipolar Consistency Conditions (ECC) have been introduced and applied to motion correction in flat-detector computed tomography (CT). They are based on redundant information in transmission images along epipolar lines. Unlike other consistency conditions for CT scans, they act directly on an arbitrary pair of X-ray images. This paper proposes an application of ECC to 3D patient tracking in interventional radiology. We evaluate the proposed method against 2D-3D registration with a previously acquired CT. Our experiments on synthetic data based on a patient CT and phantom data from an interventional C-arm demonstrate that our method is able to compensate online for rotations of up to ±10° and translations of ±25 mm between consecutive frames. We successfully track rotations of as much as 45° over 45 images. The outstanding property of the approach is that no 3D scan is required for tracking a 3D object in space. We show, that small rotations of about 3° in space and translations of about 50 mm can be tracked based on just two reference X-ray images. Since the proposed approach works directly on X-ray images, it exceeds regular 2D-3D registration with a CT in an order of magnitude in computational speed. We conclude that ECC are a simple and effective new tool for pre-aligment and online patient tracking for fluoroscopic sequences.

This paper presents the derivation of the Epipolar Consistency Conditions (ECC) between two X-ray images from the Beer-Lambert law of X-ray attenuation and the Epipolar Geometry of two pinhole cameras, using Grangeat's theorem. We motivate the use of Oriented Projective Geometry to express redundant line integrals in projection images and define a consistency metric, which can be used, for instance, to estimate patient motion directly from a set of X-ray images. We describe in detail the mathematical tools to implement an algorithm to compute the Epipolar Consistency Metric and investigate its properties with detailed random studies on both artificial and real FD-CT data. A set of six reference projections of the CT scan of a fish were used to evaluate accuracy and precision of compensating for random disturbances of the ground truth projection matrix using an optimization of the consistency metric. In addition, we use three X-ray images of a pumpkin to prove applicability to real data. We conclude, that the metric might have potential in applications related to the estimation of projection geometry. By expression of redundancy between two arbitrary projection views, we in fact support any device or acquisition trajectory which uses a cone-beam geometry. We discuss certain geometric situations, where the ECC provide the ability to correct 3D motion, without the need for 3D reconstruction.

In interventional radiology, preoperative 3-D volumes can be fused with intra-operative 2-D fluoroscopic images. Since the accuracy is crucial to the clinical usability of image fusion, patient motion resulting in misalignments has to be corrected during the procedure. In this paper, a novel gradient based differential approach is proposed to estimate the 3-D rigid motion from the 2-D tracking of contour points. The mathematical relationship between the 3-D differential motion and the 2-D motion is derived using the 3-D gradient, based on which a tracking-based motion compensation pipeline is introduced. Given the initial registration, the contour points are extracted and tracked along 2-D frames. The 3-D rigid motion is estimated using the iteratively re-weighted least square minimization to enhance the robustness. Our novel approach is evaluated on 10 datasets consisting of 1010 monoplane fluoroscopic images of a thorax phantom with 3-D rigid motion. Over all datasets, the maximum structure shift in the 2-D projection caused by the 3-D motion varies from 17.3 mm to 33.2 mm. Our approach reduces the 2-D structure shift to the range of 1.93 mm to 6.52 mm. For the most challenging longitudinal off-plane rotation, our approach achieves an average coverage of 79.9% regarding to the ground truth.

Visualizing 2D and 3D anatomical information simultaneously within an X-ray image is challenging. Multiple monitors are required in the operating room to enable visualization of anatomical data by the surgeon. Consequently, this results in an interruption during the operation for proper assessment of information. In this paper, we introduce an interactive visualization of the 3D data from new perspectives including visualizing a virtual mirror from the same viewpoint as the X-ray source. The main contribution is our development of a complete angiographic visualization system that displays simultaneous 2D X-ray and 3D anatomical information in a common monitor for the surgeon in the operating room. No previous works have conceived the integration of the virtual mirror into the projection geometry of a C-arm fluoroscope. In total, 24 participants were asked to assess the benefits of the angiographic virtual mirror with different colour-depth encodings. The results of our feasibility study show a clear improvement when deciphering the true positions of aneurysms in X-ray. Lastly, color depth encoding improves correspondence between the 3D vasculature displayed in the virtual mirror to their projection images in X-ray.