Screen-based Augmented Reality (AR) systems can be built as a window into the real world as often done in mobile AR applications or using the Magic Mirror metaphor, where users can see themselves with augmented graphics on a large display. The term Magic Mirror implies that the display shows the users enantiomorph, i.e. the mirror image, such that the system mimics a real-world physical mirror. However, the question arises whether one should design a traditional mirror, or instead display the true mirror image by means of a non-reversing mirror? We discuss the perceptual differences between these two mirror visualization concepts and present a first comparative study in the context of Magic Mirror anatomy teaching.

This paper presents the first design of a mirror based RGBD X-ray imaging system and includes an evaluation study of the depth errors induced by the mirror when used in combination with an infrared pattern-emission RGBD camera. Our evaluation consisted of three experiments. The first demonstrated almost no difference in depth measurements of the camera with and without the use of the mirror. The final two experiments demonstrated that there were no relative and location-specific errors induced by the mirror showing the feasibility of the RGBDX-ray imaging system. Lastly, we showcase the potential of the RGBDX-ray system towards a visualization application in which an X-ray image is fused to the 3D reconstruction of the surgical scene via the RGBD camera, using automatic C-arm pose estimation.

Fusing intraoperative X-ray data with real-time video in a common reference frame is not trivial since both modalities have to be acquired from the same viewpoint. The goal of this work is to design a flexible system comprising two RGBD sensors that can be attached to any mobile C-arm, with the objective of synthesizing projective color images from the X-ray source viewpoint. To achieve this, we calibrate the RGBD sensors to the X-ray source with a 3D calibration object. Then, we synthesize the projective color image from the X-ray viewpoint by applying a volumetric-based rendering method. Finally, the X-ray image is overlaid on the projective image without any further registration, offering a multimodal visualization of X-ray and color images. In this paper we present the different steps of development (i.e. hardware setup, calibration and rendering algorithm) and discuss clinical applications for the new video augmented C-arm. By placing X-ray markers on a hand patient and a spine model, we show that the overlay accuracy between the X-ray image and the synthetized image is in average 1.7 mm.

C-arm fluoroscopes are frequently used during surgeries for intraoperative guidance. Unfortunately, due to X-ray emission and scattering, increased radiation exposure occurs in the operating theatre. The objective of this work is to sensitize the surgeon to their radiation exposure, enable them to check on their exposure over time, and to help them choose their best position related to the C-arm gantry during surgery. First, we aim at simulating the amount of radiation that reaches the surgeon using the Geant4 software, a toolkit developed by CERN. Using a flexible setup in which two RGB-D cameras are mounted to the mobile C-arm, the scene is captured and modeled respectively. After the simulation of particles with specific energies, the dose at the surgeon's position, determined by the depth cameras, can be measured. The validation was performed by comparing the simulation results to both theoretical values from the C-arms user manual and real measurements made with a QUART didoSVM dosimeter. The average error was 16.46% and 16.39%, respectively. The proposed flexible setup and high simulation precision without a calibration with measured dosimeter values, has great potential to be directly used and integrated intraoperatively for dose measurement.

To help the computing device always understand the spacial relationship between the user's gesture and the ambient objects, a methodology is proposed to find the user's virtual eye center in the wearable camera coordinate system and then calculate accurately where a user is pointing at to perform the natural interaction. First, the wearable RGB-D sensor is affixed around the user forehead. A tool-free calibration is done by having the user move their fingers along their lines of sight from his eye center to the random selected targets. The fingertips are detected in the depth camera and then the interaction of these lines of sight is calculated. Then we present how to find where the user is pointing at in different scenarios with a depth map, a detected object and a controlled virtual element. To validate our methods, we perform a point-to-screen experiment. Results demonstrate that when a user is interacting with a display up to 1.5 meters away, our natural gesture interface has an average error of 2.1cm. In conclusion, the presented technique is a viable option for a reliable user interaction.

Calibration and registration are the first steps for augmented reality and mixed reality applications. In the medical field, the calibration between an RGB-D camera and a mobile C-arm fluoroscope is a new topic which introduces challenges. In this paper, we propose a precise 3D/2D calibration method to achieve a video augmented fluoroscope. With the design of a suitable calibration phantom for RGB-D/C-arm calibration, we calculate the projection matrix from the depth camera coordinates to the X-ray image. Through a comparison experiment by combining different steps leading to the calibration, we evaluate the effect of every step of our calibration process. Results demonstrated that we obtain a calibration RMS error of 0.54±1.40 mm which is promising for surgical applications. We conclude this paper by showcasing two clinical applications. One is a markerless registration application, the other is an RGB-D camera augmented mobile C-arm visualization.

We present a prototype of a system in development for pre-operative planning. The proposed NeuroTable uses a combination of traditional rendering and novel visualization techniques rendered to facilitate real-time collaboration between neurosurgeons during intervention planning. A set of multimodal 2D and 3D renderings convey the relation between the region of interest and the surrounding anatomical structures. A haptic device is used for interaction with the NeuroTable to facilitate immersive control over the 3D cursor and navigation modes for the neurosurgeons during their discourse of planning. A pilot experimental study was conducted to assess the performance of users in targeting points within the preoperative 3D scan. Then, two clinicians participated in the evaluation of the table in discussing and planning a case. Results indicate that the NeuroTable facilitated the discourse and we discuss the results of the speed and accuracy for the specification of entry and target points.

The interlocking of intramedullary nails is a technically demanding procedure which involves a considerable amount of X-ray acquisitions; one study lists as many as 48 to successfully complete the procedure and fix screws into 4-6 mm distal holes of the nail. We propose to design an augmented radiolucent drill to assist surgeons in completing the distal locking procedure without any additional X-ray acquisitions. Using an augmented reality fluoroscope that coregisters optical and X-ray images, we exploit solely the optical images to detect the augmented radiolucent drill and estimate its tip position in real-time. Consequently, the surgeons will be able to maintain the down the beam positioning required to drill the screws into the nail holes successfully. To evaluate the accuracy of the proposed augmented drill, we perform a preclinical study involving six surgeons and ask them to perform distal locking on dry bone phantoms. Surgeons completed distal locking 98.3% of the time using only a single X-ray image with an average navigation time of 1.4 ± 0.9 min per hole.

We present a unique simulator-based methodology for assessing both technical and nontechnical (cognitive) skills for surgical trainees while immersed in a complete medical simulation environment. Further, we have included two crisis scenarios which allow for the evaluation of the effect of cognitive strategy selection on the low-level surgical skills. Training these mixed-mode scenarios can thereby be evaluated on our platform, allowing for improved assessment and a stronger foundation for credentialing, with the potential to reduce the occurrence of adverse events in the operating room. Scientific evaluation and validation of our work is conducted together with 19 junior surgeons in order to achieve the following goals: 1) to provide a qualitative measure of usability, 2) to assess vertebroplasty technical performance of the surgeon, and 3) to explore the relationship between mental workload and surgical performance during crisis. Our results indicate that: 1) the surgeons scored the face validity of our modeled simulation environment very highly ( 4.68 ±0.48, using a 5-point Likert scale), 2) surgeon training enabled completion of tasks more quickly, and 3) the introduction of crisis scenarios negatively affected the surgeons' objective performance. Taken together, our results underscore the need to develop realistic simulation environments that prepare young residents to respond to emergent events in the operating room.

Depth cues are an essential part of navigation and device positioning tasks during clinical interventions. Yet, many minimally-invasive procedures, such as catheterizations, are usually performed under X-ray guidance only depicting a 2D projection of the anatomy, which lacks depth information. Previous attempts to integrate pre-operative 3D data of the patient by registering these to intra-operative data have led to virtual 3D renderings independent of the original X-ray appearance and planar 2D color overlays (e.g. roadmaps). A major drawback associated to these solutions is the trade-off between X-ray attenuation values that is completely neglected during 3D renderings, and depth perception not being incorporated into the 2D roadmaps. This paper presents a novel technique for enhancing depth perception of interventional X-ray images preserving the original attenuation appearance. Starting from patient-specific pre-operative 3D data, our method relies on GPU ray casting to compute a colored depth map, which assigns a predefined color to the first incidence of gradient magnitude value above a predefined threshold along the ray. The colored depth map values are carefully integrated into the X-Ray image while maintaining its original grey-scale intensities. The presented method was tested and analysed for three relevant clinical scenarios covering different anatomical aspects and targeting different levels of interventional expertise. Results demonstrate that improving depth perception of X-ray images has the potential to lead to safer and more efficient clinical interventions.

We present a medical Augmented Reality (AR) edutainment system for bone anatomy learning. This learning environment, called AR bone puzzle, is a metaphor for bone anatomy learning with AR visualization and intuitive interaction. AR bone puzzle uses its user's body as a puzzle frame and computer generated virtual bones as puzzle pieces. Users learn bone anatomy by assembling the virtual bone pieces on their body. Key features of this system are 3D AR visualization and intuitive gesture based user interaction.

Education of anatomy is a challenging but crucial element in educating medical professionals, but also for general education of pupils. Our research group has previously developed a prototype of an Augmented Reality (AR) magic mirror which allows intuitive visualization of realistic anatomical information on the user. However, the current overlay is imprecise as the magic mirror depends on the skeleton output from Kinect. These imprecisions affect the quality of education and learning. Hence, together with clinicians we have defined bone landmarks which users can touch easily on their body while standing in front of the sensor. We demonstrate that these landmarks allow the proper deformation of medical data within the magic mirror and onto the human body, resulting in a more precise augmentation.

In this paper, we present a fusion algorithm supplemented with appropriate visualization by selecting relevant information from different modalities in mixed and augmented reality (AR). This encompasses a learning based method upon relevance of information, defined by an expert, which ultimately enables confident interventional decisions based on mixed reality (MR) images. The performance of our developed fusion and tailored visualization techniques was evaluated by employing X-ray/optical images during surgery and validated qualitatively using a 5-point Likert scale. Our observations indicated that the proposed technique provided semantic contextual information about underlying pixels and in general was preferred over the traditional pixel-wise linear alpha-blending method.

Visualizing 2D and 3D anatomical information simultaneously within an X-ray image is challenging. Multiple monitors are required in the operating room to enable visualization of anatomical data by the surgeon. Consequently, this results in an interruption during the operation for proper assessment of information. In this paper, we introduce an interactive visualization of the 3D data from new perspectives including visualizing a virtual mirror from the same viewpoint as the X-ray source. The main contribution is our development of a complete angiographic visualization system that displays simultaneous 2D X-ray and 3D anatomical information in a common monitor for the surgeon in the operating room. No previous works have conceived the integration of the virtual mirror into the projection geometry of a C-arm fluoroscope. In total, 24 participants were asked to assess the benefits of the angiographic virtual mirror with different colour-depth encodings. The results of our feasibility study show a clear improvement when deciphering the true positions of aneurysms in X-ray. Lastly, color depth encoding improves correspondence between the 3D vasculature displayed in the virtual mirror to their projection images in X-ray.

For trauma and orthopedic surgery, maneuvering a mobile C-arm fluoroscope into a desired position to acquire an X-ray is a routine surgical task. The precision and ease of use of the C-arm becomes even more important for advanced interventional imaging techniques such as parallax-free X-ray image stitching. Today's standard mobile C-arms have been modeled with only five degrees of freedom (DOF), which definitely restricts their motions in 3-D Cartesian space. In this paper, we present a method to model both the mobile C-arm and patient's table as an integrated kinematic chain having six DOF without constraining table position. The closed-form solutions for the inverse kinematics problem are derived in order to obtain the required values for all C-arm joint and table movements to position the fluoroscope at a desired pose. The modeling method and the closed-form solutions can be applied to general isocentric or nonisocentric mobile C-arms. By achieving this we develop an efficient and intuitive inverse kinematics-based method for parallax-free panoramic X-ray imaging. In addition, we implement a 6-DOF C-arm system from a low-cost mobile fluoroscope to optimally acquire X-ray images based solely on the computation of the required movement for each joint by solving the inverse kinematics on a continuous basis. Through simulation experimentation, we demonstrate that the 6-DOF C-arm model has a larger working space than the 5-DOF model. C-arm repositioning experiments show the practicality and accuracy of our 6-DOF C-arm system. We also evaluate the novel parallax-free X-ray stitching method on phantom and dry bones. Using five trials, results show that parallax-free panoramas generated by our method are of high visual quality and within clinical tolerances for accurate evaluation of long bone geometry (i.e., image and metric measurement errors are less than 1% compared to ground-truth).