Mobile C-arm is an essential tool in everyday trauma and orthopedics surgery. Minimally invasive solutions, based on X-ray imaging and coregistered external navigation created a lot of interest within the surgical community and started to replace the traditional open surgery for many procedures. These solutions usually increase the accuracy and reduce the trauma. In general, they introduce new hardware into the OR and add the line of sight constraints imposed by optical tracking systems. They thus impose radical changes to the surgical setup and overall procedure. We augment a commonly used mobile C-arm with a standard video camera and a double mirror system allowing real-time fusion of optical and X-ray images. The video camera is mounted such that its optical center virtually coincides with the C-arm's X-ray source. After a one-time calibration routine, the acquired X-ray and optical images are coregistered. This paper describes the design of such a system, quantifies its technical accuracy, and provides a qualitative proof of its efficiency through cadaver studies conducted by trauma surgeons. In particular, it studies the relevance of this system for surgical navigation within pedicle screw placement, vertebroplasty, and intramedullary nail locking procedures. The image overlay provides an intuitive interface for surgical guidance with an accuracy of < 1 mm, ideally with the use of only one single X-ray image. The new system is smoothly integrated into the clinical application with no additional hardware especially for down-the-beam instrument guidance based on the anteroposterior oblique view, where the instrument axis is aligned with the X-ray source. Throughout all experiments, the camera augmented mobile C-arm system proved to be an intuitive and robust guidance solution for selected clinical routines.

Port position in minimally invasive surgeries is chosen to minimize the lesion of tissue and maximize the movability for endoscopic instruments. In this study, we present an evaluation of the potential of a 3D contextual in-situ visualization of the anatomic target region to help surgeons for three different surgical procedures decide where best to create ports and incisions to enable the insertion of a specific set of instruments.

Ultrasound (US) is a medical imaging modality which is extremely difficult to learn as it is user-dependent, has low image quality and requires much knowledge about US physics and human anatomy. For training US we propose an Augmented Reality (AR) ultrasound simulator where the US slice is simulated from a CT volume. The location of the US slice inside the body is visualized using contextual in-situ techniques. We also propose advanced methods how to use an AR simulator for training.

This work presents the development of an augmented reality magnification system, termed virtual loupe, implemented in a head-mounted display for surgical applications. The system provides a magnified view with a novel control based on tracked sight orientation. The system was evaluated by measuring the completion time of a suturing task performed by surgeons. The magnifying approach implemented proved to be useful by providing global context of the operating field. The sight-based activation was widely accepted by surgeons as a useful functionality to control viewing modalities.

Medical augmented reality (AR) has been widely discussed within the medical imaging as well as computer aided surgery communities. Different systems for exemplary medical applications have been proposed. Some of them produced promising results. One major issue still hindering AR technology to be regularly used in medical applications is the interaction between physician and the superimposed 3-D virtual data. Classical interaction paradigms, for instance with keyboard and mouse, to interact with visualized medical 3-D imaging data are not adequate for an AR environment. This paper introduces the concept of a tangible/controllable Virtual Mirror for medical AR applications. This concept intuitively augments the direct view of the surgeon with all desired views on volumetric medical imaging data registered with the operation site without moving around the operating table or displacing the patient. We selected two medical procedures to demonstrate and evaluate the potentials of the Virtual Mirror for the surgical workflow. Results confirm the intuitiveness of this new paradigm and its perceptive advantages for AR-based computer aided interventions.

Augmented reality (AR) for preoperative diagnostics and planning, intra operative navigation and postoperative follow-up examination has been a topic of intensive research over the last two decades. However, clinical studies showing AR technology integrated into the real clinical environment and workflow are still rare. The incorporation of an AR system as a standard tool into the real clinical workflow has not been presented so far. This paper reports on the strategies and intermediate results of the ARAV - augmented reality aided vertebroplasty project that has been initiated to make an AR system based on a stereo video see-through head mounted display that is permanently available in the operating room (OR).

In recent years, an increasing number of liver tumor indications were treated by minimally invasive laparoscopic resection. Besides the restricted view, two major intraoperative issues in laparoscopic liver resection are the optimal planning of ports as well as the enhanced visualization of (hidden) vessels, which supply the tumorous liver segment and thus need to be divided (e.g., clipped) prior to the resection. We propose an intuitive and precise method to plan the placement of ports. Preoperatively, self-adhesive fiducials are affixed to the patient's skin and a computed tomography (CT) data set is acquired while contrasting the liver vessels. Immediately prior to the intervention, the laparoscope is moved around these fiducials, which are automatically reconstructed to register the patient to its preoperative imaging data set. This enables the simulation of a camera flight through the patient's interior along the laparoscope's or instruments' axes to easily validate potential ports. Intraoperatively, surgeons need to update their surgical planning based on actual patient data after organ deformations mainly caused by application of carbon dioxide pneumoperitoneum. Therefore, preoperative imaging data can hardly be used. Instead, we propose to use an optically tracked mobile C-arm providing cone-beam CT imaging capability intraoperatively. After patient positioning, port placement, and carbon dioxide insufflation, the liver vessels are contrasted and a 3-D volume is reconstructed during patient exhalation. Without any further need for patient registration, the reconstructed volume can be directly augmented on the live laparoscope video, since prior calibration enables both the volume and the laparoscope to be positioned and oriented in the tracking coordinate frame. The augmentation provides the surgeon with advanced visual aid for the localization of veins, arteries, and bile ducts to be divided or sealed.

In this paper we present the evaluation of a virtual mirror used as a navigational tool within a medical augmented reality (AR) system for laparoscopy. 12 surgeons of our clinical partner participated in an experiment to evaluate whether laparoscope augmentation extended by a virtual mirror is useful for improved perception of complex structures. Such complex structures are encountered for instance in laparoscopic resection of tumor affected liver tissue. The blood vessels supplying the tumor have to be cut and closed before tumorous tissue can be removed. A laparoscopic camera and an optical tracking system allow for the visualization of visualized medical volumetric data registered with the real anatomy. Previously injected contrast agent provides an accentuation of blood vessels within the visualization. For evaluating the suitability of a virtual mirror to support the mentioned procedure, we designed a phantom consisting of wooden branches simulating the structure of blood vessel trees. Quantitative results of the experiment show the advantage of a mirror in certain cases, when blood vessels cannot be directly seen from the camera point of view due to self-occlusion of the structure. Results of a questionnaire filled out by the surgeons after the experiments confirm the acceptance of AR technology for particular medical procedures.

The need to improve medical diagnosis and reduce invasive surgery is dependent upon seeing into a living human system. The use of diverse types of medical imaging and endoscopic instruments has provided significant breakthroughs, but not without limiting the surgeon's natural, intuitive and direct 3D perception into the human body. This paper presents a method for the use of augmented reality (AR) for the convergence of improved perception of 3D medical imaging data (mimesis) in context to the patient's own anatomy (in-situ) incorporating the physician's intuitive multi- sensory interaction and integrating direct manipulation with endoscopic instruments. Transparency of the video images recorded by the color cameras of a video see-through, stereoscopic head- mounted-display (HMD) is adjusted according to the position and line of sight of the observer, the shape of the patient's skin and the location of the instrument. The modified video image of the real scene is then blended with the previously rendered virtual anatomy. The effectiveness has been demonstrated in a series of experiments at the Chirurgische Klinik in Munich, Germany with cadaver and in-vivo studies. The results can be applied for designing medical AR training and educational applications.