Augmented reality (AR) devices with head mounted displays (HMDs) facilitate direct superimposition of 3D preoperative imaging data onto the patient during surgery. To use an HMD-AR device as a stand-alone surgical navigation system, the device should be able to locate the patient and surgical instruments, align preoperative imaging data with the patient, and visualize navigation data during surgery. Whereas some of the technologies required for this are known, integration in such devices is cumbersome and requires specific knowledge and expertise, hampering scientific progress in this field. This work therefore aims to present and evaluate an integrated HMD-based AR surgical navigation framework that is adaptable to diverse surgical applications. The framework tracks 2D patterns as reference markers attached to the patient and surgical instruments. It allows for the calibration of surgical tools using pivot and reference-based calibration techniques. It enables image-to-patient registration using point-based matching and manual positioning. The integrated functionalities of the framework are evaluated on two HMD devices, the HoloLens 2 and Magic Leap 2, with two surgical use cases being evaluated in a phantom setup: AR-guided needle insertion and rib fracture localization. The framework was able to achieve a mean tooltip calibration accuracy of 1 mm, a registration accuracy of 3 mm, and a targeting accuracy below 5 mm on the two surgical use cases. The framework presents an easy-to-use configurable tool for HMD-based AR surgical navigation, which can be extended and adapted to many surgical applications.

This study addresses the challenges of confounding effects and interpretability in artificial-intelligence-based medical image analysis. Whereas existing literature often resolves confounding by removing confounder-related information from latent representations, this strategy risks affecting image reconstruction quality in generative models, thus limiting their applicability in feature visualization. To tackle this, we propose a different strategy that retains confounder-related information in latent representations while finding an alternative confounder-free representation of the image data. Our approach views the latent space of an autoencoder as a vector space, where imaging-related variables, such as the learning target (t) and confounder (c), have a vector capturing their variability. The confounding problem is addressed by searching a confounder-free vector which is orthogonal to the confounder-related vector but maximally collinear to the target-related vector. To achieve this, we introduce a novel correlation-based loss that not only performs vector searching in the latent space, but also encourages the encoder to generate latent representations linearly correlated with the variables. Subsequently, we interpret the confounder-free representation by sampling and reconstructing images along the confounder-free vector. The efficacy and flexibility of our proposed method are demonstrated across three applications, accommodating multiple confounders and utilizing diverse image modalities. Results affirm the method’s effectiveness in reducing confounder influences, preventing wrong or misleading associations, and offering a unique visual interpretation for in-depth investigations by clinical and epidemiological researchers. The code is released in the following GitLab repository: https://gitlab.com/radiology/compopbio/ai_based_association_analysis . • Propose a novel deep-learning method for the correction of confounders, which is crucial to avoid misleading findings in medical imaging association analysis. • Introduce geometric insights of confounders into the latent space of an autoencoder, and subsequently propose a correlation-based loss as a novel solution for confounder correction via vector orthogonalization. • Enable semantic feature interpretation in confounder-free AI prediction models, and easily handle multiple categorized or continuous confounders. • Performance evaluation of our pipeline on three applications, accommodating multiple confounders and utilizing diverse image modalities.

Augmented reality (AR) has shown potential in computer-aided surgery. It allows for the visualization of hidden anatomical structures as well as assists in navigating and locating surgical instruments at the surgical site. Various modalities (devices and/or visualizations) have been used in the literature, but few studies investigated the adequacy/superiority of one modality over the other. For instance, the use of optical see-through (OST) HMDs has not always been scientifically justified. Our goal is to compare various visualization modalities for catheter insertion in external ventricular drain and ventricular shunt procedures. We investigate two AR approaches: (1) 2D approaches consisting of a smartphone and a 2D window visualized through an OST (Microsoft HoloLens 2), and (2) 3D approaches consisting of a fully aligned patient model and a model that is adjacent to the patient and is rotationally aligned using an OST. 32 participants joined this study. For each visualization approach, participants were asked to perform five insertions after which they filled NASA-TLX and SUS forms. Moreover, the position and orientation of the needle with respect to the planning during the insertion task were collected. The results show that participants achieved a better insertion performance significantly under 3D visualizations, and the NASA-TLX and SUS forms reflected the preference of participants for these approaches compared to 2D approaches.

Instrument alignment is a common task in various surgical interventions using navigation. The goal of the task is to position and orient an instrument as it has been planned preoperatively. To this end, surgeons rely on patient-specific data visualized on screens alongside preplanned trajectories. The purpose of this manuscript is to investigate the effect of instrument visualization/non visualization on alignment tasks, and to compare it with virtual extensions approach which augments the realistic representation of the instrument with simple 3D objects. 18 volunteers performed six alignment tasks under each of the following conditions: no visualization on the instrument; realistic visualization of the instrument; realistic visualization extended with virtual elements (Virtual extensions). The first condition represents an egocentric-based alignment while the two other conditions additionally make use of exocentric depth estimation to perform the alignment. The device used was a see-through device (Microsoft HoloLens 2). The positions of the head and the instrument were acquired during the experiment. Additionally, the users were asked to fill NASA-TLX and SUS forms for each condition. The results show that instrument visualization is essential for a good alignment using see-through devices. Moreover, virtual extensions helped achieve the best performance compared to the other conditions with medians of 2 mm and 2° positional and angular error respectively. Furthermore, the virtual extensions decreased the average head velocity while similarly reducing the frustration levels. Therefore, making use of virtual extensions could facilitate alignment tasks in augmented and virtual reality (AR/VR) environments, specifically in AR navigated surgical procedures when using optical see-through devices.

In this paper, we present a novel approach to automatic 3D facial landmarking using 2D Gabor wavelets. Our algorithm considers the face to be a surface and uses map projections to derive 2D features from raw data. Extracted features include texture, relief map, and transformations thereof. We extend an established 2D landmarking method for simultaneous evaluation of these data. The method is validated by performing landmarking experiments on two data sets using 21 landmarks and compared with an active shape model implementation. On average, landmarking error for our method was 1.9 mm, whereas the active shape model resulted in an average landmarking error of 2.3 mm. A second study investigating facial shape heritability in related individuals concludes that automatic landmarking is on par with manual landmarking for some landmarks. Our algorithm can be trained in 30 min to automatically landmark 3D facial data sets of any size, and allows for fast and robust landmarking of 3D faces.