Transcranial magnetic stimulation (TMS) is a noninvasive brain stimulation technique that uses magnetic pulses to safely modulate neural activity in specific brain regions, with particular use in the treatment of major depressive disorder. However, effective treatment protocols require multiple sessions over time and accurate targeting. Current neuronavigation systems can improve pulse delivery compared to standard scalp and measuring tape methods, but are costly and have time-intensive setups for recurring sessions. We present an augmented reality neuronavigation system (AR-NS) that overcomes these limitations. The AR-NS is functionally similar to other neuronavigation systems but can operate entirely within a head-mounted display. Unlike traditional 2D navigation systems that require mentally fusing screen information with real-world actions, the AR interface allows operators to perform the procedure with reduced cognitive and hand-eye coordination demands. We measured the functional targeting accuracy of the AR-NS and the Localite TMS Navigator, a commercial neuronavigation system, using a TMS phantom we developed embedded with Hall effect sensors at four different sites. We determined the coil placement for each site that maximized the sensor response to a magnetic pulse using co-registered X-ray computed tomography and structured light scanner scans. A MagVenture C-B60 coil was placed and fired at each site 30 times in a randomized order according to each neuronavigation system. We measured the resulting magnetic pulse amplitudes. A non-inferiority test with a 5 mT margin and 97.5% confidence intervals indicated that the AR-NS demonstrated similar functional targeting accuracy as the Localite TMS Navigator across all but one stimulation site, indicating that augmented reality neuronavigation systems may offer more accessible delivery of TMS stimulation without sacrificing functional accuracy compared to current systems.

We investigate how vibrotactile wrist feedback can enhance spatial guidance for handheld tool movement in optical see-through augmented reality (AR). While AR overlays are widely used to support surgical tasks, visual occlusion, lighting conditions, and interface ambiguity can compromise precision and confidence. To address these challenges, we designed a multimodal system combining AR visuals with a custom wrist-worn haptic device delivering directional and state-based cues. A formative study with experienced surgeons and residents identified key tool maneuvers and preferences for reference mappings, guiding our cue design. In a cue identification experiment (N = 21), participants accurately recognized five vibration patterns under visual load, with higher recognition for full-actuator states than spatial direction cues. In a guidance task (N = 27), participants using both AR and haptics achieved significantly higher spatial precision (5.8 mm) and usability (SUS = 88.1) than those using either modality alone, albeit with modest increases in task time. Participants reported that haptic cues provided reassuring confirmation and reduced cognitive effort during alignment. Our results highlight the promise of integrating wrist-based haptics into AR systems for high-precision, visually complex tasks such as surgical guidance. We discuss design implications for multimodal interfaces supporting confident, efficient tool manipulation.

The use of Augmented Reality (AR) devices for surgical guidance has gained increasing traction in the medical field. Traditional registration methods often rely on external fiducial markers to achieve high accuracy and real-time performance. However, these markers introduce cumbersome calibration procedures and can be challenging to deploy in clinical settings. While commercial solutions have attempted real-time markerless tracking using the native RGB cameras of AR devices, their accuracy remains questionable for medical guidance, primarily due to occlusions and significant outliers between the live sensor data and the preoperative target anatomy point cloud derived from MRI or CT scans. In this work, we present a markerless framework that relies only on the depth sensor of AR devices and consists of two modules: a registration module for high-precision, outlier-robust target anatomy localization, and a tracking module for real-time pose estimation. The registration module integrates depth sensor error correction, a human-in-the-loop region filtering technique, and a robust global alignment with curvature-aware feature sampling, followed by local ICP refinement, for markerless alignment of preoperative models with patient a natomy. The tracking module employs a fast and robust registration algorithm that uses the initial pose from the registration module to estimate the target pose in real-time. We comprehensively evaluated the performance of both modules through simulation and real-world measurements. The results indicate that our markerless system achieves superior performance for registration and comparable performance for tracking to industrial solutions. The two-module design makes our system a one-stop solution for surgical procedures where the target anatomy moves or stays static during surgery.

About 20 percent of patients undergoing breast-conserving surgery require reoperation due to cancerous tissue remaining inside the breast. Breast cancer localization systems utilize auditory feedback to convey the distance between a localization probe and a small marker (seed) implanted into the breast tumor prior to surgery. However, no information on the location of the tumor margin is provided. To reduce the reoperation rate by improving the usability and accuracy of the surgical task, we developed an auditory display using shape sonification to assist with tumor margin localization. Accuracy and usability of the interactive shape sonification were determined on models of the female breast in three user studies with both breast surgeons and non-clinical participants. The comparative studies showed a significant increase in usability (p<0.05) and localization accuracy (p<0.001) of the shape sonification over the auditory feedback currently used in surgery.

This paper investigates the accuracy of Augmented Reality (AR) technologies, particularly commercially available optical see-through displays, in depicting virtual content inside the human body for surgical planning. Their inherent limitations result in inaccuracies in perceived object positioning. We examine how occlusion, specifically with opaque surfaces, affects perceived depth of virtual objects at arm's length working distances. A custom apparatus with a half-silvered mirror was developed, providing accurate depth cues excluding occlusion, differing from commercial displays. We carried out a study, contrasting our apparatus with a HoloLens 2, involving a depth estimation task under varied surface complexities and illuminations. In addition, we explored the effects of creating a virtual "hole" in the surface. Subjects' depth estimation accuracy and confidence were a ssessed. Results showed more depth estimation variation with HoloLens and significant depth error beneath complex occluding surfaces. However, creating a virtual hole significantly reduced depth errors and increased subjects' confidence, irrespective of accuracy enhancement. These findings have important implications for the design and use of mixed-reality technologies in surgical applications, and industrial applications such as using virtual content to guide maintenance or repair of components hidden beneath the opaque outer surface of equipment. A free copy of this paper and all supplemental materials are available at https://bit.ly/3YbkwjU.

Medical augmented reality (AR) applications where virtual renderings are aligned with the real world allow to visualize internal anatomy of the patient to a medical caregiver wearing an AR headset. Accurate alignment of virtual and real content is important for applications where the virtual rendering is used to guide the medical procedure such as a surgery. Compared to 2D AR applications, where the alignment accuracy can be directly measured on the 2D screen, 3D medical AR applications require alignment measurements using phantoms and external tracking systems. In this paper we present an approach for landmark-based alignment, validation and accuracy measurement of a 3D AR overlay of medical images on the real-world subject. This is done by performing an initial MRI of a subject’s head, an AR alignment task of the virtual rendering of the head MRI data to the subject’s real-world head using virtual fiducials, and a second MRI scan to test the accuracy of the AR alignment task. We have performed these 3D medical AR alignment measurements on seven volunteers using a MagicLeap AR head-mounted display. Across all seven volunteers we measured an alignment accuracy of $4.7 \pm 2.6$ mm. These results suggest that such an AR application can be a valuable tool for guiding non-invasive transcranial magnetic brain stimulation treatment. The presented MRI-based accuracy validation will furthermore be an important versatile tool to establish the safety of medical AR techniques.