Transcranial magnetic stimulation (TMS) is a noninvasive brain stimulation technique that uses magnetic pulses to safely modulate neural activity in specific brain regions, with particular use in the treatment of major depressive disorder. However, effective treatment protocols require multiple sessions over time and accurate targeting. Current neuronavigation systems can improve pulse delivery compared to standard scalp and measuring tape methods, but are costly and have time-intensive setups for recurring sessions. We present an augmented reality neuronavigation system (AR-NS) that overcomes these limitations. The AR-NS is functionally similar to other neuronavigation systems but can operate entirely within a head-mounted display. Unlike traditional 2D navigation systems that require mentally fusing screen information with real-world actions, the AR interface allows operators to perform the procedure with reduced cognitive and hand-eye coordination demands. We measured the functional targeting accuracy of the AR-NS and the Localite TMS Navigator, a commercial neuronavigation system, using a TMS phantom we developed embedded with Hall effect sensors at four different sites. We determined the coil placement for each site that maximized the sensor response to a magnetic pulse using co-registered X-ray computed tomography and structured light scanner scans. A MagVenture C-B60 coil was placed and fired at each site 30 times in a randomized order according to each neuronavigation system. We measured the resulting magnetic pulse amplitudes. A non-inferiority test with a 5 mT margin and 97.5% confidence intervals indicated that the AR-NS demonstrated similar functional targeting accuracy as the Localite TMS Navigator across all but one stimulation site, indicating that augmented reality neuronavigation systems may offer more accessible delivery of TMS stimulation without sacrificing functional accuracy compared to current systems.

We investigate how vibrotactile wrist feedback can enhance spatial guidance for handheld tool movement in optical see-through augmented reality (AR). While AR overlays are widely used to support surgical tasks, visual occlusion, lighting conditions, and interface ambiguity can compromise precision and confidence. To address these challenges, we designed a multimodal system combining AR visuals with a custom wrist-worn haptic device delivering directional and state-based cues. A formative study with experienced surgeons and residents identified key tool maneuvers and preferences for reference mappings, guiding our cue design. In a cue identification experiment (N = 21), participants accurately recognized five vibration patterns under visual load, with higher recognition for full-actuator states than spatial direction cues. In a guidance task (N = 27), participants using both AR and haptics achieved significantly higher spatial precision (5.8 mm) and usability (SUS = 88.1) than those using either modality alone, albeit with modest increases in task time. Participants reported that haptic cues provided reassuring confirmation and reduced cognitive effort during alignment. Our results highlight the promise of integrating wrist-based haptics into AR systems for high-precision, visually complex tasks such as surgical guidance. We discuss design implications for multimodal interfaces supporting confident, efficient tool manipulation.

The use of Augmented Reality (AR) devices for surgical guidance has gained increasing traction in the medical field. Traditional registration methods often rely on external fiducial markers to achieve high accuracy and real-time performance. However, these markers introduce cumbersome calibration procedures and can be challenging to deploy in clinical settings. While commercial solutions have attempted real-time markerless tracking using the native RGB cameras of AR devices, their accuracy remains questionable for medical guidance, primarily due to occlusions and significant outliers between the live sensor data and the preoperative target anatomy point cloud derived from MRI or CT scans. In this work, we present a markerless framework that relies only on the depth sensor of AR devices and consists of two modules: a registration module for high-precision, outlier-robust target anatomy localization, and a tracking module for real-time pose estimation. The registration module integrates depth sensor error correction, a human-in-the-loop region filtering technique, and a robust global alignment with curvature-aware feature sampling, followed by local ICP refinement, for markerless alignment of preoperative models with patient a natomy. The tracking module employs a fast and robust registration algorithm that uses the initial pose from the registration module to estimate the target pose in real-time. We comprehensively evaluated the performance of both modules through simulation and real-world measurements. The results indicate that our markerless system achieves superior performance for registration and comparable performance for tracking to industrial solutions. The two-module design makes our system a one-stop solution for surgical procedures where the target anatomy moves or stays static during surgery.

Chinese acupuncture practitioners primarily depend on muscle memory and tactile feedback to insert needles and accurately target acupuncture points, as the current workflow lacks imaging modalities and visual aids. Consequently, new practitioners often learn through trial and error, requiring years of experience to become proficient and earn the trust of patients. Medical students face similar challenges in mastering this skill. To address these challenges, we developed an innovative system, MRUCT, that integrates ultrasonic computed tomography (UCT) with mixed reality (MR) technology to visualize acupuncture points in real-time. This system offers offline image registration and real-time guidance during needle insertion, enabling them to accurately position needles based on anatomical structures such as bones, muscles, and auto-generated reference points, with the potential for clinical implementation. In this paper, we outline the non-rigid registration methods used to reconstruct anatomical structures from UCT data, as well as the key design considerations of the MR system. We evaluated two different 3D user interface (3DUI) designs and compared the performance of our system to traditional workflows for both new practitioners and medical students. The results highlight the potential of MR to enhance therapeutic medical practices and demonstrate the effectiveness of the system we developed.

About 20 percent of patients undergoing breast-conserving surgery require reoperation due to cancerous tissue remaining inside the breast. Breast cancer localization systems utilize auditory feedback to convey the distance between a localization probe and a small marker (seed) implanted into the breast tumor prior to surgery. However, no information on the location of the tumor margin is provided. To reduce the reoperation rate by improving the usability and accuracy of the surgical task, we developed an auditory display using shape sonification to assist with tumor margin localization. Accuracy and usability of the interactive shape sonification were determined on models of the female breast in three user studies with both breast surgeons and non-clinical participants. The comparative studies showed a significant increase in usability (p<0.05) and localization accuracy (p<0.001) of the shape sonification over the auditory feedback currently used in surgery.

This paper investigates the accuracy of Augmented Reality (AR) technologies, particularly commercially available optical see-through displays, in depicting virtual content inside the human body for surgical planning. Their inherent limitations result in inaccuracies in perceived object positioning. We examine how occlusion, specifically with opaque surfaces, affects perceived depth of virtual objects at arm's length working distances. A custom apparatus with a half-silvered mirror was developed, providing accurate depth cues excluding occlusion, differing from commercial displays. We carried out a study, contrasting our apparatus with a HoloLens 2, involving a depth estimation task under varied surface complexities and illuminations. In addition, we explored the effects of creating a virtual "hole" in the surface. Subjects' depth estimation accuracy and confidence were a ssessed. Results showed more depth estimation variation with HoloLens and significant depth error beneath complex occluding surfaces. However, creating a virtual hole significantly reduced depth errors and increased subjects' confidence, irrespective of accuracy enhancement. These findings have important implications for the design and use of mixed-reality technologies in surgical applications, and industrial applications such as using virtual content to guide maintenance or repair of components hidden beneath the opaque outer surface of equipment. A free copy of this paper and all supplemental materials are available at https://bit.ly/3YbkwjU.

Medical augmented reality (AR) applications where virtual renderings are aligned with the real world allow to visualize internal anatomy of the patient to a medical caregiver wearing an AR headset. Accurate alignment of virtual and real content is important for applications where the virtual rendering is used to guide the medical procedure such as a surgery. Compared to 2D AR applications, where the alignment accuracy can be directly measured on the 2D screen, 3D medical AR applications require alignment measurements using phantoms and external tracking systems. In this paper we present an approach for landmark-based alignment, validation and accuracy measurement of a 3D AR overlay of medical images on the real-world subject. This is done by performing an initial MRI of a subject’s head, an AR alignment task of the virtual rendering of the head MRI data to the subject’s real-world head using virtual fiducials, and a second MRI scan to test the accuracy of the AR alignment task. We have performed these 3D medical AR alignment measurements on seven volunteers using a MagicLeap AR head-mounted display. Across all seven volunteers we measured an alignment accuracy of $4.7 \pm 2.6$ mm. These results suggest that such an AR application can be a valuable tool for guiding non-invasive transcranial magnetic brain stimulation treatment. The presented MRI-based accuracy validation will furthermore be an important versatile tool to establish the safety of medical AR techniques.

Magnetic resonance imaging (MRI) offers many benefits, including unsurpassed soft-tissue characterization and the ability to combine detection and biopsy into a single procedure. However, limited patient access in the narrow scanner bore requires tedious iterative positioning or use of robotic assistants that isolate the physician from the patient. As an alternative, we present a teleoperation technology for percutaneous procedures to meet the needs of interventional radiologists and overcome challenges imposed by the MR environment. The technology is demonstrated for a 1-DOF needle insertion procedure. The technology uses rolling diaphragms, a clutch, and a cable-capstan drive to propel the needle while relaying forces and motions to the operator. The system demonstrates excellent position tracking (< 0.7° error in the unloaded case) and reliably transmits changes in force. During needle teleoperation, users were able to detect light membrane punctures and differentiate spring stiffnesses nearly as accurately as by hand manipulation.

An active needle is proposed for the development of magnetic resonance imaging (MRI)-guided percutaneous procedures. The needle uses a low-transition-temperature shape memory alloy (LT SMA) wire actuator to produce bending in the distal section of the needle. Actuation is achieved with internal optical heating using laser light transported via optical fibers and side coupled to the LT SMA. A prototype, with a size equivalent to a standard 16-gauge biopsy needle, exhibits significant bending, with a tip deflection of more than 14° in air and 5° in hard tissue. A single-ended optical sensor with a gold-coated tip is developed to measure the curvature independently of temperature. The experimental results in tissue phantoms show that human tissue causes fast heat dissipation from the wire actuator; however, the active needle can compensate for typical targeting errors during prostate biopsy.

This paper presents calibration and user test results of a 3-D tip-force sensing needle with haptic feedback. The needle is a modified MRI-compatible biopsy needle with embedded fiber Bragg grating (FBG) sensors for strain detection. After calibration, the needle is interrogated at 2 kHz, and dynamic forces are displayed remotely with a voice coil actuator. The needle is tested in a single-axis master/slave system, with the voice coil haptic display at the master, and the needle at the slave end. Tissue phantoms with embedded membranes were used to determine the ability of the tip-force sensors to provide real-time haptic feedback as compared to external sensors at the needle base during needle insertion via the master/slave system. Subjects were able to determine the position of the embedded membranes with significantly better accuracy using FBG tip feedback than with base feedback using a commercial force/torque sensor (p = 0.045) or with no added haptic feedback (p = 0.0024).

This study demonstrates real-time scan plane control dependent on three-dimensional needle bending, as measured from magnetic resonance imaging (MRI)-compatible optical strain sensors. A biopsy needle with embedded fiber Bragg grating (FBG) sensors to measure surface strains is used to estimate its full 3-D shape and control the imaging plane of an MR scanner in real-time, based on the needle's estimated profile. The needle and scanner coordinate frames are registered to each other via miniature radio-frequency (RF) tracking coils, and the scan planes autonomously track the needle as it is deflected, keeping its tip in view. A 3-D needle annotation is superimposed over MR-images presented in a 3-D environment with the scanner's frame of reference. Scan planes calculated based on the FBG sensors successfully follow the tip of the needle. Experiments using the FBG sensors and RF coils to track the needle shape and location in real-time had an average root mean square error of 4.2 mm when comparing the estimated shape to the needle profile as seen in high resolution MR images. This positional variance is less than the image artifact caused by the needle in high resolution SPGR (spoiled gradient recalled) images. Optical fiber strain sensors can estimate a needle's profile in real-time and be used for MRI scan plane control to potentially enable faster and more accurate physician response.

We describe an instrumented biopsy needle that provides physicians the capability to sense interaction forces directly at the tip of the needle's inner stylet. The sensors consist of optical fiber Bragg gratings (FBGs), and are unaffected by electromagnetic fields; hence the needle is suitable for MR-guided procedures. In comparison to previous instrumented needles that measure bending strains, the new design has additional sensors and a series of micro-machined holes at the tip. The holes increase strain sensitivity, especially to axial forces, without significantly reducing the stiffness or strength. A comparison of the dynamic forces measured with the new needle and those obtained using a force/torque sensor at the needle base shows that the enhanced tip sensitivity is particularly noticeable when there is significant friction along the needle sleeve.

A side optical actuation method is presented for a slender MR-compatible active needle. The needle includes an active region with a shape memory alloy (SMA) wire actuator, where the wire generates a contraction force when optically heated by a laser delivered though optical fibers, producing needle tip bending. A prototype, with multiple side heating spots, demonstrates twice as fast an initial response compared to fiber tip heating when 0.8 W of optical power is applied. A single-ended optical sensor with a gold reflector is also presented to measure the curvature as a function of optical transmission loss. Preliminary tests with the sensor prototype demonstrate approximately linear response and a repeatable signal, independent of the bending history.

An active needle is proposed for the development of MRI guided percutaneous procedures. The needle uses internal laser heating, conducted via optical fibers, of a shape memory alloy (SMA) actuator to produce bending in the distal section of the needle. Active bending of the needle as it is inserted allows it to reach small targets while overcoming the effects of interactions with surrounding tissue, which can otherwise deflect the needle away from its ideal path. The active section is designed to bend preferentially in one direction under actuation, and is also made from SMA for its combination of MR and bio-compatibility and its superelastic bending properties. A prototype, with a size equivalent to standard 16G biopsy needle, exhibits significant bending with a tip rotation of more than 10°. A numerical analysis and experiments provide information concerning the required amount of heating and guidance for design of efficient optical heating systems.

Magnetic resonance (MR) guided optical breast imaging is a promising modality to improve the specificity of breast imaging, because it provides high-resolution quantitative maps of total hemoglobin, oxygen saturation, water content, and optical scattering. These properties have been shown to distinguish malignant from benign lesions. However, the optical detection hardware required for deep tissue imaging has poor spectral sensitivity which limits accurate water quantification; this reduces the accuracy of hemoglobin quantification. We present a methodology to improve optical quantification by utilizing the ability of Dixon MR imaging to quantitatively estimate water and fat; this technique effectively reduces optical crosstalk between water and oxyhemoglobin. The techniques described in this paper reduce hemoglobin quantification error by as much as 38%, as shown in a numerical phantom, and an experimental phantom. Error is reduced by as much 20% when imperfect MR water quantification is given. These techniques may also increase contrast between diseased and normal tissue, as shown in breast tissue in vivo. It is also shown that using these techniques may permit fewer wavelengths to be used with similar quantitative accuracy, enabling higher temporal resolution. In addition, it is shown that these techniques can improve the ability of MRI to quantify water in the presence of bias in the Dixon water/fat separation.

Image-guided minimally invasive procedures have made a substantial impact in improving patient management, reducing the cost, morbidity, and mortality of treatments and making therapies available to patients who would otherwise have no option. X-ray fluoroscopy and magnetic resonance imaging (MRI) are two powerful tools for guiding interventional procedures but with very different strengths and weaknesses. X-ray fluoroscopy offers very high spatial and temporal resolution and is excellent for guiding and deploying devices. MRI offers tomographic imaging with complete freedom of plane orientation, outstanding soft tissue discrimination, and the ability to portray physiological responses during treatment. We have shown that it is feasible to fully integrate an X-ray fluoroscopy system into the bore of an interventional MR scanner to provide a single congruent field of view, with integration requiring minor modifications to the flat-panel digital detector, and using a static-anode X-ray tube. Given the limited availability of the MR scanner platform (0.5T GE Signa SP magnet), and the X-ray fluence limitations of the static-anode X-ray tube, we are now investigating the technology developments required to place a rotating-anode digital flat-panel X-ray system immediately adjacent to a closed-bore MRI system. These types of hybrid systems could have enormous impact in the diagnosis and treatment of oncologic, cardiovascular, and other disorders.