We investigate how vibrotactile wrist feedback can enhance spatial guidance for handheld tool movement in optical see-through augmented reality (AR). While AR overlays are widely used to support surgical tasks, visual occlusion, lighting conditions, and interface ambiguity can compromise precision and confidence. To address these challenges, we designed a multimodal system combining AR visuals with a custom wrist-worn haptic device delivering directional and state-based cues. A formative study with experienced surgeons and residents identified key tool maneuvers and preferences for reference mappings, guiding our cue design. In a cue identification experiment (N = 21), participants accurately recognized five vibration patterns under visual load, with higher recognition for full-actuator states than spatial direction cues. In a guidance task (N = 27), participants using both AR and haptics achieved significantly higher spatial precision (5.8 mm) and usability (SUS = 88.1) than those using either modality alone, albeit with modest increases in task time. Participants reported that haptic cues provided reassuring confirmation and reduced cognitive effort during alignment. Our results highlight the promise of integrating wrist-based haptics into AR systems for high-precision, visually complex tasks such as surgical guidance. We discuss design implications for multimodal interfaces supporting confident, efficient tool manipulation.

The use of Augmented Reality (AR) devices for surgical guidance has gained increasing traction in the medical field. Traditional registration methods often rely on external fiducial markers to achieve high accuracy and real-time performance. However, these markers introduce cumbersome calibration procedures and can be challenging to deploy in clinical settings. While commercial solutions have attempted real-time markerless tracking using the native RGB cameras of AR devices, their accuracy remains questionable for medical guidance, primarily due to occlusions and significant outliers between the live sensor data and the preoperative target anatomy point cloud derived from MRI or CT scans. In this work, we present a markerless framework that relies only on the depth sensor of AR devices and consists of two modules: a registration module for high-precision, outlier-robust target anatomy localization, and a tracking module for real-time pose estimation. The registration module integrates depth sensor error correction, a human-in-the-loop region filtering technique, and a robust global alignment with curvature-aware feature sampling, followed by local ICP refinement, for markerless alignment of preoperative models with patient a natomy. The tracking module employs a fast and robust registration algorithm that uses the initial pose from the registration module to estimate the target pose in real-time. We comprehensively evaluated the performance of both modules through simulation and real-world measurements. The results indicate that our markerless system achieves superior performance for registration and comparable performance for tracking to industrial solutions. The two-module design makes our system a one-stop solution for surgical procedures where the target anatomy moves or stays static during surgery.

Accelerated MRI reconstruction involves solving an ill-posed inverse problem where noise in acquired data propagates to the reconstructed images. Noise analyses are central to MRI reconstruction for providing an explicit measure of solution fidelity and for guiding the design and deployment of novel reconstruction methods. However, deep learning (DL)-based reconstruction methods have often overlooked noise propagation due to inherent analytical and computational challenges, despite its critical importance. This work proposes a theoretically grounded, memory-efficient technique to calculate voxel-wise variance for quantifying uncertainty due to acquisition noise in accelerated MRI reconstructions. Our approach is based on approximating the noise covariance using the DL network’s Jacobian, which is intractable to calculate. To circumvent this, we derive an unbiased estimator for the diagonal of this covariance matrix—voxel-wise variance—, and introduce a Jacobian sketching technique to efficiently implement it. We evaluate our method on knee and brain MRI datasets for both data-driven and physics-driven networks trained in supervised and unsupervised manners. Compared to empirical references obtained via Monte-Carlo simulations, our technique achieves near-equivalent performance while reducing computational and memory demands by an order of magnitude or more. Furthermore, our method is robust across varying input noise levels, acceleration factors, and diverse undersampling schemes, highlighting its broad applicability. Our work reintroduces accurate and efficient noise analysis as a central tenet of reconstruction algorithms, holding promise to reshape how we evaluate and deploy DL-based MRI.

Analyzing anatomic shapes of tissues and organs is pivotal for accurate disease diagnostics and clinical decision-making. One prominent disease that depends on anatomic shape analysis is osteoarthritis, which affects 30 million Americans. To advance osteoarthritis diagnostics and prognostics, we introduce ShapeMed-Knee, a 3D shape dataset with 9,376 high-resolution, medical-imaging-based 3D shapes of both femur bone and cartilage. Besides data, ShapeMed-Knee includes two benchmarks for assessing reconstruction accuracy and five clinical prediction tasks that assess the utility of learned shape representations. Leveraging ShapeMed-Knee, we develop and evaluate a novel hybrid explicit-implicit neural shape model which achieves up to 40% better reconstruction accuracy than a statistical shape model and two implicit neural shape models. Our hybrid models achieve state-of-the-art performance for preserving cartilage biomarkers (root mean squared error ≤ 0.05 vs. ≤ 0.07, 0.10, and 0.14). Our models are also the first to successfully predict localized structural features of osteoarthritis, outperforming shape models and convolutional neural networks applied to raw magnetic resonance images and segmentations (e.g., osteophyte size and localization 63% accuracy vs. 49-61%). The ShapeMed-Knee dataset provides medical evaluations to reconstruct multiple anatomic surfaces and embed meaningful disease-specific information. ShapeMed-Knee reduces barriers to applying 3D modeling in medicine, and our benchmarks highlight that advancements in 3D modeling can enhance the diagnosis and risk stratification for complex diseases. The dataset, code, and benchmarks are freely accessible.

This paper investigates the accuracy of Augmented Reality (AR) technologies, particularly commercially available optical see-through displays, in depicting virtual content inside the human body for surgical planning. Their inherent limitations result in inaccuracies in perceived object positioning. We examine how occlusion, specifically with opaque surfaces, affects perceived depth of virtual objects at arm's length working distances. A custom apparatus with a half-silvered mirror was developed, providing accurate depth cues excluding occlusion, differing from commercial displays. We carried out a study, contrasting our apparatus with a HoloLens 2, involving a depth estimation task under varied surface complexities and illuminations. In addition, we explored the effects of creating a virtual "hole" in the surface. Subjects' depth estimation accuracy and confidence were a ssessed. Results showed more depth estimation variation with HoloLens and significant depth error beneath complex occluding surfaces. However, creating a virtual hole significantly reduced depth errors and increased subjects' confidence, irrespective of accuracy enhancement. These findings have important implications for the design and use of mixed-reality technologies in surgical applications, and industrial applications such as using virtual content to guide maintenance or repair of components hidden beneath the opaque outer surface of equipment. A free copy of this paper and all supplemental materials are available at https://bit.ly/3YbkwjU.

In high-dimensional magnetic resonance imaging applications, time-consuming, sequential acquisition of data samples in the spatial frequency domain ( -space) can often be accelerated by accounting for dependencies in linear reconstruction, at the cost of noise amplification that depends on the sampling pattern. Common examples are support-constrained, parallel, and dynamic MRI, and -space sampling strategies are primarily driven by image-domain metrics that are expensive to compute for arbitrary sampling patterns. It remains challenging to provide systematic and computationally efficient automatic designs of arbitrary multidimensional Cartesian sampling patterns that mitigate noise amplification, given the subspace to which the object is confined. To address this problem, this paper introduces a theoretical framework that describes local geometric properties of the sampling pattern and relates them to the spread in the eigenvalues of the information matrix described by its first two spectral moments. This new criterion is then used for very efficient optimization of complex multidimensional sampling patterns that does not require reconstructing images or explicitly mapping noise amplification. Experiments with in vivo data show strong agreement between this criterion and traditional, comprehensive image-domain- and -space-based metrics, indicating the potential of the approach for computationally efficient (on-the-fly), automatic, and adaptive design of sampling patterns.

Magnetic resonance (MR) guided optical breast imaging is a promising modality to improve the specificity of breast imaging, because it provides high-resolution quantitative maps of total hemoglobin, oxygen saturation, water content, and optical scattering. These properties have been shown to distinguish malignant from benign lesions. However, the optical detection hardware required for deep tissue imaging has poor spectral sensitivity which limits accurate water quantification; this reduces the accuracy of hemoglobin quantification. We present a methodology to improve optical quantification by utilizing the ability of Dixon MR imaging to quantitatively estimate water and fat; this technique effectively reduces optical crosstalk between water and oxyhemoglobin. The techniques described in this paper reduce hemoglobin quantification error by as much as 38%, as shown in a numerical phantom, and an experimental phantom. Error is reduced by as much 20% when imperfect MR water quantification is given. These techniques may also increase contrast between diseased and normal tissue, as shown in breast tissue in vivo. It is also shown that using these techniques may permit fewer wavelengths to be used with similar quantitative accuracy, enabling higher temporal resolution. In addition, it is shown that these techniques can improve the ability of MRI to quantify water in the presence of bias in the Dixon water/fat separation.