For an optical see-through (OST) augmented reality (AR) head-mounted display (HMD) to assist in performing high-precision activities in the peripersonal space, a fundamental requirement is the correct spatial registration between the virtual information and the real environment. This registration can be achieved through a calibration procedure involving the parameterization of the virtual rendering camera via an eye-replacement camera that observes a calibration pattern rendered onto the OST display. In a previous feasibility study, we demonstrated and proved, with the same eye-replacement camera used for the calibration, that, in the case of an OST display with a focal plane close to the user's working distance, there is no need for prior-to-use viewpoint-specific calibration refinements obtained through eye-tracking cameras or additional alignment-based calibration steps. The viewpoint parallax-related AR registration error is indeed submillimetric within a reasonable range of depths around the display focal plane. This article confirms, through a user study based on a monocular virtual-to-real alignment task, that this finding is accurate and usable. In addition, we found that by performing the alignment-free calibration procedure via a high-resolution camera, the AR registration accuracy is substantially improved compared with that of other state-of-the-art approaches, with an error lower than 1mm over a notable range of distances. These results demonstrate the safe usability of OST HMDs for high-precision task guidance in the peripersonal space.

Augmented reality (AR) devices are gaining popularity in industrial development and healthcare as they provide information that would not be accessible in a rapid and intuitive way. Head-mounted displays dominate in this field and are currently being comprehensively tested. Alongside its informative function, AR can be used to steer the user's actions to aid complex or high precision tasks. This is the case in surgery, which is recently seeing the development of ad-hoc head-mounted displays to meet the requirements of safety, ergonomics, and reliability. However, head-mounted displays are subject to perceptual problems that can affect their use in delicate and demanding tasks. This article aims to evaluate projected AR as an alternative to head mounted displays (HMDs) when accurate guidance on the surface is needed. We directly compare them in a user study and evaluate both user accuracy and user perception to assess whether projected AR can be a practical and useful paradigm for precision manual tasks. Ten users performed tracing trajectory tasks under the guidance of an HMD and a projected AR device. Three accuracy levels were quantitatively tested: 0.5, 1, and 2 mm. Statistical analysis showed no significant difference in the accuracy of the two AR visualization modes, whereas the user perception assessment revealed statistical differences in virtual-to-real perception and visual discomfort. The quantitative results of this article proved that both technologies can guide manual precision tasks with the same accuracy, but projected AR features some perceptual advantages.

Visual augmented reality (AR) has the potential to improve the accuracy, efficiency and reproducibility of computer-assisted orthopaedic surgery (CAOS). AR Head-mounted displays (HMDs) further allow non-eye-shift target observation and egocentric view. Recently, a markerless tracking and registration (MTR) algorithm was proposed to avoid the artificial markers that are conventionally pinned into the target anatomy for tracking, as their use prolongs surgical workflow, introduces human-induced errors, and necessitates additional surgical invasion in patients. However, such an MTR-based method has neither been explored for surgical applications nor integrated into current AR HMDs, making the ergonomic HMD-based markerless AR CAOS navigation hard to achieve. To these aims, we present a versatile, device-agnostic and accurate HMD-based AR platform. Our software platform, supporting both video see-through (VST) and optical see-through (OST) modes, integrates two proposed fast calibration procedures using a specially designed calibration tool. According to the camera-based evaluation, our AR platform achieves a display error of 6.31$\pm$2.55 arcmin for VST and 7.72$\pm$3.73 arcmin for OST. A proof-of-concept markerless surgical navigation system to assist in femoral bone drilling was then developed based on the platform and Microsoft HoloLens 1. According to the user study, both VST and OST markerless navigation systems are reliable, with the OST system providing the best usability. The measured navigation error is 4.90$\pm$1.04 mm, 5.96$\pm$2.22$^\circ$for the VST system, and 4.36$\pm$0.80 mm, 5.65$\pm$1.42$^\circ$for the OST system.

Egocentric augmented reality (AR) interfaces are quickly becoming a key asset for assisting high precision activities in the peripersonal space in several application fields. In these applications, accurate and robust registration of computer-generated information to the real scene is hard to achieve with traditional Optical See-Through (OST) displays given that it relies on the accurate calibration of the combined eye-display projection model. The calibration is required to efficiently estimate the projection parameters of the pinhole model that encapsulate the optical features of the display and whose values vary according to the position of the user's eye. In this article, we describe an approach that prevents any parallax-related AR misregistration at a pre-defined working distance in OST displays with infinity focus; our strategy relies on the use of a magnifier placed in front of the OST display, and features a proper parameterization of the virtual rendering camera achieved through a dedicated calibration procedure that accounts for the contribution of the magnifier. We model the registration error due to the viewpoint parallax outside the ideal working distance. Finally, we validate our strategy on a OST display, and we show that sub-millimetric registration accuracy can be achieved for working distances of ±100 mm around the focal length of the magnifier.

The inaccessibility of user-perceived reality remains an open issue in pursuing the accurate calibration of optical see-through (OST) head-mounted displays (HMDs). Manual user alignment is usually required to collect a set of virtual-to-real correspondences, so that a default or an offline display calibration can be updated to account for the user’s eye position(s). Current alignment-based calibration procedures usually require point-wise alignments between rendered image point(s) and associated physical landmark(s) of a target calibration tool. As each alignment can only provide one or a few correspondences, repeated alignments are required to ensure calibration quality. This work presents an accurate and tool-less online OST calibration method to update an offline-calibrated eye-display model. The user’s bare hand is markerlessly tracked by a commercial RGBD camera anchored to the OST headset to generate a user-specific cursor for correspondence collection. The required alignment is object-wise, and can provide thousands of unordered corresponding points in tracked space. The collected correspondences are registered by a proposed rotation-constrained iterative closest point (rcICP) method to optimise the viewpoint-related calibration parameters. We implemented such a method for the Microsoft HoloLens 1. The resiliency of the proposed procedure to noisy data was evaluated through simulated tests and real experiments performed with an eye-replacement camera. According to the simulation test, the rcICP registration is robust against possible user-induced rotational misalignment. With a single alignment, our method achieves 8.81 arcmin (1.37 mm) positional error and 1. 76° rotational error by camera-based tests in the arm-reach distance, and 10.79 arcmin (7.71 pixels) reprojection error by user tests.

We present an intuitive and ergonomic AR strategy to be coupled with a standard external fixation system aimed at aiding the accurate closed reduction of long-bone shaft fractures. The correct six DOF alignment between the bone fragments can be retrieved by manually repositioning a pair of reference frames constrained to the two extremities of the fixator so as to minimize the geometric distance, on the image plane, between planned/virtual landmarks and their observed/real counterparts. The reduction accuracy was positively validated in vitro in a pilot study that involved an orthopedic surgeon.

In the context of image-guided surgery (IGS), AR technology appears as a significant development in the field since it complements and integrates the concepts of surgical navigation based on virtual reality. The aim of the project is to optimize and validate an ergonomic, accurate and cheap video see-through AR system as an aid in various typologies of surgical procedures. The system will ideally have to be inexpensive and user-friendly to be successfully introduced in the clinical practice.

Stereoscopic video see though AR systems permit accurate marker video based registration. To guarantee accurate registration, cameras are normally rigidly blocked while the user could require changing their vergence. We propose a solution working with lightweight hardware that, without the need for a new calibration of the cameras relative pose after each vergence adjustment, guarantees registration accuracy using pre-determined calibration data.

Handshaking represents a complex motor and cognitive task that poses several challenges from both engineering and neuroscientific viewpoints. In particular, it is an intriguing application which can be profitably studied in the field of Human Robot Interaction (HRI). In this work an experimental paradigm is proposed to investigate the neural correlates of handshaking between humans and between humans and robots using functional Magnetic Resonance Imaging. More specifically the role of visual and haptic components during handshaking interaction will be studied. A wearable sensing glove will be used to monitor hand finger position and movement. Preliminary results will be reported and discussed.