Objective: The use of commercially available optical-see-through (OST) head-mounted displays (HMDs) in their own peripersonal space leads the user to experience two perception conflicts that deteriorate their performance in precision manual tasks: the vergence-accommodation conflict (VAC) and the focus rivalry. In this work, we aim characterizing for the first time the psychophysiological response associated with user's incorrect focus cues during the execution of an augmented reality (AR)-guided manual task with the Microsoft HoloLens OST-HMD. Methods: 21 subjects underwent to a “connecting-the-dots” experiment with and without the use of AR, and in both binocular and monocular conditions. For each condition, we quantified the changes in autonomic nervous system (ANS) activity of subjects by analyzing the electrodermal activity (EDA) and heart rate variability. Moreover, we analyzed the neural central correlates by means of power measures of brain activity and multivariate autoregressive measures of brain connectivity extracted from the electroencephalogram (EEG). Results: No statistically significant differences of ANS correlates were observed among tasks, although all EDA-related features varied between rest and task conditions. Conversely, significant differences among conditions were present in terms of EEG-power variations in the$\mu$(8–13) Hz and$\beta$(13–30) Hz bands. In addition, significant changes in the causal interactions of a brain network involved in motor movement and eye-hand coordination comprising the precentral gyrus, the precuneus, and the fusiform gyrus were observed. Conclusion: The physiological plausibility of our results suggest promising future applicability to investigate more complex scenarios, such as AR-guided surgery.

For an optical see-through (OST) augmented reality (AR) head-mounted display (HMD) to assist in performing high-precision activities in the peripersonal space, a fundamental requirement is the correct spatial registration between the virtual information and the real environment. This registration can be achieved through a calibration procedure involving the parameterization of the virtual rendering camera via an eye-replacement camera that observes a calibration pattern rendered onto the OST display. In a previous feasibility study, we demonstrated and proved, with the same eye-replacement camera used for the calibration, that, in the case of an OST display with a focal plane close to the user's working distance, there is no need for prior-to-use viewpoint-specific calibration refinements obtained through eye-tracking cameras or additional alignment-based calibration steps. The viewpoint parallax-related AR registration error is indeed submillimetric within a reasonable range of depths around the display focal plane. This article confirms, through a user study based on a monocular virtual-to-real alignment task, that this finding is accurate and usable. In addition, we found that by performing the alignment-free calibration procedure via a high-resolution camera, the AR registration accuracy is substantially improved compared with that of other state-of-the-art approaches, with an error lower than 1mm over a notable range of distances. These results demonstrate the safe usability of OST HMDs for high-precision task guidance in the peripersonal space.

Aortic valve surgery is the preferred procedure for replacing a damaged valve with an artificial one. The ValveTech robotic platform comprises a flexible articulated manipulator and surgical interface supporting the effective delivery of an artificial valve by teleoperation and endoscopic vision. This article presents our recent work on force-perceptive, safe, semiautonomous navigation of the ValveTech platform prior to valve implantation. First, we present a force observer that transfers forces from the manipulator body and tip to a haptic interface. Second, we demonstrate how hybrid forward/inverse mechanics, together with endoscopic visual servoing, lead to autonomous valve positioning. Benchtop experiments and an artificial phantom quantify the performance of the developed robot controller and navigator. Valves can be autonomously delivered with a 2.0±0.5 mm position error and a minimal misalignment of 3.4±0.9°. The hybrid force/shape observer (FSO) algorithm was able to predict distributed external forces on the articulated manipulator body with an average error of 0.09 N. FSO can also estimate loads on the tip with an average accuracy of 3.3%. The presented system can lead to better patient care, delivery outcome, and surgeon comfort during aortic valve surgery, without requiring sensorization of the robot tip, and therefore obviating miniaturization constraints.

Augmented reality (AR) devices are gaining popularity in industrial development and healthcare as they provide information that would not be accessible in a rapid and intuitive way. Head-mounted displays dominate in this field and are currently being comprehensively tested. Alongside its informative function, AR can be used to steer the user's actions to aid complex or high precision tasks. This is the case in surgery, which is recently seeing the development of ad-hoc head-mounted displays to meet the requirements of safety, ergonomics, and reliability. However, head-mounted displays are subject to perceptual problems that can affect their use in delicate and demanding tasks. This article aims to evaluate projected AR as an alternative to head mounted displays (HMDs) when accurate guidance on the surface is needed. We directly compare them in a user study and evaluate both user accuracy and user perception to assess whether projected AR can be a practical and useful paradigm for precision manual tasks. Ten users performed tracing trajectory tasks under the guidance of an HMD and a projected AR device. Three accuracy levels were quantitatively tested: 0.5, 1, and 2 mm. Statistical analysis showed no significant difference in the accuracy of the two AR visualization modes, whereas the user perception assessment revealed statistical differences in virtual-to-real perception and visual discomfort. The quantitative results of this article proved that both technologies can guide manual precision tasks with the same accuracy, but projected AR features some perceptual advantages.

Egocentric augmented reality (AR) interfaces are quickly becoming a key asset for assisting high precision activities in the peripersonal space in several application fields. In these applications, accurate and robust registration of computer-generated information to the real scene is hard to achieve with traditional Optical See-Through (OST) displays given that it relies on the accurate calibration of the combined eye-display projection model. The calibration is required to efficiently estimate the projection parameters of the pinhole model that encapsulate the optical features of the display and whose values vary according to the position of the user's eye. In this article, we describe an approach that prevents any parallax-related AR misregistration at a pre-defined working distance in OST displays with infinity focus; our strategy relies on the use of a magnifier placed in front of the OST display, and features a proper parameterization of the virtual rendering camera achieved through a dedicated calibration procedure that accounts for the contribution of the magnifier. We model the registration error due to the viewpoint parallax outside the ideal working distance. Finally, we validate our strategy on a OST display, and we show that sub-millimetric registration accuracy can be achieved for working distances of ±100 mm around the focal length of the magnifier.

The adoption of real-time object detection systems via video streaming analysis is currently exploited in several contexts, from security monitoring to safety prevention. In industrial environments, proper usage of Personal Protective Equipment (PPE) is paramount to ensure workers’ safety. However, the use of some types of PPE, such as helmets, is often neglected by workers, especially in indoor areas. Thus, in order to reduce the risks of accidents, real-time video streaming-based monitoring systems may be used to monitor areas in which workers operate and alert them not to wear PPEs via acoustic alarms or visual signals. In case of a remote analysis, there are potential issues related to the high rate of data streams to be transported and analyzed and workers’ privacy. In this work, we propose an embedded smart system for real-time PPE detection based on video streaming analysis and deep learning models. We discuss the deployment of different versions of the YOLOv4 network fine-tuned using a public PPE dataset. In the end, we assess the performance of the proposed system in terms of accuracy and latency and of the overall PPE detection procedure.

We present an intuitive and ergonomic AR strategy to be coupled with a standard external fixation system aimed at aiding the accurate closed reduction of long-bone shaft fractures. The correct six DOF alignment between the bone fragments can be retrieved by manually repositioning a pair of reference frames constrained to the two extremities of the fixator so as to minimize the geometric distance, on the image plane, between planned/virtual landmarks and their observed/real counterparts. The reduction accuracy was positively validated in vitro in a pilot study that involved an orthopedic surgeon.

In the context of image-guided surgery (IGS), AR technology appears as a significant development in the field since it complements and integrates the concepts of surgical navigation based on virtual reality. The aim of the project is to optimize and validate an ergonomic, accurate and cheap video see-through AR system as an aid in various typologies of surgical procedures. The system will ideally have to be inexpensive and user-friendly to be successfully introduced in the clinical practice.

Stereoscopic video see though AR systems permit accurate marker video based registration. To guarantee accurate registration, cameras are normally rigidly blocked while the user could require changing their vergence. We propose a solution working with lightweight hardware that, without the need for a new calibration of the cameras relative pose after each vergence adjustment, guarantees registration accuracy using pre-determined calibration data.