Enabling collaborative telepresence in healthcare, especially surgical procedures, presents a critical challenge. The decompressive craniotomy procedure stands out as particularly complex and time-sensitive. The current teleconsultation approach relies on 2D color cameras, often offering only a fixed view and limited visual capabilities between experts and surgeons. However, teleconsultation can be addressed with Mixed Reality and immersive technology to potentially enable a better consultation of the procedure. We conducted an extensive user study focusing on decompressive craniotomy to investigate the advantages and challenges of our 3D teleconsultation system compared to a 2D video-based consultation system. Our 3D teleconsultation system leverages real-time 3D reconstruction of the patient and environment to empower experts to provide guidance and create virtual 3D annotations. The study utilized 3D-printed head models to perform a lifelike surgical intervention. It involved 14 medical residents and demonstrated an in-vitro 17% improvement in accurately describing the incision size on the patient’s head, contributing to potentially improved patient outcomes.

When two or more users attempt to collaborate in the same space with Augmented Reality, they often encounter conflicting intentions regarding the occupation of the same working area and self-positioning around such without mutual interference. Augmented Reality is a powerful tool for communicating ideas and intentions during a co-assisting task that requires multi-disciplinary expertise. To relax the constraint of physical co-location, we propose the concept of Duplicated Reality, where a digital copy of a 3D region of interest of the users' environment is reconstructed in real-time and visualized in-situ through an Augmented Reality user interface. This enables users to remotely annotate the region of interest while being co-located with others in Augmented Reality. We perform a user study to gain an in-depth understanding of the proposed method compared to an in-situ augmentation, including collaboration, effort, awareness, usability, and the quality of the task. The result indicates almost identical objective and subjective results, except a decrease in the consulting user's awareness of co-located users when using our method. The added benefit from duplicating the working area into a designated consulting area opens up new interaction paradigms to be further investigated for future co-located Augmented Reality collaboration systems.

When users create hand-drawn annotations in Virtual Reality they often reach their physical limits in terms of precision, especially if the region to be annotated is small. One intuitive solution employs magnification beyond natural scale. However, scaling the whole environment results in wrong assumptions about the coherence between physical and virtual space. In this paper, we introduce Mag-noramas, a novel interaction method for selecting and extracting a region of interest that the user can subsequently scale and transform inside the virtual space. Our technique enhances the user's capabilities to perform supernaturally precise virtual annotations on virtual objects. We explored our technique in a user study within asimplified clinical scenario of a teleconsultation-supported craniectomy procedure that requires accurate annotations on a human head. Teleconsultation was performed asymmetrically between a remote expert in Virtual Reality that collaborated with a local user through Augmented Reality. The remote expert operates inside a reconstructed environment, captured from RGB-D sensors at the local site, and is embodied by an avatar to establish co-presence. The results show that Magnoramas significantly improve the precision of annotations while preserving usability and perceived presence measures compared to the baseline method. By hiding the 3D reconstruction while keeping the Magnorama, users can intentionally choose to lower their perceived social presence and focus on their tasks.

We present a unique simulator-based methodology for assessing both technical and nontechnical (cognitive) skills for surgical trainees while immersed in a complete medical simulation environment. Further, we have included two crisis scenarios which allow for the evaluation of the effect of cognitive strategy selection on the low-level surgical skills. Training these mixed-mode scenarios can thereby be evaluated on our platform, allowing for improved assessment and a stronger foundation for credentialing, with the potential to reduce the occurrence of adverse events in the operating room. Scientific evaluation and validation of our work is conducted together with 19 junior surgeons in order to achieve the following goals: 1) to provide a qualitative measure of usability, 2) to assess vertebroplasty technical performance of the surgeon, and 3) to explore the relationship between mental workload and surgical performance during crisis. Our results indicate that: 1) the surgeons scored the face validity of our modeled simulation environment very highly ( 4.68 ±0.48, using a 5-point Likert scale), 2) surgeon training enabled completion of tasks more quickly, and 3) the introduction of crisis scenarios negatively affected the surgeons' objective performance. Taken together, our results underscore the need to develop realistic simulation environments that prepare young residents to respond to emergent events in the operating room.