Directly controlling robot motion through human–robot interaction enables the integration of human expertise into robot control. However, this requires efficient interaction methods. While previous work has shown that hand gestures offer potential for natural control, the lack of haptic feedback remains a key challenge. One solution is partial automation, in which the contact force is autonomously controlled. We implemented force automation enhanced gesture control, as well as the state-of-the-art baseline of hand-guiding, to systematically investigate hand gesture interaction for robot control. A user study (n=28) was conducted to investigate the efficiency and workload of these interaction concepts. As the interaction method efficiency may depend on task demands, two tasks with different precision requirements were evaluated in the context of robotic ultrasound. The results indicate that force automation significantly reduces perceived workload and task duration when using hand gestures. Moreover, gesture-based interaction can match the efficiency of hand-guiding for broad tasks and even outperform it in precise tasks. These findings highlight the potential of gesture-based interaction for robot control, particularly when the absence of haptic feedback is compensated by partial automation.

Immersive virtual reality learning environments (IVRLEs) are increasingly used in medical education, yet the role of environmental fidelity-particularly scene design-remains underexplored. This study examines how varying levels of fidelity and contextualization affect motivational and cognitive outcomes. Eighty-seven medical students were randomly assigned to one of three scene conditions: a minimalistic "Blank Scene," a "Reconstructed Classroom Scene", or an "Inside-Human Scene". All students used a custom-developed application to learn about embryonic heart development. We measured virtual presence, intrinsic motivation, cognitive load, learning outcomes, and usability. Results showed that scene design influenced virtual presence, selected aspects of intrinsic motivation, cognitive load, and learning outcomes. The Inside-Human Scene elicited higher physical and self-presence as well as higher comprehension scores compared to the Reconstructed Classroom Scene. The Reconstructed Classroom Scene was associated with higher extraneous cognitive load. Intrinsic cognitive load was rated higher in the Inside-Human Scene, while germane cognitive load did not differ between conditions. No significant differences were found for task performance or factual recall. Overall, the findings indicate that scene design in IVRLEs affects how learners engage with complex content and may support deeper understanding when perceptual and contextual properties are coherent, while visually detailed environments may increase extraneous cognitive demands without improving learning.

Robotic systems have the potential to enhance a wide range of domains, such as medical workflows, by automating individual steps of complex processes. However, human-robot interaction (HRI) is of critical importance, as effective collaboration between humans and robots is essential even in highly automated environments. Recent research has predominantly focused on the development of interaction methods rather than systematically comparing existing approaches. Therefore, we conducted a user study (n=20) to compare different HRI concepts for end effector manipulation combined with clutching mechanisms for manip-ulation activation in an alignment task using the example of robotic ultrasound (US). Manipulation methods included hand-guiding, teleoperation, and touchless interaction, while clutching mechanisms were realized through hand, voice, and foot interaction. The results indicate advantages of hand-guiding for manipulation. While no significant differences were observed between clutching mechanisms, strong evidence suggests comparable performance across these modalities. Notably, significant interaction effects on perceived workload reveal that the optimal clutching mechanism depends on the selected manipulation technique. This work underscores the critical importance of selecting appropriate HRI concepts and understanding the dependencies of manipulation techniques with clutching mechanisms. While our study included the usage of a robotic US, the insights gained are broadly transferable across various domains involving robotic manipulation tasks in human-robot collaboration.

Augmented Reality (AR) offers potential for enhancing human-robot collaboration by enabling intuitive interaction and real-time feedback. A crucial aspect of AR-robot integration is accurate spatial registration to align virtual content with the physical robotic workspace. This paper systematically investigates the effects of different tracking techniques and registration parameters on AR-to-robot registration accuracy, focusing on paired-point methods. We evaluate four marker detection algorithms - ARToolkit, Vuforia, ArUco, and retroreflective tracking - analyzing the influence of viewing distance, angle, marker size, point distance, distribution, and quantity. Our results show that ARToolkit provides the highest registration accuracy. While larger markers and positioning registration point centroids close to target locations consistently improved accuracy, other factors such as point distance and quantity were highly dependent on the tracking techniques used. Additionally, we propose an effective refinement method using point cloud registration, significantly improving accuracy by integrating data from points recorded between registration locations. These findings offer practical guidelines for enhancing AR-robot registration, with future work needed to assess the transferability to other AR devices and robots.

While collaborative robots effectively combine robotic precision with human capabilities, traditional control methods such as button presses or hand guidance can be slow and physically demanding. This has led to an increasing interest in natural user interfaces that integrate hand gesturebased interactions for more intuitive and flexible robot control. Therefore, this paper systematically explores mid-air robot control by comparing position and rate control modes with different state-of-the-art and novel sensor placements. A user study was conducted to evaluate each combination in terms of accuracy, task duration, perceived workload, and physical exertion. Our results indicate that position control is more efficient than rate control. Traditional desk-mounted sensors can provide a good balance between accuracy and comfort. However, robot-mounted sensors are a viable alternative for short-term, accurate control with less spatial requirements. Legmounted sensors, while comfortable, pose challenges to handeye coordination. Based on these findings, we provide design implications for improving the usability and comfort of midair human-robot interaction. Future research should extend this evaluation to a wider range of tasks and environments.

Extended Reality (XR) has the potential to improve efficiency and safety in the user-centered development of human-robot interaction. However, the validity of using XR prototyping for user studies for contact-intensive robotic tasks remains underexplored. These in-contact tasks are particularly relevant due to challenges arising from indirect force perception in robot control. Therefore, in this work, we investigate a representative example of such a task: robotic ultrasound. A user study was conducted to assess the transferability of results from a simulated user study to real-world conditions, comparing two force-assistance approaches. The XR simulation replicates the physical study set-up employing a virtual robotic arm, its control interface, ultrasound imaging, and two force-assistance methods: automation and force visualization. Our results indicate that while differences in force deviation, perceived workload, and trust emerge between real and simulated setups, the overall findings remain consistent. Specifically, partial automation of robot control improves performance and trust while reducing workload, and visual feedback decreases force deviation in both real and simulated conditions. These findings highlight the potential of XR for comparative studies, even in complex robotic tasks.

The lack of haptic feedback in touchless human-robot interaction is critical in applications such as robotic ultrasound, where force perception is crucial to ensure image quality. Augmented reality (AR) is a promising tool to address this limitation by providing sensory substitution through visual or vibrotactile feedback. The implementation of visual force feedback requires consideration not only of feedback design but also of positioning. Therefore, we implemented two different visualization types at three different positions and investigated the effects of vibrotactile feedback on these approaches. Furthermore, we examined the effects of multimodal feedback compared to visual or vibrotactile output alone. Our results indicate that sensory substitution eases the interaction in contrast to a feedback-less baseline condition, with the presence of visual support reducing average force errors and being subjectively preferred by the participants. However, the more feedback was provided, the longer users needed to complete their tasks. Regarding visualization design, a 2D bar visualization reduced force errors compared to a 3D arrow concept. Additionally, the visualizations being displayed directly on the ultrasound screen were subjectively preferred. With findings regarding feedback modality and visualization design our work represents an important step toward sensory substitution for touchless human-robot interaction.

Where traditional media and methods reach their limits in anatomy education, mixed-reality (MR) environments can provide effective learning support because of their high interactivity and spatial visualization capabilities. However, the underlying design and pedagogical requirements are as diverse as the technologies themselves. This paper examines the effectiveness of individual- and collaborative learning environments for anatomy education, using embryonic heart development as an example. Both applications deliver the same content using identical visualizations and hardware but differ in interactivity and pedagogical approach. The environments were evaluated in a user study with medical students (n = 90) during their examination phase, assessing usability, user experience, social interaction/co-presence, cognitive load, and personal preference. Additionally, we conducted a knowledge test before and after an MR learning session to determine educational effects compared to a conventional anatomy seminar. Results indicate that the individual learning environment was generally preferred. However, no significant difference in learning effectiveness could be shown between the conventional approach and the MR applications. This suggests that both can effectively complement traditional seminars despite their different natures. Our study contributes to understanding how different MR settings could be tailored for anatomical education.

Virtual surgical training systems are crucial for enabling mental preparation, supporting decision-making, and improving surgical skills. Many virtual surgical training environments focus only on training for a specific medical skill and take place in a single virtual room. However, surgical education and training include the planning of procedures as well as interventions in the operating room context. Moreover, collaboration among surgeons and other medical professionals is only applicable to a limited extent. This work presents a collaborative VR environment similar to a virtual teaching hospital to support surgical training and interprofessional collaboration in a co-located or remote environment. The environment supports photo-realistic avatars and scenarios ranging from planning to training procedures in the virtual operating room. It includes a lobby, a virtual surgical planning room with four surgical planning stations, laparoscopic liver surgery training with the integration of laparoscopic surgical instruments, and medical training scenarios for interprofessional team training in a virtual operating room. Each component was evaluated by domain experts as well as in a series of user studies, providing insights on usability, usefulness, and potential research directions. The proposed environment may serve as a foundation for future medical training simulators.

One of the biggest challenges in using ultrasound (US) is learning to create a spatial mental model of the interior of the scanned object based on the US image and the probe position. As intraoperative ultrasound (IOUS) cannot be easily trained on patients, we present LiVRSono, an immersive VR application to train this skill. The immersive environment, including an US simulation with patientspecific data as well as haptics to support hand-eye coordination, provides a realistic setting. Four clinically relevant training scenarios were identified based on the described learning goal and the workflow of IOUS for liver. The realism of the setting and the training scenarios were evaluated with eleven physicians, of which six participants are experts in IOUS for liver and five participants are potential users of the training system. The setting, handling of the US probe, and US image were considered realistic enough for the learning goal. Regarding the haptic feedback, a limitation is the restricted workspace of the input device. Three of the four training scenarios were rated as meaningful and effective. A pilot study regarding learning outcome shows positive results, especially with respect to confidence and perceived competence. Besides the drawbacks of the input device, our training system provides a realistic learning environment with meaningful scenarios to train the creation of a mental 3D model when performing IOUS. We also identified important improvements to the training scenarios to further enhance the training experience.

Recent developments and research advances contribute to an ever-increasing trend towards quality levels close to what we experience in reality. In this work, we investigate how different degrees of these quality characteristics affect user performance, qualia of user experience (UX), and sense of presence in an example medical task. To this end, a two-way within-subjects design user study was conducted, in which three different levels of visual fidelity were compared. In addition, two different interaction modalities were considered: (1) the use of conventional VR controllers and (2) natural hand interaction using 3D-printed, spatially-registered replicas of medical devices, to interact with their virtual representations. Consistent results indicate that higher degrees of visual fidelity evoke a higher sense of presence and UX. However, user performance was less affected. Moreover, no differences were detected between both interaction modalities for the examined task. Future work should investigate the discovered interaction effects between quality levels and interaction modalities in more detail and examine whether these results can be reproduced in tasks that require more precision. This work provides insights into the implications to consider when studying interactions in VR and paves the way for investigations into early phases of medical product development and workflow analysis.

Presenting medical volume data using augmented reality (AR) can facilitate the identification of anatomical structures, the perception of their spatial relations and the development of mental maps compared to more commonly used monitors. However, interaction methods explored in these conventional settings may not be applicable in AR environments, or perform differently. In terms of mode activation, gestural interaction was shown to be a viable, touchless alternative to traditional input devices, which is desirable in sterile medical use cases. Therefore, we present a user study (n = 21) comparing hand and foot gestures with voice commands for the activation of interaction modes within a projector-based, spatial AR prototype to visualize medical volume data. Interaction itself was performed via hand movements captured by a data glove. Consistent, statistically significant results across measured variables suggest advantages of voice commands. In addition, a second experiment (n = 17) compared the hand-based interaction with two motion-sensitive devices held in power and in precision grip respectively. All modes were activated using voice commands. No considerable differences between tested grip styles could be determined. The findings suggest that the choice of preferable interaction devices is user and use case dependent.

In the embryonic human heart, complex dynamic shape changes take place in a short period of time on a microscopic scale, making this development difficult to visualize. However, spatial understanding of these processes is essential for students and future cardiologists to properly diagnose and treat congenital heart defects. Following a user centered approach, the most crucial embryological stages were identified and translated into a virtual reality learning environment (VRLE) to enable the understanding of the morphological transitions of these stages through advanced interactions. To address individual learning types, we implemented different features and evaluated the application regarding usability, perceived task load, and sense of presence in a user study. We also assessed spatial awareness and knowledge gain, and finally obtained feedback from domain experts. Overall, students and professionals rated the application positively. To minimize distraction from interactive learning content, such VRLEs should consider features for different learning types, allow for gradual habituation, and at the same time provide enough playful stimuli. Our work previews how VR can be integrated into a cardiac embryology education curriculum.

Surgical procedures requiring needle navigation assistance suffer from complicated hand-eye coordination and are mentally demanding. Augmented reality (AR) can help overcome these issues. How-ever, only an insufficient amount of fundamental research has focused on the design and hardware selection of such AR needle navigation systems. This work contributes to this research area by presenting a user study (n=24) comparing three state-of-the-art navigation concepts displayed by an optical see-through head-mounted display and a stereoscopic projection system. A two-dimensional glyph visualization resulted in higher targeting accuracy but required more needle insertion time. In contrast, punctures guided by a three-dimensional see-through vision concept were less accurate but faster and were favored in a qualitative interview. The third concept, a static representation of the correctly positioned needle, showed too high target errors for clinical accuracy needs. This concept per-formed worse when displayed by the projection system. Besides that, no meaningful differences between the evaluated AR display devices were detected. User preferences and use case restrictions, e.g., sterility requirements, seem to be more crucial selection criteria. Future work should focus on improving the accuracy of the see-through vision concept. Until then, the glyph visualization is recommended.

Since for most consumers the Virtual Reality (VR) experience exceeds that of desktop applications, an increasing number of applications is being transferred from desktop to VR. Industrial and entertainment applications primarily expect for a richer consumer experience, while others, such as surgical applications, seek for improved precision over their desktop counterparts. One way to improve the performance of precision-based VR applications is to provide suitable visualizations. Today, these "suitable" visualizations are mostly transferred from desktop to VR without considering their spatial and temporal performance might change in VR. This may not lead to an optimal solution, which can be crucial for precision-based tasks. Misinterpretation of shape or distance in a surgical or pre-operative simulation can affect the chosen treatment and thus directly impact the outcome. Therefore, we evaluate the performance differences of multiple visualizations for 3D surfaces based on their shape and distance estimation for desktop and VR applications. We conducted a quantitative user study with 56 participants evaluating seven visualizations (Phong, Toon, Fresnel, Pseudo-Chromadepth, Heatmap, Isolines, and Arrow Glyphs). Our results show that the performance of each visualization varies depending on the task, system, and surface type, with VR generally providing improved results. While Isolines are able to improve distance estimation, Phong and Heatmaps are beneficial for shape estimation.

The identification of subsurface structures during resection wound repair is a challenge during minimally invasive partial nephrectomy. Specifically, major blood vessels and branches of the urinary collecting system need to be localized under time pressure as target or risk structures during suture placement. This work presents concepts for AR visualization and auditory guidance based on tool position that support this task. We evaluated the concepts in a laboratory user study with a simplified, simulated task: The localization of subsurface target points in a healthy kidney phantom. We evaluated the task time, localization accuracy, and perceived workload for our concepts and a control condition without navigation support. The AR visualization improved the accuracy and perceived workload over the control condition. We observed similar, non-significant trends for the auditory display. Further, clinically realistic evaluation is pending. Our initial results indicate the potential benefits of our concepts in supporting laparoscopic resection wound repair.

A high level of motivation and frequent training are relevant in software-based rehabilitation to improve cognitive functioning after acquired brain injury. We evaluated the benefit of tailored user-centered gamification elements in a clinical study with N=83 outpatients undergoing three weeks of cognitive training in their home environment. The use of gamification in relation to the patient’s player type was explored in three steps. First, we determined the individual player types and related requests for specific game elements by means of questionnaires. Afterwards, we examined the effect of gamified training based on a non-player character and training progress within a metaphor. We considered secondly the individual perception and emotional effect and thirdly the performance based on training duration. 37 elements were requested by patients of all types, 18 elements were partially requested, and 4 elements were rejected. A comparison shows that the requested game elements partly differ between healthy persons and patients. Overall, gamification was perceived positively and gamified training leads to an increase in enjoyment compared to non-gamified training. In detail, however, there were different effects on the individual player types: socialisers experienced more enjoyment while achievers perceived higher competence throughout gamified cognitive training. Also, differences in performance in training duration were found. Within gamified training, socialisers trained significantly more than patients not primarily assigned to this type. In contrast, no significant difference was found for achievers. By showing modulating requests and effects in player types, our results support user-centered tailoring of game elements in the development of software-based cognitive training in rehabilitation.

We present a Virtual and Augmented Reality multi-user prototype of a learning environment for liver anatomy education. Our system supports various training scenarios ranging from small learning groups to classroom-size education, where students and teachers can participate in virtual reality, augmented reality, or via desktop PCs. In an iterative development process with surgeons and teachers, a virtual organ library was created. Nineteen liver data sets were used comprising 3D surface models, 2D image data, pathology information, diagnosis and treatment decisions. These data sets can interactively be sorted and investigated individually regarding their volumetric and meta information. The three participation modes were evaluated within a user study with surgery lecturers (5) and medical students (5). We assessed the usability and presence using questionnaires. Additionally, we collected qualitative data with semistructured interviews. A total of 435 individual statements were recorded and summarized to 49 statements. The results show that our prototype is usable, induces presence, and potentially support the teaching of liver anatomy and surgery in the future.

Complex bi-manual tasks often benefit from supporting visual information and guidance. Controlling the system that provides this information is a secondary task that forces the user to perform concurrent multitasking, which in turn may affect the main task performance. Interactions based on natural behavior are a promising solution to this challenge. We investigated the performance of these interactions in a hands-free image manipulation task during a primary manual task with an upright stance. Essential tasks were extracted from the example of clinical workflow and turned into an abstract simulation to gain general insights into how different interaction techniques impact the user's performance and workload. The interaction techniques we compared were full-body movements, facial expression, gesture and speech input. We found that leaning as an interaction technique facilitates significantly faster image manipulation at lower subjective workloads than facial expression. Our results pave the way towards efficient, natural, hands-free interaction in a challenging multitasking environment.

Augmented reality (AR) may be a useful technique to overcome issues of conventionally used navigation systems supporting medical needle insertions, like increased mental workload and complicated hand-eye coordination. Previous research primarily focused on the development of AR navigation systems designed for specific displaying devices, but differences between employed methods have not been investigated before. To this end, a user study involving a needle insertion task was conducted comparing different AR display techniques with a monitor-based approach as baseline condition for the visualization of navigation information. A video see-through stationary display, an optical see-through head-mounted display and a spatial AR projector-camera-system were investigated in this comparison. Results suggest advantages of using projected navigation information in terms of lower task completion time, lower angular deviation and affirmative subjective participant feedback. Techniques requiring the intermediate view on screens, i.e. the stationary display and the baseline condition, showed less favorable results. Thus, benefits of providing AR navigation information compared to a conventionally used method could be identified. Significant objective measures results, as well as an identification of advantages and disadvantages of individual display techniques contribute to the development and design of improved needle navigation systems.

Augmented reality (AR) is a promising tool to convey useful information at the place where it is needed. However, perceptual issues with augmented reality visualizations affect the estimation of distances and depth and thus can lead to critically wrong assumptions. These issues have been successfully investigated for video see-through modalities. Moreover, advanced visualization methods encoding depth information by displaying additional depth cues were developed. In this work, state-of-the-art visualization concepts were adopted for a projective AR setup. We conducted a user study to assess the concepts’ suitability to convey depth information. Participants were asked to sort virtual cubes by using the provided depth cues. The investigated visualization concepts consisted of conventional Phong shading, a virtual mirror, depth-encoding silhouettes, pseudo-chromadepth rendering and an illustrative visualization using supporting line depth cues. Besides different concepts, we altered between a monoscopic and a stereoscopic display mode to examine the effects of stereopsis. Consistent results across variables show a clear ranking of examined concepts. The supporting lines approach and the pseudo-chromadepth rendering performed best. Stereopsis was shown to provide significant advantages for depth perception, while the current visualization technique had only little effect on investigated measures in this condition. However, similar results were achieved using the supporting lines and the pseudo-chromadepth concepts in a monoscopic setup. Our study showed the suitability of advanced visualization concepts for the rendering of virtual content in projective AR. Specific depth estimation results contribute to the future design and development of applications for these systems.

The estimation of distances and spatial relations between surgical instruments and surrounding anatomical structures is a challenging task for clinicians in image-guided surgery. Using augmented reality (AR), navigation aids can be displayed directly at the intervention site to support the assessment of distances and reduce the risk of damage to healthy tissue. To this end, four distance-encoding visualisation concepts were developed using a head-mounted optical see-through AR setup and evaluated by conducting a comparison study. Results suggest the general advantage of the proposed methods compared to a blank visualisation providing no additional information. Using a Distance Sensor concept signalising the proximity of nearby structures resulted in the least time the instrument was located below 5mm to surrounding risk structures and yielded the least amount of collisions with them.

Augmented reality (AR) is a promising tool to improve instrument navigation in needle-based interventions. Limited research has been conducted regarding suitable navigation visualizations. In this work, three navigation concepts based on existing approaches were compared in a user study using a projective AR setup. Each concept was implemented with three different scales for accuracy-to-color mapping and two methods of navigation indicator scaling. Participants were asked to perform simulated needle insertion tasks with each of the resulting 18 prototypes. Insertion angle and insertion depth accuracies were measured and analyzed, as well as task completion time and participants' subjectively perceived task difficulty. Results show a clear ranking of visualization concepts across variables. Less consistent results were obtained for the color and indicator scaling factors. Results suggest that logarithmic indicator scaling achieved better accuracy, but participants perceived it to be more difficult than linear scaling. With specific results for angle and depth accuracy, our study contributes to the future composition of improved navigation support and systems for precise needle insertion or similar applications.

During medical interventions, direct interaction with medical image data is a cumbersome task for physicians due to the sterile environment. Even though touchless input via hand, foot or voice is possible, these modalities are not available for these tasks all the time. Therefore, we investigated touchless input methods as alternatives to each other with focus on two common interaction tasks in sterile settings: activation of a system to avoid unintentional input and manipulation of continuous values. We created a system where activation could be achieved via voice, hand or foot gestures and continuous manipulation via hand and foot gestures. We conducted a comparative user study and found that foot interaction performed best in terms of task completion times and scored highest in the subjectively assessed measures usability and usefulness. Usability and usefulness scores for hand and voice were only slightly worse and all participants were able to perform all tasks in a sufficient short amount of time. This work contributes by proposing methods to interact with computers in sterile, dynamic environments and by providing evaluation results for direct comparison of alternative modalities for common interaction tasks.

During minimally-invasive interventions, physicians need to interact with medical image data, which cannot be done while the hands are occupied. To address this challenge, we propose two interaction techniques which use gaze and foot as input modalities for hands-free interaction. To investigate the feasibility of these techniques, we created a setup consisting of a mobile eye-tracking device, a tactile floor, two laptops, and the large screen of an angiography suite. We conducted a user study to evaluate how to navigate medical images without the need for hand interaction. Both multimodal approaches, as well as a foot-only interaction technique, were compared regarding task completion time and subjective workload. The results revealed comparable performance of all methods. Selection is accomplished faster via gaze than with a foot only approach, but gaze and foot easily interfere when used at the same time. This paper contributes to HCI by providing techniques and evaluation results for combined gaze and foot interaction when standing. Our method may enable more effective computer interactions in the operating room, resulting in a more beneficial use of medical information.

We present real-time vascular visualization methods, which extend on illustrative rendering techniques to particularly accentuate spatial depth and to improve the perceptive separation of important vascular properties such as branching level and supply area. The resulting visualization can and has already been used for direct projection on a patient's organ in the operation theater where the varying absorption and reflection characteristics of the surface limit the use of color. The important contributions of our work are a GPU-based hatching algorithm for complex tubular structures that emphasizes shape and depth as well as GPU-accelerated shadow-like depth indicators, which enable reliable comparisons of depth distances in a static monoscopic 3D visualization. In addition, we verify the expressiveness of our illustration methods in a large, quantitative study with 160 subjects.