In our current work we explore the feasibility of using visuo-haptic augmented reality techniques for training and remote collaboration. Accurate calibration of system components is a key factor in this context. In this paper we deal with the calibration of the kinematic parameters of the haptic device. We compare a previously developed technique using an external optical tracker to a new approach not relying on additional devices. Simulated and real experiments are carried out to investigate the performance of the new technique. As will be shown, both methods yield reasonable results with average errors in the workspace below 2mm. It is found that the trackerless technique is an acceptable alternative to using a tracker for calibration, with only slightly reduced accuracy. It is straightforward to apply and allows to improve rendering fidelity.

Haptic augmented reality (AR) allows to modulate the haptic properties of a real object by providing virtual haptic feedback. We previously developed a haptic AR system wherein the stiffness of a real object can be augmented with the aid of a haptic interface. To demonstrate its potential, this paper presents a case study for medical training of breast cancer palpation. A real breast model made of soft silicone is augmented with a virtual tumor rendered inside. Haptic stimuli for the virtual tumor are generated based on a contact dynamics model identified via real measurements, without the need of geometric information on the breast. A subjective evaluation confirmed the realism and fidelity of our palpation system.

In haptic augmented reality (AR), a stimulus from a real object is augmented by a synthetic haptic signal. We previously developed a haptic AR system wherein the stiffness of a real object can be augmented with the aid of a haptic interface. This demonstration presents a case study of this technology for medical training of breast cancer palpation. A real breast model made of soft silicone is augmented with a virtual tumor rendered inside. Haptic stimuli for the virtual tumor are generated based on a contact dynamics model identified via real measurements. With our palpation system, a user can experience excellent realism comparable to that of a real breast mock-up containing a real tumor.

Visual delays are unavoidable in augmented reality setups and occur in different steps of the rendering pipeline. In the context of haptic interaction with virtual objects, it has been shown that delayed force feedback can alter the perception of object stiffness. We hypothesize that delays in augmented reality systems can have similar consequences. To test this, we carried out a user study to investigate the effect of visual and haptic delays on the perception of stiffness. The experiment has been performed in an optimized visuo-haptic augmented reality setup, which allows to artificially manipulate delays during visual and haptic rendering. In line with previous results, delays for haptic feedback resulted in decreased perceived stiffness. In contrast, visual delays caused an increase in perceived stiffness. However, the simultaneous occurrence of delays in both sensory channels led to a partial compensation of these effects. This could potentially help to correct stiffness perception of virtual objects in visuo-haptic augmented reality systems.

In our current research we examine the application of visuo-haptic augmented reality setups in medical training. To this end, highly accurate calibration, system stability, and low latency are indispensable prerequisites. These are necessary to maintain user immersion and avoid breaks in presence which potentially diminish the training outcome. In this paper we describe the developed calibration methods for visuo-haptic integration, the hybrid tracking technique for stable alignment of the augmentation, and the distributed framework ensuring low latency and component synchronization. Finally, we outline an early prototype system based on the multimodal augmented reality framework. The latter allows colocated visuo-haptic interaction with real and virtual scene components in a simplified open surgery setting.

The driving force of our research is the precise combination of real and - possibly indistinguishable - virtual objects in an interactive augmented reality environment. This requires real-time, multimodal simulation, as well as stable and accurate overlay of the computer-generated objects. This paper describes several methods to improve accuracy and stability of our hybrid augmented reality system. In a comparison of two approaches to hybrid head pose refinement, we show that Quasi-Newton method enables high performance optimization for image space error minimization. Moreover, a 3D landmark refinement step is proposed, which significantly improves quality and robustness of the overlay process. The enhanced system is demonstrated in an interactive AR environment, which provides accurate haptic feedback from real and virtual deformable objects. Finally, the effect of landmark occlusion on tracking stability during user interaction is also analyzed.