In this paper, we introduce a flexible framework that can facilitate the definition of 3D gesture-based interfaces. Highlighting the need for context awareness in complex domains, such as the operating room, we argue how the proposed architecture can overcome integration challenges. Through a real-life scenario, an intra-operative medical image viewer, we demonstrate how the proposed framework can be used in practice to define user interfaces in collaborative environments, where the behavior and the system response can be adapted based on the current workflow stage and individual user requirements. Finally, we demonstrate how the defined interface can be manipulated using a high-level visual programming interface. The extensibility of the proposed architecture makes it applicable to a wide range of scenarios.

Computerized medical systems play a vital role in the operating room, however, sterility requirements and interventional workflow often make interaction with these devices challenging for surgeons. Typical solutions, such as delegating physical control of keyboard and mouse to assistants, add an undesirable level of indirection. We present a touchless, gesture-based interaction framework for the operating room that lets surgeons define a personalized set of gestures for controlling arbitrary medical computerized systems. Instead of using cameras for capturing gestures, we rely on a few wireless inertial sensors, placed on the arms of the surgeon, eliminating the dependence on illumination and line-of-sight. A discriminative gesture recognition approach based on kernel regression allows us to simultaneously classify performed gestures and to track the relative spatial pose within each gesture, giving surgeons fine-grained control of continuous parameters. An extensible software architecture enables a dynamic association of learned gestures to arbitrary intraoperative computerized systems. Our experiments illustrate the performance of our approach and encourage its practical applicability.

For the past two decades, medical Augmented Reality visualization has been researched and prototype systems have been tested in laboratory setups and limited clinical trials. Up to our knowledge, until now, no commercial system incorporating Augmented Reality visualization has been developed and used routinely within the real-life surgical environment. In this paper, we are reporting on observations and analysis concerning the usage of a commercially developed and clinically approved Freehand SPECT system, which incorporates monitor-based Mixed Reality visualization, during real-life surgeries. The workflow-based analysis we present is focused on an atomic sub-task of sentinel lymph node biopsy. We analyzed the usage of the Augmented and Virtual Reality visualization modes by the surgical team, while leaving the staff completely uninfluenced and unbiased in order to capture the natural interaction with the system. We report on our observations in over 100 Freehand SPECT acquisitions within different phases of 52 surgeries.

We propose a novel algorithm for incrementally reconstructing the visual hull in dynamic scenes by exploiting temporal consistency. Using the difference in the silhouette images between two frames we can efficiently locate the scene parts that have to be updated using ray casting. By only concentrating on the parts of the scene that have to be checked for changes, we achieve a significant speed up over traditional methods, which reconstruct each frame independently. Our method does not use any kind of scene model and works regardless of the number and shape of the objects in the scene. We test our algorithm on sequences taken with a multi-camera system and perform a detailed performance analysis showing that our method outperforms other existing methods.