We present a real-time projection mapping system for visualizing information and displaying user interfaces on uneven, deformable surfaces in dynamic environments, where the surface gets partially and dynamically occluded. An important application area is the operating room, where this technology would allow for projection onto the surgical drapes. To achieve precise and adaptive geometric correction in setups with multiple projectors and overlapping, partially occluded projection regions, we adapt a point cloud rendering technique that accurately and efficiently reconstructs surface geometry in the projectors’ image space. This enables an overlap precision of 1.6 mm at a projection distance of 2 m, even on uneven surfaces. In addition, we propose two novel GPU-based blur mitigation methods that address blur caused by inevitable inaccuracies of the depth sensors and the overlapping projector images. A user study (n = 23) shows that our blur mitigation strategies significantly enhance perceived readability, reduce visual artifacts, and lower user workload compared to conventional multi-projector blending. Our system supports projection on arbitrary surfaces, without requiring explicit segmentation and is well-suited to meet the demands of sensitive environments, including those with sterility constraints or limited display access.

Optimal illumination of the surgical site is crucial for successful surgeries. Current lighting systems, however, suffer from significant drawbacks, particularly shadows cast by surgeons and operating room personnel. We introduce an innovative, module-based lighting system that actively prevents shadows using an array of swiveling, ceiling-mounted light modules. The intensity and orientation of these modules are autonomously controlled by novel algorithms utilizing multiple depth sensors mounted above the operating table. This article presents our complete system, detailing the algorithms for autonomous control and the initial optimization of the light module setup. Unlike prior work that was largely conceptual and based on simulations, this study introduces a real prototype featuring 56 light modules and three depth sensors. We evaluate this prototype through measurements, semi-structured interviews ( n \(=\) 4), and an extensive quantitative user study ( n \(=\) 11). The evaluation focuses on illumination quality, shadow elimination, and suitability for open surgeries compared to conventional OR lights. Our results demonstrate that the novel lighting system and optimization algorithms outperform conventional OR lights for abdominal surgeries, according to both objective measures and subjective ratings by surgeons.

Indirect vision through a mirror, while bi-manually manipulating both the mirror and another tool is a relatively common way to perform operations in various types of surgery. However, learning such psychomotor skills requires extensive training; they are difficult to teach; and they can be quite costly, for instance, for dentistry schools. In order to study the effectiveness of VR simulators for learning these kinds of skills, we developed a simulator for training dental surgery procedures, which supports tracking of eye gaze and tool trajectories (mirror and drill), as well as automated outcome scoring. We carried out a pre-/post-test study in which 30 fifth-year dental students received six training sessions in the access opening stage of the root canal procedure using the simulator. In addition, six experts performed three trials using the simulator. The outcomes of drilling performed on realistic plastic teeth showed a significant learning effect due to the training sessions. Also, students with larger improvements in the simulator tended to improve more in the real-world tests. Analysis of the tracking data revealed novel relationships between several metrics w.r.t. eye gaze and mirror use, and performance and learning effectiveness: high rates of correct mirror placement during active drilling and high continuity of fixation on the tooth are associated with increased skills and increased learning effectiveness. Larger time allocation for tooth inspections using the mirror, i.e., indirect vision, and frequency of inspection are associated with increased learning effectiveness. Our findings suggest that eye tracking can provide valuable insights into student learning gains of bi-manual psychomotor skills, particularly in indirect vision environments.

Collecting training data for pose estimation methods on images is a time-consuming task and usually involves some kind of manual labeling of the 6D pose of objects. This time could be reduced considerably by using marker-based tracking that would allow for automatic labeling of training images. However, images containing markers may reduce the accuracy of pose estimation due to a bias introduced by the markers. In this paper, we analyze the influence of markers in training images on pose estimation accuracy. We investigate the accuracy of estimated poses for three different cases: i) training on images with markers, ii) removing markers by inpainting, and iii) augmenting the dataset with randomly generated markers to reduce spatial learning of marker features. Our results demonstrate that utilizing marker-based techniques is an effective strategy for collecting large amounts of ground truth data for pose prediction. Moreover, our findings suggest that the usage of inpainting techniques do not reduce prediction accuracy. Additionally, we investigate the effect of inaccuracies of labeling in training data on prediction accuracy. We show that the precise ground truth data obtained through marker tracking proves to be superior compared to markerless datasets if labeling errors of 6D ground truth exist. Our data generation tools are available online: https://github.com/JHRosskamp/6DPoseDataGenTools

Abstract We propose a novel method to avoid stereo window violations at screen borders. These occur for objects in front of the zero parallax plane, which appear in front of the (physical) screen, and that are clipped for one eye while still being visible for the other eye. This contradicts other stereo cues, particularly disparity, potentially resulting in eye strain and simulator sickness. In interactive and dynamic virtual environments, where the user controls the camera, for example, via head tracking, it is impossible to avoid stereo window violations completely. We propose Dynparity, a novel rendering method to eliminate the conflict between clipping and negative disparity, by introducing a nonuniform stereoscopic projection. For each vertex in front of the zero parallax plane, we compute the stereoscopic projection such that the parallax approaches zero toward the edge of the screen. Our approach works entirely on the GPU in real‐time and can be easily included in modern game engines. We conducted a user study comparing our method to the standard stereo projection on a large‐screen stereo wall with head tracking. Our results show significantly reduced simulator sickness when using Dynparity compared to the standard stereo rendering.

Abstract Registration is an essential prerequisite for many applications when a multiple-camera setup is used. Due to the noise in depth images, registration procedures for depth sensors frequently rely on the detection of a target object in color or infrared images. However, this prohibits use cases where color and infrared images are not available or where there is no mapping between the pixels of different image types, e.g., due to separate sensors or different projections. We present our novel registration method that requires only the point cloud resulting from the depth image of each camera. For feature detection, we propose a combination of a custom-designed 3D registration target and an algorithm that is able to reliably detect that target and its features in noisy point clouds. Our evaluation indicates that our lattice detection is very robust (with a precision of more than 0.99) and very fast (on average about 20 ms with a single core). We have also compared our registration method with known methods: Our registration method achieves an accuracy of 1.6 mm at a distance of 2 m using only the noisy depth image, while the most accurate registration method achieves an accuracy of 0.7 mm requiring both the infrared and depth image.

Abstract We present a novel approach of redirected walking (RDW) based on step feedback sounds to redirect users in virtual reality. The main idea is to achieve path manipulation by changing step noises to deviate the users, who still believe that they are walking a straight line. Our approach can be combined with traditional visual approaches for RDW based on eye-blinking. Moreover, we have conducted a user study in a large area ( $$10\times 20$$ 10 × 20 m) using a within-subject design. We achieved a translational redirection of 1.7m in average with pure audio feedback. Moreover, our results show that visual methods can amplify the deviation of our new auditory approach by 80cm in average at the distance of 20 m.

The precise registration between multiple depth sensors is a crucial prerequisite for many applications. Previous techniques frequently rely on RGB or IR images and checkerboard targets for feature detection. However, this prohibits the usage for use-cases where neither is available or where IR and depth images have different projections. Therefore, we present a novel registration approach that uses depth data exclusively for feature detection, making it more universally applicable while still achieving robust and precise results. We propose a combination of a custom 3D registration target — a lattice with regularly-spaced holes — and a feature detection algorithm that is able to reliably extract the lattice and its features from noisy depth images. In addition, we have integrated the registration procedure to a publicly available Unreal Engine 4 plugin that allows multiple point clouds captured by several depth cameras to be registered in a virtual environment. Despite the rather noisy depth images, we are able to quickly obtain a robust registration that yields an average deviation of 3.8 mm to 4.4 mm in our test scenarios.

Abstract Advances in computer technology and increasing usage of computer graphics in a broad field of applications lead to rapidly rising demands regarding size and detail of virtual landscapes. Manually creating huge, realistic looking terrains and populating them densely with assets is an expensive and laborious task. In consequence, (semi-)automatic procedural terrain generation is a popular method to reduce the amount of manual work. However, such methods are usually highly specialized for certain terrain types and especially the procedural generation of landscapes composed of different biomes is a scarcely explored topic. We present a novel system, called AutoBiomes, which is capable of efficiently creating vast terrains with plausible biome distributions and therefore different spatial characteristics. The main idea is to combine several synthetic procedural terrain generation techniques with digital elevation models (DEMs) and a simplified climate simulation. Moreover, we include an easy-to-use asset placement component which creates complex multi-object distributions. Our system relies on a pipeline approach with a major focus on usability. Our results show that our system allows the fast creation of realistic looking terrains.

We present a novel data structure for SIMD optimized simultaneous bounding volume hierarchy (BVH) traversals like they appear for instance in collision detection tasks. In contrast to all previous approaches, we consider both the traversal algorithm and the construction of the BVH. The main idea is to increase the branching factor of the BVH according to the available SIMD registers and parallelize the simultaneous BVH traversal using SIMD operations. This requires a novel BVH construction method because traditional BVHs for collision detection usually are simple binary trees. To do that, we present a new BVH construction method based on a clustering algorithm, Batch Neural Gas, that is able to build efficient n-ary tree structures along with SIMD optimized simultaneous BVH traversal. Our results show that our new data structure outperforms binary trees significantly.

We present a study investigating the question whether and how people's intention to change their environmental behavior depends on the degrees of immersion and freedom of navigation when they experience a virtual coral reef. The most striking result is, perhaps, that the highest level of immersion combined with the highest level of navigation did not lead to the highest intentions to change behavior.

We present a novel haptic rendering approach to simulate material removal in medical simulations at haptic rates. The core of our method is a new massively-parallel continuous collision detection algorithm in combination with a stable and flexible 6-DOF collision response scheme that combines penalty-based and constraint-based force computation.

We present a novel algorithm, called AstroGen, to procedurally generate highly detailed and realistic 3D meshes of small celestial bodies automatically. AstroGen gains it's realism from learning surface details from real world asteroid data. We use a sphere packing-based metaball approach to represent the rough shape and a set of noise functions for the surface details. The main idea is to apply an optimization algorithm to adopt these representations to available highly detailed asteroid models with respect to a similarity measure. Our results show that our approach is able to generate a wide variety of different celestial bodies with very complex surface structures like caves and craters.

We present the first VR training simulator for hip replacement surgeries. We solved the main challenges of this task - high and stable forces during the milling process while simultaneously a very sensitive feedback is required - by using an industrial robot for the force output and the development of a novel massively parallel haptic rendering algorithm with support for material removal.

We present the first VR training simulator for hip replacement surgeries. We solved the main challenges of this task - high and stable forces during the milling process while simultaneously a very sensitive feedback is required - by using an industrial robot for the force output and the development of a novel massively parallel haptic rendering algorithm with support for material removal.

We present a novel statistical shape descriptor for arbitrary three-dimensional shapes as a six-dimensional feature for generic classification purposes. Our feature parameterizes the complete geo-metrical relation of the global shape and additionally considers local dissimilarities while being invariant to the shape appearance. Our approach allows the classification of large-scale shapes with only small local dissimilarities. Our feature can be easily quantized and mapped into a histogram, which can be used for efficient and effective classification. We take advantage of GPU processing in order to efficiently compute our invariant local shape descriptor feature even for large-scale shapes. Our synthetic benchmarks show that our approach outperforms state-of-the-art methods for local shape dissimilarity classification. In general, it yields robust and promising recognition rates even for noisy data.

We present a novel approach for approximating objective functions in arbitrary deterministic and stochastic multi-objective blackbox simulations. Usually, simulated-based optimization approaches require pre-defined objective functions for optimization techniques in order to find a local or global minimum of the specified simulation objectives and multi-objective constraints. Due to the increasing complexity of state-of-the-art simulations, such objective functions are not always available, leading to so-called blackbox simulations.

We define a novel geometric predicate and a class of objects that enables us to prove a linear bound on the number of intersecting polygon pairs for colliding 3D objects in that class. Our predicate is relevant both in theory and in practice: it is easy to check and it needs to consider only the geometric properties of the individual objects – it does not depend on the configuration of a given pair of objects. In addition, it characterizes a practically relevant class of objects: we checked our predicate on a large database of real-world 3D objects and the results show that it holds for all but the most pathological ones. Our proof is constructive in that it is the basis for a novel collision detection algorithm that realizes this linear complexity also in practice. Additionally, we present a parallelization of this algorithm with a worst-case running time that is independent of the number of polygons. Our algorithm is very well suited not only for rigid but also for deformable and even topology-changing objects, because it does not require any complex data structures or pre-processing. We have implemented our algorithm on the GPU and the results show that it is able to find in real-time all colliding polygons for pairs of deformable objects consisting of more than 200k triangles, including self-collisions.

We present a new graph-based approach called GraphPool for the generation, management and distribution of simulation states for 3D simulation applications. Currently, relational databases are often used for this task in simulation applications. In contrast, our approach combines novel wait-free nested hash map techniques with traditional graphs which results in a schema-less, in-memory, highly efficient data management. Our GraphPool stores static and dynamic parts of a simulation model, distributes changes caused by the simulation and logs the simulation run. Even more, the GraphPool supports sophisticated query types of traditional relational databases. As a consequence, our GraphPool overcomes the associated drawbacks of relational database technology for sophisticated 3D simulation applications. Our GraphPool has several advantages compared to other state-of-the-art decentralized methods, such as persistence for simulation state over time, object identification, standardized interfaces for software components as well as a consistent world model for the overall simulation system. We tested our approach in a synthetic benchmark scenario but also in real-world use cases. The results show that it outperforms state-of-the-art relational databases by several orders of magnitude.

We present a novel knowledge discovery approach for automatic feasible design space approximation and parameter optimization in arbitrary multiobjective blackbox simulations. Our approach does not need any supervision of simulation experts. Usually simulation experts conduct simulation experiments for a predetermined system specification by manually reducing the complexity and number of simulation runs by varying input parameters through educated assumptions and according to prior defined goals. This leads to a error-prone trial-and-error approach for determining suitable parameters for successful simulations.In contrast, our approach autonomously discovers unknown relationships in model behavior and approximates the feasible design space. Furthermore, we show how Pareto gradient information can be obtained from this design space approximation for state-of-the-art optimization algorithms. Our approach gains its efficiency from a novel spline-based sampling of the parameter space in combination within novel forest-based simulation dataflow analysis. We have applied our new method to several artificial and real-world scenarios and the results show that our approach is able to discover relationships between parameters and simulation goals. Additionally, the computed multiobjective solutions are close to the Pareto front.

Filling objects densely with sets of non overlapping spheres has been investigated for centuries. Once started as a pure intellectual challenge, today, sphere packings have diverse applications in a wide spectrum of scientific and engineering disciplines, for example in automated radiosurgical treatment planning, investigation of processes such as sedimentation, compaction and sintering, in powder metallurgy for three-dimensional laser cutting, in cutting different natural crystals, the discrete element method is based on them, and so forth.

We present a benchmarking suite for rigid object collision detection and collision response schemes. The proposed benchmarking suite can evaluate both the performance as well as the quality of the collision response. The former is achieved by densely sampling the configuration space of a large number of highly detailed objects; the latter is achieved by a novel methodology that comprises a number of models for certain collision scenarios. With these models, we compare the force and torque signals both in direction and magnitude.