In virtual reality (VR), using the hands instead of controllers has enormous potential for intuitive user interactions, e.g., direct grasping, touching, or moving virtual objects. In this context, the realism of the hands plays an important role in the body ownership illusion and the user’s feeling of presence. Yet, most applications use generic 3D hand models, so they do not include individual user characteristics, like wrinkles, moles, or skin features. In this work, we present a pipeline to include fully articulated, photorealistic, fully tracked hand scans for VR applications. Our pipeline captures photos of the hand from different perspectives simultaneously using a customized hand scanner. Using open-source photogrammetry software, a rigged 3D model of the hand is created and integrated into a Unity application. At last, we provide practitioners and researchers with a fast and easy way to integrate a user’s real hands into a VR application. Finally, we present guidelines for generating a fully articulated, personalized virtual hand model within 30 minutes from a set of 2D images.

Uncertainty cues are inherent in natural human interaction, as they signal to communication partners how much they can rely on conveyed information. Humans subconsciously provide such signals both verbally (e.g., through expressions such as “maybe’’ or “I think’’) and non-verbally (e.g., by diverting their gaze). In contrast, artificial intelligence (AI)-based services and machine learning (ML) models such as ChatGPT usually do not disclose the reliability of answers to their users.

Real-time detection of saccades is of major interest for many applications in human-computer interaction and mixed reality. However, due to relatively low update rates and high latencies of current commercially available eye trackers, gaze events are typically detected after they occur with some delay. This limits interaction scenarios such as intent-based gaze interaction, redirected walking, or gaze forecasting.

Recent advancements in Neural Radiance Fields (NeRF) provide enormous potential for a wide range of Mixed Reality (MR) applications. However, the applicability of NeRF to real-time MR systems is still largely limited by the rendering performance of NeRF. In this paper, we present a novel approach for Variable Rate Shading for Neural Radiance Fields (VRS-NeRF). In contrast to previous techniques, our approach does not require training multiple neural networks or re-training of already existing ones, but instead utilizes the raytracing properties of NeRF. This is achieved by merging rays depending on a variable shading rate, which reduces the overall number of queries to the neural network. We demonstrate the generalizability of our approach by implementing three alternative functions for the determination of the shading rate. The first method uses the gaze of users to effectively implement a foveated rendering technique in NeRF. For the other two techniques, we utilize shading rates based on edges and saliency. Based on a psychophysical experiment and multiple image-based metrics, we suggest a set of parameters for each technique, yielding an optimal tradeoff between rendering performance gain and perceived visual quality.

We present a novel view on gaze event classification by redefining it as a time-to-event problem. In contrast to previous models, which consider the classification as discrete events, our redefinition allows for estimating the remaining time until the next saccade event. Therefore, we provide a feature analysis and an initial solution for compensating the latency of wearable eye-trackers build in today’s head-mounted displays. Similar to previous classifiers, we utilize oculomotor features such as velocity, acceleration, and event durations. In total, we analyze 104 extracted features of three datasets and apply different regression methods. We identify optimal window sizes for each feature and extract the importance of all extracted windows using recursive feature elimination. Afterwards, we evaluate the performance of all regressors using earlier selected features. We show that our selected regressors can predict the time-to-event better than the baseline, indicating the potential usage of time-to-event prediction of saccades.