Achieving temporal synchrony between sensory modalities is crucial for natural perception of object interaction in virtual reality (VR). While subjective questionnaires are currently used to evaluate users’ VR experiences, leveraging behavior and psychophysiological responses can provide additional insights. We investigated motion and ocular behavior as discriminators between realistic and unrealistic object interactions. Participants grasped and placed a virtual object while experiencing sensory feedback that either matched their expectations or occurred too early. We also explored visual-only feedback vs. combined visual and haptic feedback. Due to technological limitations, a condition with delayed feedback was added post hoc . Gaze-based metrics revealed discrimination between high- and low-feedback realism. Increased interaction uncertainty was associated with longer fixations on the avatar hand and temporal shifts in the gaze–action relationship. Our findings enable real-time evaluation of users’ perception of realism in interactions. They facilitate the optimization of interaction realism in virtual environments and beyond.

OBJECTIVE: Mixed-Reality (XR) technologies promise a user experience (UX) that rivals the interactive experience with the real-world. The key facilitators in the design of such a natural UX are that the interaction has zero lag and that users experience no excess mental load. This is difficult to achieve due to technical constraints such as motion-to-photon latency as well as false-positives during gesture-based interaction. METHODS: In this paper, we explored the use of physiological features to model the user's intent to interact with a virtual reality (VR) environment. Accurate predictions about when users want to express an interaction intent could overcome the limitations of an interactive device that lags behind the intention of a user. We computed time-domain features from electroencephalography (EEG) and electromyography (EMG) recordings during a grab-and-drop task in VR and cross-validated a Linear Discriminant Analysis (LDA) for three different combinations of (1) EEG, (2) EMG and (3) EEG-EMG features. RESULTS & CONCLUSION: We found the classifiers to detect the presence of a pre-movement state from background idle activity reflecting the users' intent to interact with the virtual objects (EEG: 62 % ± 10 %, EMG: 72 % ± 9 %, EEG-EMG: 69 % ± 10 %) above simulated chance level. The features leveraged in our classification scheme have a low computational cost and are especially useful for fast decoding of users' mental states. Our work is a further step towards a useful classification of users' intent to interact, as a high temporal resolution and speed of detection is crucial. This facilitates natural experiences through zero-lag adaptive interfaces.

During listening to music, the brain expects specific acoustic events based on learned musical rules. During music performance expectancy is additionally created based on motor action by linking keypresses to their sounds. We investigated EEG (Electroencephalography) signals to auditory expectancy violations in piano performance and perception. In our study, pianists experienced manipulations of different acoustic features, such as pitch and loudness, during playing and listening to piano sequences. We found that manipulations during performance elicited deflections with stronger amplitudes compared to manipulations during perception indicating that the action of producing sounds strengthens auditory expectancy. Loudness manipulations, violating musical regularity, elicited deflections with smaller latencies compared to pitch manipulations, which violate harmonic expectancy, suggesting that the brain processes expectancy violations of distinct acoustic features in a different way. These EEG signatures may prove useful for applications in intelligent music interfaces by providing information about sensory-motor accuracy.

Designing immersion is the key challenge in virtual reality; this challenge has driven advancements in displays, rendering and recently, haptics. To increase our sense of physical immersion, for instance, vibrotactile gloves render the sense of touching, while electrical muscle stimulation (EMS) renders forces. Unfortunately, the established metric to assess the effectiveness of haptic devices relies on the user's subjective interpretation of unspecific, yet standardized, questions.

When human operators locomote actively in virtual environments (VE), the movement range often has to be adapted to the limited dimensions of the physical space. This however might lead to a conflict between sensory information originating from user movements and sensory feedback provided through the virtual locomotion. To investigate whether different locomotion strategies that adapt virtual movement to the limited physical space impact cognitive processes, two experiments were conducted. The first experiment used walking in place, the second study scale of locomotion to investigate the impact of locomotion adaptation on the acquisition of spatial knowledge and user experience. We systematically analyzed body-based sensorial conflicts for the different adaptation strategies and reveal that neither walking in place nor scale of locomotion impacts spatial knowledge acquisition or user experience. We can conclude that visual cues indicating locomotion combined with body-based rotational cues seem to be sufficient for the acquisition of spatial knowledge and that locomotion with controllers seems efficient and preferable for users. The results link system-driven typologies with human-centered factors to guide systematic tests of locomotion techniques in virtual environments for future studies.

Passive brain-computer interfacing allows computer systems direct access to aspects of their user's cognition. In essence, a computer system can gain information about its user without this user needing to explicitly communicate it. Based on this information, human-computer interaction can be made more symmetrical, solving an age-old but still fundamental problem of present-day interaction techniques. For practical real-world application of this technology, it is important that cognitive states can be identified accurately and efficiently. Here we present preliminary data demonstrating it is possible to calibrate a task-independent classifier to identify when a user is under heavy workload across different activities. We used different types of mental arithmetic and even a semantic task. Task-independent classification is an important step towards real-world practical application of this technology.

The study of brain-computer interfaces (BCIs) has undergone 30 years of intense development and has grown into a rich and diverse field. BCIs are technologies that enable direct communication between the human brain and external devices. Conventionally, wet electrodes have been employed to obtain unprecedented sensitivity to high-temporal-resolution brain activity; recently, the growing availability of various sensors that can be used to detect high-quality brain signals in a wide range of clinical and everyday environments is being exploited. This development of biosensing neurotechnologies and the desire to implement them in real-world applications have led to the opportunity to develop augmented BCIs (ABCIs) in the upcoming decades. An ABCI is similar to a BCI in that it relies on biosensors that record signals from the brain in everyday environments; the signals are then processed in real time to monitor the behavior of the human. To use an ABCI as a mobile brain imaging technique for everyday, real-life applications, the sensors and the corresponding device must be lightweight and the equipment response time must be short. This study presents an overview of the wide range of biosensor approaches currently being applied to ABCIs, from their use in the laboratory to their application in clinical and everyday use. The basic principles of each technique are described along with examples of current applications of cutting-edge neuroscience research. In summary, we show that ABCI techniques continue to grow and evolve, incorporating new technologies and advances to address ever more complex and important neuroscience issues, with advancements that are envisioned to lead to a wide range of real-life applications.

The aim of this study is to investigate the difference of EEG dynamics on navigation performance. A tunnel task was designed to classify subjects into allocentric or egocentric spatial representation users. Despite of the differences of mental spatial representation, behavioral performance in general were compatible between the two strategies subjects in the tunnel task. Task-related EEG dynamics in power changes were analyzed using independent component analysis (ICA), time-frequency and non-parametric statistic test. ERSP image results revealed navigation performance-predictive EEG activities which is is expressed in the parietal component by source reconstruction. For egocentric subjects, comparing to trails with well-estimation of homing angle, the power attenuation at the frequencies from 8 to 30 Hz (around alpha and beta band) was stronger when subjects overestimated homing directions, but the attenuated power was decreased when subjects were underestimated the homing angles. However, we did not found performance related brain activities for allocentric subjects, which may due to the functional dissociation between the use of allo- and egocentric reference frames.

The performance of EEG-based Brain-Computer-Interfaces (BCIs) critically depends on the extraction of features from the EEG carrying information relevant for the classification of different mental states. For BCIs employing imaginary movements of different limbs, the method of Common Spatial Patterns (CSP) has been shown to achieve excellent classification results. The CSP-algorithm however suffers from a lack of robustness, requiring training data without artifacts for good performance. To overcome this lack of robustness, we propose an adaptive spatial filter that replaces the training data in the CSP approach by a-priori information. More specifically, we design an adaptive spatial filter that maximizes the ratio of the variance of the electric field originating in a predefined region of interest (ROI) and the overall variance of the measured EEG. Since it is known that the component of the EEG used for discriminating imaginary movements originates in the motor cortex, we design two adaptive spatial filters with the ROIs centered in the hand areas of the left and right motor cortex. We then use these to classify EEG data recorded during imaginary movements of the right and left hand of three subjects, and show that the adaptive spatial filters outperform the CSP-algorithm, enabling classification rates of up to 94.7 % without artifact rejection.