Redirected walking (RDW) alters the relationship between physical locomotion and visual feedback to guide users along virtual paths that vary from their real-world trajectories. Over time these changes can lead to adaptation of the user and produce novel sensorimotor settings that users might retain across sessions and apply directly once they enter virtual reality (VR). Although short-term adaptation to altered sensorimotor contingencies is well established, it remains unclear whether such adaptation is retained across multiple days. We investigated long-term retention of RDW adaptation by repeatedly exposing ten participants to a fixed rightward curvature gain of π /30 across nine sessions over two weeks. Each session consisted of 200 walking repetitions with gain applied. Adaptation was assessed before and after each session using blind walking and a pointing task. Moreover, perceptual detection thresholds for curvature gains were measured before the first session (Baseline), a day after the ninth session (Final), and once in-between before the 5th adaptation session. Results showed clear retention of adapted locomotor behavior: during blind walking at the beginning of the sessions, when instructed to walk straight, participants consistently exhibited curved trajectories, which indicates that newly acquired sensorimotor contingencies can be retained over days and are immediately available upon re-entering VR. At the same time, pointing accuracy remained stable throughout the experiment and detection thresholds showed no consistent changes across sessions. In summary, our study provides evidence that adaptation to altered sensorimotor contingencies in VR can be retained across multiple sessions and days and can be available as soon as the user enters VR. This may be useful for many scenarios in which users repeatedly use VR tools over a long period of time. The complete data set, all supplemental materials and the preprint of the manuscript are available at https://osf.io/z973w.

We present a gaze-based augmented reality control interface for electric wheelchairs, addressing the challenges faced by individuals with mobility impairments. The development transitions through three stages: model training with offline evaluation, Virtual Reality (VR) simulations, and physical deployment. First, we trained deep learning models, comparing Transformers and LSTMs, to predict locomotion intentions based on gaze data. While gaze predicts steering intentions well, it sometimes diverges from locomotion goals. To tackle this, we classify gaze movements as either indicative of locomotor intention or not. This novel approach addresses the Midas Touch Problem of gaze. Datasets were collected in controlled VR environments featuring different tasks. We find that data sets with tasks that encouraged diverse navigation and gaze behaviors enable strong generalization. The online VR simulation evaluation phase enabled safe and immersive testing, allowing the assessment of system performance and the integration of feedback for user guidance. Our approach provided smoother navigation control compared to traditional “Where-You-Look-Is-Where-You-Go” methods. Feedback improved user ratings of the system. In the final stage, the system was deployed on a physical wheelchair equipped with an augmented reality (AR) device to provide feedback about the predictions to the user, allowing real-world evaluation. Despite differences in user behavior between VR and physical environments, the system successfully translated gaze inputs into precise and safe navigation commands. Users were able to steer the wheelchair solely using their eyes while simultaneously being able to look at destinations at the side of the path.

In current computational models on oculomotor learning 'the' movement vector is adapted in response to targeting errors. However, for saccadic eye movements, learning exhibits a spatially distributive nature, i.e. it transfers to surrounding positions. This adaptation field resembles the topographic maps of visual and motor activity in the brain and suggests that learning does not act on the population vector but already on the level of the 2D population response. Here we present a population-based gain field model for saccade adaptation in which sensorimotor transformations are implemented as error-sensitive gain field maps that modulate the population response of visual and motor signals and of the internal saccade estimate based on corollary discharge (CD). We fit the model to saccades and visual target localizations across adaptation, showing that adaptation and its spatial transfer can be explained by locally distributive learning that operates on visual, motor and CD gain field maps. We show that 1) the scaled locality of the adaptation field is explained by population coding, 2) its radial shape is explained by error encoding in polar-angle coordinates, and 3) its asymmetry is explained by an asymmetric shape of learning rates along the amplitude dimension. Learning exhibits the highest peak rate, the widest spatial extension and a pronounced asymmetry in the motor domain, while in the visual and the internal saccade domain learning appears more localized. Moreover, our results suggest that the CD-based internal saccade representation has a response field that monitors only part of the ongoing saccade changes during learning. Our framework opens the door to study spatial generalization and interference of learning in multiple contexts.

Predicting future locomotion based on intrinsic data serves many purposes, including optimizing the utilization of physical space in virtual reality environments and enhancing the control of electronic aids for patients with motor impairments. However, predicting human locomotion intentions proves challenging due to the inherent difficulty arising from the highly complex and nonlinear interactions among the relevant parameters. Deep neural networks offer a significant advantage over conventional approaches in addressing this challenge. We treat this task as a time series prediction problem and compare LSTM networks to transformer models. A distinctive aspect of our work is our approach’s emphasis on eye movements as a central feature, contributing to its novel predictive capabilities. Besides gaze data, we evaluate the addition of EEG as a data source for this prediction task to be used in brain-computer interfaces. To achieve this, we conducted two data collection experiments in custom virtual environments that feature different tasks utilizing joystick control. We present these novel datasets in conjunction with this work. The results demonstrate that gaze data proves to be a valuable tool for locomotion prediction in different contexts, even when there is not a strong and direct connection between gaze and future waypoints. Transformer models were able to achieve better performance than LSTM networks, and we conclude that successful prediction across diverse situations requires datasets containing a wide range of movement scenarios.

The cerebellum implements error-based motor learning via synaptic gain adaptation of an inverse model, i.e. the mapping of a spatial movement goal onto a motor command. Recently, we modeled the motor and perceptual changes during learning of saccadic eye movements, showing that learning is actually a threefold process. Besides motor recalibration of (1) the inverse model, learning also comprises perceptual recalibration of (2) the visuospatial target map and (3) of a forward dynamics model that estimates the saccade size from corollary discharge. Yet, the site of perceptual recalibration remains unclear. Here we dissociate cerebellar contributions to the three stages of learning by modeling the learning data of eight cerebellar patients and eight healthy controls. Results showed that cerebellar pathology restrains short-term recalibration of the inverse model while the forward dynamics model is well informed about the reduced saccade change. Adaptation of the visuospatial target map trended in learning direction only in control subjects, yet without reaching significance. Moreover, some patients showed a tendency for uncompensated oculomotor fatigue caused by insufficient upregulation of saccade duration. According to our model, this could induce long-term perceptual compensation, consistent with the overestimation of target eccentricity found in the patients' baseline data. We conclude that the cerebellum mediates short-term adaptation of the inverse model, especially by control of saccade duration, while the forward dynamics model was not affected by cerebellar pathology.

Virtual Reality (VR) allows users to perform natural movements such as hand movements, turning the head and natural walking in virtual environments. While such movements enable seamless natural interaction, they come with the need for a large tracking space, particularly in the case of walking. To optimise use of the available physical space, prediction models for upcoming behavior are helpful. In this study, we examined whether a user’s eye movements tracked by current VR hardware can improve such predictions. Eighteen participants walked through a virtual environment while performing different tasks, including walking in curved paths, avoiding or approaching objects, and conducting a search. The recorded position, orientation and eye-tracking features from 2.5 s segments of the data were used to train an LSTM model to predict the user’s position 2.5 s into the future. We found that future positions can be predicted with an average error of 65 cm. The benefit of eye movement data depended on the task and environment. In particular, situations with changes in walking speed benefited from the inclusion of eye data. We conclude that a model utilizing eye tracking data can improve VR applications in which path predictions are helpful.

Real walking is the most natural way to locomote in virtual reality (VR), but a confined physical walking space limits its applicability. Redirected walking (RDW) is a collection of techniques to solve this problem. One of these techniques aims to imperceptibly rotate the user's view of the virtual scene in order to steer her along a confined path whilst giving the impression of walking in a straight line in a large virtual space. Measurement of perceptual thresholds for the detection of such a modified curvature gain have indicated a radius that is still larger than most room sizes. Since the brain is an adaptive system and thresholds usually depend on previous stimulations, we tested if prolonged exposure to an immersive virtual environment (IVE) with increased curvature gain produces adaptation to that gain and modifies thresholds such that, over time, larger curvature gains can be applied for RDW. Therefore, participants first completed a measurement of their perceptual threshold for curvature gain. In a second session, the same participants were exposed to an IVE with a constant curvature gain in which they walked between two targets for about 20 minutes. Afterwards, their perceptual thresholds were measured again. The results show that the psychometric curves shifted after the exposure session and perceptual thresholds for increased curvature gain further increased. The increase of the detection threshold suggests that participants adapt to the manipulation and stronger curvature gains can be applied in RDW, and therefore improves its applicability in such situations.

Immersive computer-generated environments (aka virtual reality, VR ) are limited by the physical space around them, e.g., enabling natural walking in VR is only possible by perceptually-inspired locomotion techniques such as redirected walking (RDW). We introduce a completely new approach to imperceptible position and orientation redirection that takes advantage of the fact that even healthy humans are functionally blind for circa ten percent of the time under normal circumstances due to motor processes preventing light from reaching the retina (such as eye blinks) or perceptual processes suppressing degraded visual information (such as blink-induced suppression). During such periods of missing visual input, change blindness occurs, which denotes the inability to perceive a visual change such as the motion of an object or self-motion of the observer. We show that this phenomenon can be exploited in VR by synchronizing the computer graphics rendering system with the human visual processes for imperceptible camera movements, in particular to implement position and orientation redirection. We analyzed human sensitivity to such visual changes with detection thresholds, which revealed that commercial off-the-shelf eye trackers and head-mounted displays suffice to translate a user by circa 4 -- 9 cm and rotate the user by circa 2 -- 5 degrees in any direction, which could be accumulated each time the user blinks. Moreover, we show the potential for RDW, whose performance could be improved by approximately 50% when using our technique.

It is known for decades that users tend to significantly underestimate or overestimate distances or speed in immersive virtual environments (IVEs) compared to corresponding judgments in the real world. Although several factors have been identified in the past that could explain small portions of this effect, the main causes of these perceptual discrepancies still remain elusive. One of the factors that has received less attention in the literature is the amount of blur presented in the visual imagery, for example, when using a head-mounted display (HMD).

Virtual reality strives to provide a user with an experience of a simulated world that feels as natural as the real world. Yet, to induce this feeling, sometimes it becomes necessary for technical reasons to deviate from a one-to-one correspondence between the real and the virtual world, and to reorient or reposition the user's viewpoint. Ideally, users should not notice the change of the viewpoint to avoid breaks in perceptual continuity. Saccades, the fast eye movements that we make in order to switch gaze from one object to another, produce a visual discontinuity on the retina, but this is not perceived because the visual system suppresses perception during saccades. As a consequence, our perception fails to detect rotations of the visual scene during saccades. We investigated whether saccadic suppression of image displacement (SSID) can be used in an immersive virtual environment (VE) to unconsciously rotate and translate the observer's viewpoint. To do this, the scene changes have to be precisely time-locked to the saccade onset. We used electrooculography (EOG) for eye movement tracking and assessed the performance of two modified eye movement classification algorithms for the challenging task of online saccade detection that is fast enough for SSID. We investigated the sensitivity of participants to translations (forward/backward) and rotations (in the transverse plane) during trans-saccadic scene changes. We found that participants were unable to detect approximately ±0.5m translations along the line of gaze and ±5° rotations in the transverse plane during saccades with an amplitude of 15°. If the user stands still, our approach exploiting SSID thus provides the means to unconsciously change the user's virtual position and/or orientation. For future research and applications, exploiting SSID has the potential to improve existing redirected walking and change blindness techniques for unlimited navigation through arbitrarily-sized VEs by real walking.

One major benefit of wearable computers is that users can naturally move and explore computer-mediated realities. However, researchers often observe that users' space and motion perception severely differ in such environments compared to the real world, an effect that is often attributed to slight discrepancies in sensory cues, for instance, caused by tracking inaccuracy or system latency. This is particularly true for virtual reality (VR), but such conflicts are also inherent to augmented reality (AR) technologies. Although, head-worn displays will become more and more available soon, the effects on motion perception have rarely been studied, and techniques to modify self-motion in AR environments have not been leveraged so far. In this paper we introduce the concept of computer-mediated optic flow, and analyze its effects on self-motion perception in AR environments. First, we introduce different techniques to modify optic flow patterns and velocity. We present a psychophysical experiment which reveals differences in self-motion perception with a video see-through head-worn display compared to the real-world viewing condition. We show that computer-mediated optic flow has the potential to make a user perceive self-motion as faster or slower than it actually is, and we discuss its potential for future AR setups.

Computational modeling of the primate visual system yields insights of potential relevance to some of the challenges that computer vision is facing, such as object recognition and categorization, motion detection and activity recognition, or vision-based navigation and manipulation. This paper reviews some functional principles and structures that are generally thought to underlie the primate visual cortex, and attempts to extract biological principles that could further advance computer vision research. Organized for a computer vision audience, we present functional principles of the processing hierarchies present in the primate visual system considering recent discoveries in neurophysiology. The hierarchical processing in the primate visual system is characterized by a sequence of different levels of processing (on the order of 10) that constitute a deep hierarchy in contrast to the flat vision architectures predominantly used in today's mainstream computer vision. We hope that the functional description of the deep hierarchies realized in the primate visual system provides valuable insights for the design of computer vision algorithms, fostering increasingly productive interaction between biological and computer vision research.

Motion perception in immersive virtual environments significantly differs from the real world. For example, previous work has shown that users tend to underestimate travel distances in virtual environments (VEs). As a solution to this problem, researchers proposed to scale the mapped virtual camera motion relative to the tracked real-world movement of a user until real and virtual motion are perceived as equal, i.e., real-world movements could be mapped with a larger gain to the VE in order to compensate for the underestimation. However, introducing discrepancies between real and virtual motion can become a problem, in particular, due to misalignments of both worlds and distorted space cognition. In this paper, we describe a different approach that introduces apparent self-motion illusions by manipulating optic flow fields during movements in VEs. These manipulations can affect self-motion perception in VEs, but omit a quantitative discrepancy between real and virtual motions. In particular, we consider to which regions of the virtual view these apparent self-motion illusions can be applied, i.e., the ground plane or peripheral vision. Therefore, we introduce four illusions and show in experiments that optic flow manipulation can significantly affect users' self-motion judgments. Furthermore, we show that with such manipulations of optic flow fields the underestimation of travel distances can be compensated.

Head-mounted displays (HMDs) allow users to observe virtual environments (VEs) from an egocentric perspective. However, several experiments have provided evidence that egocentric distances are perceived as compressed in VEs relative to the real world. Recent experiments suggest that the virtual view frustum set for rendering the VE has an essential impact on the user's estimation of distances. In this article we analyze if distance estimation can be improved by calibrating the view frustum for a given HMD and user. Unfortunately, in an immersive virtual reality (VR) environment, a full per user calibration is not trivial and manual per user adjustment often leads to mini- or magnification of the scene. Therefore, we propose a novel per user calibration approach with optical see-through displays commonly used in augmented reality (AR). This calibration takes advantage of a geometric scheme based on 2D point - 3D line correspondences, which can be used intuitively by inexperienced users and requires less than a minute to complete. The required user interaction is based on taking aim at a distant target marker with a close marker, which ensures non-planar measurements covering a large area of the interaction space while also reducing the number of required measurements to five. We found the tendency that a calibrated view frustum reduced the average distance underestimation of users in an immersive VR environment, but even the correctly calibrated view frustum could not entirely compensate for the distance underestimation effects.

The display units integrated in today's head-mounted displays (HMDs) provide only a limited field of view (FOV) to the virtual world. In order to present an undistorted view to the virtual environment (VE), the perspective projection used to render the VE has to be adjusted to the limitations caused by the HMD characteristics. In particular, the geometric field of view (GFOV), which defines the virtual aperture angle used for rendering of the 3D scene, is set up according to the display field of view (DFOV). A discrepancy between these two fields of view distorts the geometry of the VE in a way that either minifies or magnifies the imagery displayed to the user. It has been shown that this distortion has the potential to affect a user's perception of the virtual space, sense of presence, and performance on visual search tasks. In this paper, we analyze the user's perception of a VE displayed in a HMD, which is rendered with different GFOVs. We introduce a psychophysical calibration method to determine the HMD's actual field of view, which may vary from the nominal values specified by the manufacturer. Furthermore, we conducted two experiments to identify perspective projections for HMDs, which are identified as natural by subjects--even if these perspectives deviate from the perspectives that are inherently defined by the DFOV. In the first experiment, subjects had to adjust the GFOV for a rendered virtual laboratory such that their perception of the virtual replica matched the perception of the real laboratory, which they saw before the virtual one. In the second experiment, we displayed the same virtual laboratory, but restricted the viewing condition in the real world to simulate the limited viewing condition in a HMD environment. We found that subjects evaluate a GFOV as natural when it is larger than the actual DFOV of the HMD--in some cases up to 50 percent--even when subjects viewed the real space with a limited field of view.

Physical characteristics and constraints of today's head-mounted displays (HMDs) often impair interaction in immersive virtual environments (VEs). For instance, due to the limited field of view (FOV) subtended by the display units in front of the user's eyes more effort is required to explore a VE by head rotations than for exploration in the real world.

In immersive virtual environments (IVEs), users can control their virtual viewpoint by moving their tracked head and walking through the real world. Usually, movements in the real world are mapped one-to-one to virtual camera motions. With redirection techniques, the virtual camera is manipulated by applying gains to user motion so that the virtual world moves differently than the real world. Thus, users can walk through large-scale IVEs while physically remaining in a reasonably small workspace. In psychophysical experiments with a two-alternative forced-choice task, we have quantified how much humans can unknowingly be redirected on physical paths that are different from the visually perceived paths. We tested 12 subjects in three different experiments: (E1) discrimination between virtual and physical rotations, (E2) discrimination between virtual and physical straightforward movements, and (E3) discrimination of path curvature. In experiment E1, subjects performed rotations with different gains, and then had to choose whether the visually perceived rotation was smaller or greater than the physical rotation. In experiment E2, subjects chose whether the physical walk was shorter or longer than the visually perceived scaled travel distance. In experiment E3, subjects estimate the path curvature when walking a curved path in the real world while the visual display shows a straight path in the virtual world. Our results show that users can be turned physically about 49 percent more or 20 percent less than the perceived virtual rotation, distances can be downscaled by 14 percent and upscaled by 26 percent, and users can be redirected on a circular arc with a radius greater than 22 m while they believe that they are walking straight.

The display units integrated in todays head-mounted displays (HMDs) provide only a limited field of view (FOV) to the virtual world. In order to present an undistorted view to the virtual environment (VE), the perspective projection used to render the VE has to be adjusted to the limitations caused by the HMD characteristics. In particular, the geometric field of view (GFOV), which defines the virtual aperture angle used for rendering of the 3D scene, is set up according to the display's field of view. A discrepancy between these two fields of view distorts the geometry of the VE in a way that either minifies or magnifies the imagery displayed to the user. Discrepancies between the geometric and physical FOV causes the imagery to be minified or magnified. This distortion has the potential to negatively or positively affect a user's perception of the virtual space, sense of presence, and performance on visual search tasks.

In this paper we introduce a virtual locomotion system that allows navigation within any large-scale virtual environment (VE) by real walking. In contrast to the work of Razzaque et al. (2001) we have developed generic redirected walking concepts by combining motion compression, i. e., scaling the real distance users walk, rotation gains, which make the real turns smaller or larger, and curvature gains, which bend the user's walking direction such that s/he walks on a curve. Furthermore, we introduce the new concept of dynamic passive haptics which extends passive haptics (Insko et al., 2001; Kohli et al., 2005) in such a way that any number of virtual objects can be sensed by means of real proxy objects having similar haptic capabilities, i. e., size, shape and surface structure. We have evaluated these concepts and explain technical details regarding their integration into legacy 3D graphics applications.

Redirected walking allows users to walk through large-scale immersive virtual environments (IVEs) while physically remaining in a reasonably small workspace by intentionally injecting scene motion into the IVE. In a constant stimuli experiment with a twoalternative-forced-choice task we have quantified how much humans can unknowingly be redirected on virtual paths which are different from the paths they actually walk. 18 subjects have been tested in four different experiments: (E1a) discrimination between virtual and physical rotation, (E1b) discrimination between two successive rotations, (E2) discrimination between virtual and physical translation, and discrimination of walking direction (E3a) without and (E3b) with start-up. In experiment E1a subjects performed rotations to which different gains have been applied, and then had to choose whether or not the visually perceived rotation was greater than the physical rotation. In experiment E1b subjects discriminated between two successive rotations where different gains have been applied to the physical rotation. In experiment E2 subjects chose if they thought that the physical walk was longer than the visually perceived scaled travel distance. In experiment E3a subjects walked a straight path in the IVE which was physically bent to the left or to the right, and they estimate the direction of the curvature. In experiment E3a the gain was applied immediately, whereas the gain was applied after a start-up of two meters in experiment E3b. Our results show that users can be turned physically about 68% more or 10 % less than the perceived virtual rotation, distances can be up- or down-scaled by 22%, and users can be redirected on an circular arc with a radius greater than 24 meters while they believe they are walking straight.

Eye movements affect object localization and object recognition. Around saccade onset, briefly flashed stimuli appear compressed towards the saccade target, receptive fields dynamically change position, and the recognition of objects near the saccade target is improved. These effects have been attributed to different mechanisms. We provide a unifying account of peri-saccadic perception explaining all three phenomena by a quantitative computational approach simulating cortical cell responses on the population level. Contrary to the common view of spatial attention as a spotlight, our model suggests that oculomotor feedback alters the receptive field structure in multiple visual areas at an intermediate level of the cortical hierarchy to dynamically recruit cells for processing a relevant part of the visual field. The compression of visual space occurs at the expense of this locally enhanced processing capacity.

Distance estimation of visually simulated self-motion is difficult, because one has to know or make assumptions about scene layout to judge ego speed. Discrimination of the travel distances of two sequentially simulated self-motions in the same scene can be performed quite accurately (Bremmer and Lappe 1999; Frenz et al., 2003). However, the indication of the perceived distance of a single movement in terms of a spatial interval results in a depth scaling error: Intervals are correlated with the true travel distance, but underestimate travel distance by about 25% (Frenz and Lappe, 2005). Here we investigated whether the inclusion of further depth cues (disparity/motion parallax/figural cues) in the virtual environment allows more veridical interval adjustment. Experiments were conducted on a large single projection screen and in a fully immersive computer-animated virtual environment (CAVE). Forward movements in simple virtual environments were simulated with distances between 1.5 and 13 m with varying speeds. Subjects indicated the perceived distance of each movement in terms of a depth interval on the virtual ground plane. We found good correlation between simulated and indicated distances, indicative of an internal representation of the perceived distance. The slopes of the fitted regression lines revealed an underestimation of distance by about 25% under all conditions. We conclude that estimation of travel distance from optic flow is subject to scaling when compared to static intervals in the environment, irrespective of additional depth cues.

Observer translation relative to the world creates image flow that expands from the observer's direction of translation (heading) from which the observer can recover heading direction. Yet, the image flow is often more complex, depending on rotation of the eye, scene layout and translation velocity. A number of models [1-4] have been proposed on how the human visual system extracts heading from flow in a neurophysiologic ally plausible way. These models represent heading by a set of neurons that respond to large image flow patterns and receive input from motion sensed at different im(cid:173) age locations. We analysed these models to determine the exact receptive field of these heading detectors. We find most models predict that, contrary to widespread believe, the contribut ing mo(cid:173) tion sensors have a preferred motion directed circularly rather than radially around the detector's preferred heading. Moreover, the re(cid:173) sults suggest to look for more refined structure within the circular flow, such as bi-circularity or local motion-opponency.

Experimental evidence from neurophysiological recordings in the middle temporal (MT) area of the macaque monkey suggests that motion-selective cells can use disparity information to separate motion signals that originate from different depths. This finding of a cross-talk between different visual channels has implications for the understanding of the processing of motion in the primate visual system and especially for behavioral tasks requiring the determination of global motion. In this paper, the consequences for the analysis of optic flow fields are explored. A network model is presented that effectively uses the disparity sensitivity of MT-like neurons for the reduction of noise in optic flow fields. Simulations reproduce the recent psychophysical finding that the robustness of the human optic flow processing system is improved by stereoscopic depth information, but that the use of this information depends on the structure of the visual environment.

Interest in the processing of optic flow has increased recently in both the neurophysiological and the psychophysical communities. We have designed a neural network model of the visual motion pathway in higher mammals that detects the direction of heading from optic flow. The model is a neural implementation of the subspace algorithm introduced by Heeger and Jepson (1990). We have tested the network in simulations that are closely related to psychophysical and neurophysiological experiments and show that our results are consistent with recent data from both fields. The network reproduces some key properties of human ego-motion perception. At the same time, it produces neurons that are selective for different components of ego-motion flow fields, such as expansions and rotations. These properties are reminiscent of a subclass of neurons in cortical area MSTd, the triple-component neurons. We propose that the output of such neurons could be used to generate a computational map of heading directions in or beyond MST.