Non-player characters (NPCs) are a vital part of many 3D and virtual reality (VR) experiences, but animating them can be a time-consuming and costly process, often requiring expensive motion-capture setups, especially when tracking multiple people simultaneously. Single-user record and replay in VR enables a single user to animate multiple NPCs on their own, offering a cheaper and more convenient solution as the devices are consumer-grade and already track user movements. A limitation of single-user record and replay is that an early recorded character has no information about a later recorded character and might not be able to interact with them believably. We therefore study interactivity between NPCs as an emergent property when actors can see other characters in later recordings as opposed to acting primarily on their own in the first recording. We conducted a user study (N=144) where participants compared first, second, and third recording runs of different social scenarios with multiple NPCs. Our results suggest that for scenarios with low and medium interactivity where timing is not crucial and characters are not in close contact, a single run can be sufficient, whereas for high-interactivity scenarios, with close-contact group interactions, two recordings are more appropriate.

With the increasing popularity of virtual reality (VR) applications, allowing interaction with non-immersed spectators has become an important supporting functionality. To this end, a variety of VR streaming systems focus on allowing remote spectating experiences. However, such systems lack mechanisms that allow spectators to assist or explore inside VR applications alongside the VR players. To address, we explore AccompliceVR (AVR), a novel streaming and remote collaboration system. Using built-in OpenVR functionality, AVR introduces avatar representations of spectators overlaid into the normal stereo views of unmodified VR applications. AVR also enhances the remote spectator experience by stitching frames of video from the VR player to reconstruct a pseudo-panoramic view. This allows them to move around and point at the environment with an avatar. We first evaluated that AVR is compatible with a selection of the most popular VR games. The system is then validated through a paired user study with 30 participants using a mixed-design pattern. Participants played two well-known VR games (as 'pilot') and were assisted by their partner in the pair (as 'copilot'). The results support that AVR shows significant improvement in communication between pilots and copilots and enhances the pilot's understanding of instructions from their copilot.

Music in virtual environments has long been treated as a temporal evolving element, enhancing atmosphere and game pace but rarely considered as a performance adaptive element. Recent advances in artificial intelligence (AI) and procedural audio make it possible to generate music that adapts in real time to player actions and system state. Yet, despite its potential, the behavioural impact of such adaptive generative soundtracks in head-mounted display-based virtual reality (VR) remains largely unexplored. To address this gap, we introduce a VR archery system that integrates Google MusicFXDJ with Ubiq-Genie to deliver continuous AI-generated adaptive music driven by gameplay events. In a within-subjects experiment (N = 22), participants completed trials with either a stylistically-matched fixed soundtrack or an adaptive soundtrack that escalated tension across four phases as arrows depleted. Measures combined self-reported ratings of presence, focus, stress, and emotional impact, with performance metrics of accuracy and aiming time. Results reveal that adaptive generative music not only heightens immersion and emotional salience but also modulates motor precision in an arousal-dependent inverted-U pattern: moderate musical tension improved accuracy and speed, whereas excessive tension impaired them. These findings establish AI-generated music as a powerful behavioural feedback modality in VR, opening pathways for training, rehabilitation, and next-generation immersive entertainment.

Extended Reality (XR) technologies offer many possibilities to support collaboration between remote and local team members. However, in a scenario with multiple devices and multiple interaction options, including virtual, augmented and the continuum of realities among them, it is still unclear how to select the right option to make collaboration more efficient. To provide some design insights in this context, this paper presents a comparative study of collaboration behaviours and outcomes when using different XR technologies, where X stands for Virtual or Augmented. In our experimental setup, pairs of users collaborated on three tasks drawn from visual analytics, each designed to require a different level of synchronization: low, medium, and high. We conducted a study with 40 participants (20 dyads), who performed tasks in two conditions: Augmented Reality (AR) and Virtual Reality (VR). All used the same equipment and interactions; only the immersive environment differed. The results suggest that collaboration outcomes differed only in high-coupling tasks, with AR showing a performance advantage. This may stem from a greater difficulty of synchronizing actions when perceiving a partner through avatars rather than directly. However, no significant differences were observed in collaboration dynamics, and the choice of XR technology did not appear to affect collaboration patterns. These findings suggest that, for collaborative analytical tasks comparable to those in this study, collaborators can choose the XR mode that best fits their preferences and current needs without concern about hindering teamwork.

Authoring site-specific outdoor augmented reality (AR) experiences requires a nuanced understanding of real-world context to create immersive and relevant content. Existing ex-situ authoring tools typically rely on static 3D models to represent spatial information. However, in our formative study (n=25), we identified key limitations of this approach: models are often outdated, incomplete, or insufficient for capturing critical factors such as safety considerations, user flow, and dynamic environmental changes. These issues necessitate frequent on-site visits and additional iterations, making the authoring process more time-consuming and resource-intensive. To mitigate these challenges, we introduce CoCreatAR, an asymmetric collaborative mixed reality authoring system that integrates the flexibility of ex-situ workflows with the immediate contextual awareness of in-situ authoring. We conducted an exploratory study (n=32) comparing CoCreatAR to an asynchronous workflow baseline, finding that it enhances engagement, creativity, and confidence in the authored output while also providing preliminary insights into its impact on task load. We conclude by discussing the implications of our findings for integrating real-world context into site-specific AR authoring systems.

Embodiment in mixed reality describes the sensation of experiencing a virtual representation as an extension of one’s own body. While research has extensively examined embodiment in virtual reality (VR) and head-mounted augmented reality (AR), its impact on smartphones remains underexplored. This study examines how smartphone-based AR embodiment affects user engagement and cognitive performance in a comprehension task. A study involving 24 participants explored whether using a smartphone AR face-filter to embody a virtual audience member influenced the recall of a historical speech. Findings show that participants in the AR condition scored higher on a factual quiz than those in the control group. At the same time, stronger perceived embodiment, especially self-location, was negatively associated with quiz performance, consistent with Cognitive Load Theory. These results should be interpreted cautiously: our comparison contrasted a static image (no AR) with AR that included facial embodiment, so we did not include an “AR without embodiment” condition to fully separate AR novelty from embodiment. Stimuli were also restricted to a single speech and a single historical scene presented as a static image, limiting generalizability to other content and to dynamic or interactive AR. Finally, the sample was modest (N=24), so estimates are preliminary and warrant replication. We discuss implications for designing smartphone AR that balances engagement with cognitive efficiency.

Social virtual reality applications aim to provide immersive, interactive experiences in populated environments with virtual characters. However, developing characters capable of natural verbal and non-verbal interactions remains a significant challenge, particularly when it comes to managing complex and unexpected interactions with users. To address these issues, we present a system that supports full-body avatars with six-point tracking and a streamlined switch control procedure, allowing one actor to assume control of multiple virtual humans and interact seamlessly with the users. The system supports both verbal and non-verbal interactions. In an experiment, we showed that our system enhances the sense of co-presence, creating the feeling that multiple distinct, human-controlled characters are present in the scene.

Real camera footage is subject to noise, motion blur (MB) and depth of field (DoF). In some applications these might be considered distortions to be removed, but in others it is important to model them because it would be ineffective, or interfere with an aesthetic choice, to simply remove them. In augmented reality applications where virtual content is composed into a live video feed, we can model noise, MB and DoF to make the virtual content visually consistent with the video. Existing methods for this typically suffer two main limitations. First, they require a camera calibration step to relate a known calibration target to the specific cameras response. Second, existing work require methods that can be (differentiably) tuned to the calibration, such as slow and specialized neural networks. We propose a method which estimates parameters for noise, MB and DoF instantly, which allows using off-the-shelf real-time simulation methods from e.g., a game engine in compositing augmented content. Our main idea is to unlock both features by showing how to use modern computer vision methods that can remove noise, MB and DoF from the video stream, essentially providing self-calibration. This allows to auto-tune any black-box real-time nose+MB-DoF method to deliver fast and high-fidelity augmentation consistency.

We compared the body movements of five virtual reality (VR) avatar representations in a user study $(\mathrm{N}=53)$ to ascertain how well these representations could convey body motions associated with different emotions: one head-and-hands representation using only tracking data, one upper-body representation using inverse kinematics (IK), and three full-body representations using IK, motioncapture, and the state-of-the-art deep-learning model AGRoL. Participants’ emotion detection accuracies were similar for the IK and AGRoL representations, highest for the full-body motion-capture representation and lowest for the head-and-hands representation. Our findings suggest that from the perspective of emotion expressivity, connected upper-body parts that provide visual continuity improve clarity, and that current techniques for algorithmically animating the lower-body are ineffective. In particular, the deep-learning technique studied did not produce more expressive results, suggesting the need for training data specifically made for social VR applications.

Virtual Reality (VR) has revolutionized how we interact with digital worlds. However, programming for VR remains a complex and challenging task, requiring specialized skills and knowledge. Powered by large language models (LLMs), DreamCodeVR is designed to assist users, irrespective of their coding skills, in crafting basic object behavior in VR environments by translating spoken language into code within an active application. This approach seeks to simplify the process of defining behaviors visual changes through speech. Our preliminary user study indicated that the system’s speech interface supports elementary programming tasks, highlighting its potential to improve accessibility for users with varying technical skills. However, it also uncovered a wide range of challenges and opportunities. In an extensive discussion, we detail the system’s strengths, weaknesses, and areas for future research.

Presence is a powerful aspect of Virtual Reality (VR). However, there has been no consensus on how to achieve presence in Augmented Reality (AR) or whether it exists at all. The Place Illusion, a key component in presence as defined in VR, cannot be obtained in AR as there is no way to make the user feel as though they are transported somewhere else when they are limited to what they can physically see in front of them. However, recently it has been argued that coherence or congruence are important parts of the Place and Plausibility Illusions. The implication for AR is that the AR content might invoke a higher Plausibility Illusion if it is consistent with the physical place the content is situated in. In this study, we define the concept of a Context-Relevant Location (CRL), a physical place that is congruent with the experience. We present a study with a between-subjects design that allowed users to interact with AR objects in a CRL and in a generic environment. The results indicate that presence was higher in the CRL setting than the generic environment, contribute to the debate about providing a concrete description of presence-like phenomena in AR, and posit that CRLs play a similar role to the Place Illusion in an AR setting.

In this abstract, we discuss the demand for replication and extension efforts related to two seminal studies focused on virtual reality (VR) locomotion interfaces, initially centered around a VR implementation of the Visual Cliff, often referred to as Virtual Pit. The original experiments by Slater et al. (1995) and Usoh et al. (1999) compared different locomotion methods, including Real Walking, Walking-in-Place, and Flying/Steering, with a focus on presence and ease of use. We discuss the importance of these studies for the field, motivate replication efforts focused on these studies, discuss potential confounding factors, and present considerations for a concerted effort to reproduce the findings with state-of-the-art VR systems and measures, extensions to locomotion methods like Teleportation, and means to support future replications and extensions.

Social Virtual Reality (SVR) systems are growing in both popularity and breadth. New systems are attempting to support a wider range of increasingly large social situations. A key characteristic of SVR systems is user capacity. Many systems are designed for small group discussion or team games, and support up to 20-30 users. Some support larger counts, but via special features coupled to a specific social situation. As SVR broaches medium to large gatherings (10s-1000s of people), capacity becomes a major challenge. Naïve implementations where each user communicates directly with every other will quickly overwhelm the resources of the system. The constraints on scalability have been known since the first SVR systems were created. However, while there are many systems out there, their scalability mechanisms are often application specific, and rarely openly discussed. In this paper we explore scalability in the open-source SVR Ubiq. We consider two partitioning schemes to increase relay-server capacity. We examine the current failure modes of the relay-server system, and demonstrate how these schemes improve capacity by a factor of 2-3x. We look at the interactions between the schemes and simulated user behaviour, to see what lessons can be gleaned for research into scalability for SVR.

Online social virtual worlds are now becoming widely available on consumer devices including virtual reality headsets. One goal of a virtual world could be to give a user an experience of a crowded environment with many virtual humans. However, gathering enough personnel to control the necessary number of avatars for creating a realistic scene is usually difficult. Additionally, current technology is not capable of fully simulating avatars with behaviours, especially when interaction with users is required. In this paper, we develop a system that enables an actor to take over control of one of a set of avatars. We built an immersive interface that allows an actor to select an avatar to take over and then segue into the currently playing animation. By allowing one person to take control of multiple avatars, we can enhance the plausibility of environments inhabited by simulated characters. In an experiment, we show that in a cafe scenario, one actor can take over the roles of a barista and two customers. Experiment participants reported experiencing the scene as if it were populated by more than one actor. This system and experiment demonstrate the feasibility of one actor controlling multiple avatars sequentially, thus enhancing users’ feelings of being in a populated environment.

The evaluation of users’ experiences in virtual environments is an important task for researchers in the fields of human-computer interaction and extended reality. It can be used to understand and enhance the quality of users’ mediated interactions and communications. In a constantly evolving world, where people are growing with technology, it is important to understand, evaluate and enhance the use of immersive media. In the research agenda of this Ph.D. thesis, the challenge of developing multi-user experiences in virtual environments and setting evaluation metrics for researchers are considered. This Ph.D. thesis showcases an interest in how to enhance trust formation in media-based social environments. The findings of this Ph.D. are expected to help create new open-source tools to facilitate the understanding of individuals and groups in extended reality applications.

Non-player characters are an essential element of many 3D and virtual reality experiences. They can make the experiences feel more lively and populated. Animation for non-player characters is often motion-captured using expensive hardware and the post-processing steps are time-consuming, especially when capturing multiple people at once. Using record and replay techniques in virtual reality can offer cheaper and easier ways of motion capture since the user is already tracked. We use immersive record and replay to enable a single user to create stacked recordings of themselves. We provide tools to help the user interact with their previous recorded self and in doing so allow them to create believable interactive scenarios with multiple characters that can be used to populate virtual environments. We create a small dialogue dataset with two amateur actors who used our tool to record dialogues alone and together in virtual reality. To evaluate whether stacked recordings are qualitatively comparable to conventional multi-user recordings and whether people could tell the difference between the two, we conducted two user studies, one online and one in virtual reality with 89 participants in total. We found that participants could not tell the difference and even slightly preferred stacked recordings.

Mixed-reality systems provide a number of different ways of representing users to each other in collaborative scenarios. There is an obvious tension between using media such as video for remote users compared to representations as avatars. This paper includes two experiments (total n = 80) on user trust when exposed to two of three different user representations in an immersive virtual reality environment that also acts as a simulation of typical augmented reality simulations: full body video, head and shoulder video and an animated 3D model. These representations acted as advisors in a trivia quiz. By evaluating trust through advisor selection and self-report, we found only minor differences between representations, but a strong effect of perceived advisor expertise. Unlike prior work, we did not find the 3D model scored poorly on trust, perhaps as a result of greater congruence within an immersive context.

Image-warping, a per-pixel deformation of one image into another, is an essential component in immersive visual experiences such as virtual reality or augmented reality. The primary issue with image warping is disocclusions, where occluded (and hence unknown) parts of the input image would be required to compose the output image. We introduce a new image warping method, Metameric image inpainting - an approach for hole-filling in real-time with foundations in human visual perception. Our method estimates image feature statistics of disoccluded regions from their neighbours. These statistics are inpainted and used to synthesise visuals in real-time that are less noticeable to study participants, particularly in peripheral vision. Our method offers speed improvements over the standard structured image inpainting methods while improving realism over colour-based inpainting such as push-pull. Hence, our work paves the way towards future applications such as depth image-based rendering, 6-DoF 360 rendering, and remote render-streaming.

We present Monte-Carlo Redirected Walking (MCRDW), a gain selection algorithm for redirected walking. MCRDW applies the Monte-Carlo method to redirected walking by simulating a large number of simple virtual walks, then inversely applying redirection to the virtual paths. Different gain levels and directions are applied, producing differing physical paths. Each physical path is scored and the results used to select the best gain level and direction. We provide a simple example implementation and a simulation-based study for validation. In our study, when compared with the next best technique, MCRDW reduced incidence of boundary collisions by over 50% while reducing total rotation and position gain.

Blue noise is known for its uniformity in the spatial domain, avoiding the appearance of structures such as voids and clusters. Because of this characteristic, it has been adopted in a wide range of visual computing applications, such as image dithering, rendering and visualisation. This has motivated the development of a variety of generative methods for blue noise, with different trade-offs in terms of accuracy and computational performance. We propose a novel unsupervised learning approach that leverages a neural network architecture to generate blue noise masks with high accuracy and real-time performance, starting from a white noise input. We train our model by combining three unsupervised losses that work by conditioning the Fourier spectrum and intensity histogram of noise masks predicted by the network. We evaluate our method by leveraging the generated noise for two applications: grayscale blue noise masks for image dithering, and blue noise samples for Monte Carlo integration.

Computer-Generated Holography (CGH) offers the potential for genuine, high-quality three-dimensional visuals. However, fulfilling this potential remains a practical challenge due to computational complexity and visual quality issues. We propose a new CGH method that exploits gaze-contingency and perceptual graphics to accelerate the development of practical holographic display systems. Firstly, our method infers the user’s focal depth and generates images only at their focus plane without using any moving parts. Second, the images displayed are metamers; in the user’s peripheral vision, they need only be statistically correct and blend with the fovea seamlessly. Unlike previous methods, our method prioritises and improves foveal visual quality without causing perceptually visible distortions at the periphery. To enable our method, we introduce a novel metameric loss function that robustly compares the statistics of two given images for a known gaze location. In parallel, we implement a model representing the relation between holograms and their image reconstructions. We couple our differentiable loss function and model to metameric varifocal holograms using a stochastic gradient descent solver. We evaluate our method with an actual proof-of-concept holographic display, and we show that our CGH method leads to practical and perceptually three-dimensional image reconstructions.

A common issue for collaborative mixed reality is the asymmetry of interaction with the shared virtual environment. For example, an augmented reality (AR) user might use one type of head-mounted display (HMD) in a physical environment, while a virtual reality (VR) user might wear a different type of HMD and see a virtual model of that physical environment. To explore the effects of such asymmetric interfaces on collaboration we present a study that investigates the behaviour of dyads performing a word puzzle task where one uses AR and the other VR. We examined the collaborative process through questionnaires and behavioural measures based on positional and audio data. We identified relationships between presence and co-presence, accord and co-presence, leadership and talkativeness, head rotation velocity and leadership, and head rotation velocity and talkativeness. We did not find that AR or VR biased subjective responses, though there were interesting behavioural differences: AR users spoke more words, AR users had a higher median head rotation velocity, and VR users travelled further.

Mutual awareness of visual attention is essential for collaborative work. In the field of collaborative virtual environments (CVE), it has been proposed to use Field-of-View (FoV) frustum visualisations as a cue to support mutual awareness during collaboration. Recent studies on FoV frustum visualisations focus on asymmetric collaboration with AR/VR hardware setups and 3D reconstructed environments. In contrast, we focus on the general-purpose CVEs (i.e., VR shared offices), whose popularity is increasing due to the availability of low-cost headsets, and the restrictions imposed by the pandemic. In these CVEs collaboration roles are symmetrical, and the same 2D content available on desktop computers is displayed on 2D surfaces in a 3D space (VR screens). We prototyped one such CVE to evaluate FoV frustrum visualisation within this collaboration scenario. We also implement a FoV visualisation generated from an average fixation map (AFM), therefore directly generated by users' gaze behaviour which we call Cone of Vision (CoV). Our approach to displaying the frustum visualisations is tailored for 2D surfaces in 3D space and allows for self-awareness of this visual cue. We evaluate CoV in the context of a general exploratory data analysis (EDA) with 10 pairs of participants. Our findings indicate that CoV is beneficial during shifts between independent and collaborative work and supports collaborative progression across the visualisation. Self-perception of the CoV improves visual attention coupling, reduces the number of times users watch the collaborator's avatars and offers a consistent representation of the shared reality.

Distributed virtual environments (DVEs) are challenging to create as the goals of consistency and responsiveness become contradictory under increasing latency. DVEs have been considered as both distributed transactional databases and force-reflection systems. Both are good approaches, but they do have drawbacks. Transactional systems do not support Level 3 (L3) collaboration: manipulating the same degree-of-freedom at the same time. Force-reflection requires a client-server architecture and stabilisation techniques. With Consensus Based Networking (CBN), we suggest DVEs be considered as a distributed data-fusion problem. Many simulations run in parallel and exchange their states, with remote states integrated with continous authority. Over time the exchanges average out local differences, performing a distribued-average of a consistent, shared state. CBN aims to build simulations that are highly responsive, but consistent enough for use cases such as the piano-movers problem. CBN's support for heterogeneous nodes can transparently couple different input methods, avoid the requirement of determinism, and provide more options for personal control over the shared experience. Our work is early, however we demonstrate many successes, including L3 collaboration in room-scale VR, 1000's of interacting objects, complex configurations such as stacking, and transparent coupling of haptic devices. These have been shown before, but each with a different technique; CBN supports them all within a single, unified system.

Sketch and speech are intuitive interaction methods that convey complementary information and have been independently used for 3D model retrieval in virtual environments. While sketch has been shown to be an effective retrieval method, not all collections are easily navigable using this modality alone. We design a new challenging database for sketch comprised of 3D chairs where each of the components (arms, legs, seat, back) are independently colored. To overcome this, we implement a multimodal interface for querying 3D model databases within a virtual environment. We base the sketch on the state-of-the-art for 3D Sketch Retrieval, and use a Wizard-of-Oz style experiment to process the voice input. In this way, we avoid the complexities of natural language processing which frequently requires fine-tuning to be robust. We conduct two user studies and show that hybrid search strategies emerge from the combination of interactions, fostering the advantages provided by both modalities.

While they have long been a subject of academic study, social virtual reality (SVR) systems are now attracting increasingly large audiences on current consumer virtual reality systems. The design space of SVR systems is very large, and relatively little is known about how these systems should be constructed in order to be usable and efficient. In this paper we present Ubiq, a toolkit that focuses on facilitating the construction of SVR systems. We argue for the design strategy of Ubiq and its scope. Ubiq is built on the Unity platform. It provides core functionality of many SVR systems such as connection management, voice, avatars, etc. However, its design remains easy to extend. We demonstrate examples built on Ubiq and how it has been successfully used in classroom teaching. Ubiq is open source (Apache License) and thus enables several use cases that commercial systems cannot.

Crowd simulation algorithms play an essential role in populating Virtual Reality (VR) environments with multiple autonomous humanoid agents. The generation of plausible trajectories can be a significant computational cost for real-time graphics engines, especially in untethered and mobile devices such as portable VR devices. Previous research explores the plausibility and realism of crowd simulations on desktop computers but fails to account the impact it has on immersion. This study explores how the realism of crowd trajectories affects the perceived immersion in VR. We do so by running a psychophysical experiment in which participants rate the realism of real/synthetic trajectories data, showing similar level of perceived realism.

To peripheral vision, a pair of physically different images can look the same. Such pairs are metamers relative to each other, just as physically-different spectra of light are perceived as the same color. We propose a real-time method to compute such ventral metamers for foveated rendering where, in particular for near-eye displays, the largest part of the framebuffer maps to the periphery. This improves in quality over state-of-the-art foveation methods which blur the periphery. Work in Vision Science has established how peripheral stimuli are ventral metamers if their statistics are similar. Existing methods, however, require a costly optimization process to find such metamers. To this end, we propose a novel type of statistics particularly well-suited for practical real-time rendering: smooth moments of steerable filter responses. These can be extracted from images in time constant in the number of pixels and in parallel over all pixels using a GPU. Further, we show that they can be compressed effectively and transmitted at low bandwidth. Finally, computing realizations of those statistics can again be performed in constant time and in parallel. This enables a new level of quality for foveated applications such as such as remote rendering, level-of-detail and Monte-Carlo denoising. In a user study, we finally show how human task performance increases and foveation artifacts are less suspicious, when using our method compared to common blurring.

Mobile HMDs must sacrifice compute performance to achieve ergonomic and power requirements for extended use. Consequently, applications must either reduce rendering and simulation complexity - along with the richness of the experience - or offload complexity to a server. Within the context of edge-computing, a popular way to do this is through render streaming. Render streaming has been demonstrated for desktops and consoles. It has also been explored for HMDs. However, the latency requirements of head tracking make this application much more challenging. While mobile GPUs are not yet as capable as their desktop counterparts, we note that they are becoming more powerful and efficient. With the hard requirements of VR, it is worth continuing to investigate what schemes could optimally balance load, latency and quality. We propose an alternative we call edge-physics: streaming at the scene-graph level from a simulation running on edge-resources, analogous to cluster rendering. Scene streaming is not only straightforward, but compute and bandwidth efficient. The most demanding loops run locally. Jobs that hit the power-wall of mobile CPUs are off-loaded, while improving GPUs are leveraged, maximising compute utilisation. In this paper we create a prototypical implementation and evaluate its potential in terms of fidelity, bandwidth and performance. We show that an effective system which maintains high consistencies on typical edge-links can be easily built, but that some traditional concepts are not applicable, and a better understanding of the perception of motion is required to evaluate such a system comprehensively.

With the continuing development of affordable immersive virtual reality (VR) systems, there is now a growing market for consumer content. The current form of consumer systems is not dissimilar to the lab-based VR systems of the past 30 years: the primary input mechanism is a head-tracked display and one or two tracked hands with buttons and joysticks on hand-held controllers. Over those 30 years, a very diverse academic literature has emerged that covers design and ergonomics of 3D user interfaces (3DUIs). However, the growing consumer market has engaged a very broad range of creatives that have built a very diverse set of designs. Sometimes these designs adopt findings from the academic literature, but other times they experiment with completely novel or counter-intuitive mechanisms. In this paper and its online adjunct, we report on novel 3DUI design patterns that are interesting from both design and research perspectives: they are highly novel, potentially broadly re-usable and/or suggest interesting avenues for evaluation. The supplemental material, which is a living document, is a crowd-sourced repository of interesting patterns. This paper is a curated snapshot of those patterns that were considered to be the most fruitful for further elaboration.

Measuring Visual Latency in VR and AR devices has become increasingly complicated as many of the components will influence others in multiple loops and ultimately affect the human cognitive and sensory perception. In this paper we present a new method based on the idea that the performance of humans on a rapid motor task will remain constant, and that any added delay will correspond to the system latency. We ask users to perform a task inside different video see-through devices and also in front of a computer. We also calculate the latency of the systems using a hardware instrumentation-based measurement technique for bench-marking. Results show that this new form of latency measurement through human cognitive performance can be reliable and comparable to hardware instrumentation-based measurement. Our method is adaptable to many forms of user interaction. It is particularly suitable for systems, such as AR and VR, where externalizing signals is difficult, or where it is important to measure latency while the system is in use by a user.

Filmmakers of panoramic videos frequently struggle to guide attention to Regions of Interest (ROIs) due to consumers’ freedom to explore. Some researchers hypothesize that peripheral cues attract reflexive/involuntary attention whereas cues within central vision engage and direct voluntary attention. This mixed-methods study evaluated the effectiveness of using central arrows and peripheral flickers to guide and focus attention in panoramic videos. Twenty-five adults wore a head-mounted display with an eye tracker and were guided to 14 ROIs in two panoramic videos. No significant differences emerged in regard to the number of followed cues, the time taken to reach and observe ROIs, ROI-related memory and user engagement. However, participants’ gaze travelled a significantly greater distance toward ROIs within the first 500 ms after flicker-onsets compared to arrow-onsets. Nevertheless, most users preferred the arrow and perceived it as significantly more rewarding than the flicker. The findings imply that traditional attention paradigms are not entirely applicable to panoramic videos, as peripheral cues appear to engage both involuntary and voluntary attention. Theoretical and practical implications as well as limitations are discussed.

Grounded haptic devices can provide a variety of forces but have limited working volumes. Wearable haptic devices operate over a large volume but are relatively restricted in the types of stimuli they can generate. We propose the concept of docking haptics, in which different types of haptic devices are dynamically docked at run time. This creates a hybrid system, where the potential feedback depends on the user’s location. We show a prototype docking haptic workspace, combining a grounded six degree-of-freedom force feedback arm with a hand exoskeleton. We are able to create the sensation of weight on the hand when it is within reach of the grounded device, but away from the grounded device, hand-referenced force feedback is still available. A user study demonstrates that users can successfully discriminate weight when using docking haptics, but not with the exoskeleton alone. Such hybrid systems would be able to change configuration further, for example docking two grounded devices to a hand in order to deliver twice the force, or extend the working volume. We suggest that the docking haptics concept can thus extend the practical utility of haptics in user interfaces.

Abstract Users of smart devices frequently need to exchange data with people nearby to them. Yet despite the availability of various communication methods, data exchange between co-located devices is often complicated by technical and user experience barriers. A potential solution to these issues is the emerging technology of device-to-device acoustic data transmission. In this work, we investigate the medium-specific properties of sound as a data exchange mechanism, and question how these contribute to the user experience of sharing data. We present a user study comparing three wireless communication technologies (acoustic data transmission, QR codes and Bluetooth), when used for a common and familiar scenario: peer-to-peer sharing of contact information. Overall, the results show that acoustic data transmission provides a rapid means of transferring data (mean transaction time of 2.4 s), in contrast to Bluetooth (8.3 s) and QR (6.3 s), whilst requiring minimal physical effort and user coordination. All QR code transactions were successful on the first attempt; however, some acoustic (5.6%) and Bluetooth (16.7%) transactions required multiple attempts to successfully share a contact. Participants also provided feedback on their user experience via surveys and semi-structured interviews. Perceived transaction time, physical effort, and connectivity issues. Specifically, users expressed frustration with Bluetooth due to device selection issues, and with QR for the physical coordination required to scan codes. The findings indicate that acoustic data transmission has unique advantages in facilitating information sharing and interaction between co-located users.

Through avatar embodiment in Virtual Reality (VR) we can achieve the illusion that an avatar is substituting our body: the avatar moves as we move and we see it from a first person perspective. However, self-identification, the process of identifying a representation as being oneself, poses new challenges because a key determinant is that we see and have agency in our own face. Providing control over the face is hard with current HMD technologies because face tracking is either cumbersome or error prone. However, limited animation is easily achieved based on speaking. We investigate the level of avatar enfacement, that is believing that a picture of a face is one's own face, with three levels of facial animation: (i) one in which the facial expressions of the avatars are static, (ii) one in which we implement lip-sync motion and (iii) one in which the avatar presents lip-sync plus additional facial animations, with blinks, designed by a professional animator. We measure self-identification using a face morphing tool that morphs from the face of the participant to the face of a gender matched avatar. We find that self-identification on avatars can be increased through pre-baked animations even when these are not photorealistic nor look like the participant.

Advances in tracking technology and wireless headsets enable walking as a means of locomotion in Virtual Reality. When exploring virtual environments larger than room-scale, it is often desirable to increase users' perceived walking speed, for which we investigate three methods. (1) Ground-Level Scaling increases users' avatar size, allowing them to walk farther. (2) Eye-Level Scaling enables users to walk through a World in Miniature, while maintaining a street-level view. (3) Seven-League Boots amplifies users' movements along their walking path. We conduct a study comparing these methods and find that users feel most embodied using Ground-Level Scaling and consequently increase their stride length. Using Seven-League Boots, unlike the other two methods, diminishes positional accuracy at high gains, and users modify their walking behavior to compensate for the lack of control. We conclude with a discussion on each technique's strength and weaknesses and the types of situation they might be appropriate for.

There are important individual differences when experiencing VR setups. We ran a study with 20 participants who got a scale-matched avatar and were asked to blind-walk to a VR target placed 2.5 meters away. In such setups, people typically underestimate distances by approximately 10% when virtual environments are viewed through head mounted displays. Consistent with previous studies we found that the underestimation was significantly reduced the more embodied the participants were. However, not all participants developed the same level of embodiment when exposed to the exact same conditions.

Volumetric capture allows the creation of near-video-quality content that can be explored with six degrees of freedom. Due to limitations in these experiences, such as the content being fixed at the point of filming, an understanding of eye-gaze awareness is critical. A repeated measures experiment was conducted that explored users' ability to evaluate where a volumetrically captured avatar (VCA) was looking. Wearing one of two head-mounted displays (HMDs), 36 participants rotated a VCA to look at a target. The HMD resolution, target position, and VCA's eye-gaze direction were varied. Results did not show a difference in accuracy between HMD resolutions, while the task became significantly harder for target locations further away from the user. In contrast to real-world studies, participants consistently misjudged eye-gaze direction based on target location, but not based on the avatar's head turn direction. Implications are discussed, as results for VCAs viewed in HMDs appear to differ from face-to-face scenarios.

Redirected walking allows for natural locomotion in virtual environments that are larger than a user’s physical environment. The mapping between real and virtual motion is modified by scaling some aspect of motion. As a user traverses the virtual environment these modifications (or gains) must be dynamically adjusted to prevent collision with physical obstacles. A significant body of work has established perceptual thresholds on rates of absolute gain, but the effect of changing gain is little understood.

Current consumer virtual reality applications typically represent the user by an avatar comprising a simple head/torso and decoupled hands. In the prior work of Steed et al. it was shown that the presence or absence of an avatar could have a significant impact on the cognitive load of the user. We extend that work in two ways. First they only used a full-body avatar with articulated arms, so we add a condition with hands-only representation similar to the majority of current consumer applications. Second we provide a real-world benchmark so as to start to get at the impact of using any immersive system. We validate the prior results: real and full body avatar performance on a memory task is significantly better than no avatar. However the hands only condition is not significantly different than either these two extremes. We discuss why this might be, in particular we discuss the potential for a individual variation in response to the embodiment level.

Abstract In real‐time rendering, the appearance of scenes is greatly affected by the quality and resolution of the textures used for image synthesis. At the same time, the size of textures determines the performance and the memory requirements of rendering. As a result, finding the optimal texture resolution is critical, but also a non‐trivial task since the visibility of texture imperfections depends on underlying geometry, illumination, interactions between several texture maps, and viewing positions. Ideally, we would like to automate the task with a visibility metric, which could predict the optimal texture resolution. To maximize the performance of such a metric, it should be trained on a given task. This, however, requires sufficient user data which is often difficult to obtain. To address this problem, we develop a procedure for training an image visibility metric for a specific task while reducing the effort required to collect new data. The procedure involves generating a large dataset using an existing visibility metric followed by refining that dataset with the help of an efficient perceptual experiment. Then, such a refined dataset is used to retune the metric. This way, we augment sparse perceptual data to a large number of per‐pixel annotated visibility maps which serve as the training data for application‐specific visibility metrics. While our approach is general and can be potentially applied for different image distortions, we demonstrate an application in a game‐engine where we optimize the resolution of various textures, such as albedo and normal maps.

We suggest a rasterization pipeline tailored towards the needs of HMDs, where latency and field-of-view requirements pose new challenges beyond those of traditional desktop displays. Instead of image warping for low latency, or using multiple passes for foveation, we show how both can be produced directly in a single perceptual rasterization pass. We do this with per-fragment ray-casting. This is enabled by derivations of tight space-time-fovea pixel bounds, introducing just enough flexibility for the requisite geometric tests, but retaining most of the simplicity and efficiency of the traditional rasterizaton pipeline. To produce foveated images, we rasterize to an image with spatially varying pixel density. To compensate for latency, we extend the image formation model to directly produce "rolling" images where the time at each pixel depends on its display location. Our approach overcomes limitations of warping with respect to disocclusions, object motion and view-dependent shading, as well as geometric aliasing artifacts in other foveated rendering techniques. A set of perceptual user studies demonstrates the efficacy of our approach.

Many techniques facilitate real-time collision detection against complex models. These typically work by pre-computing information about the spatial distribution of geometry into a form that can be quickly queried. When models deform though, expensive pre-computations are impractical. We present radial fields: a variant of distance fields parameterised in cylindrical space, rather than Cartesian space. This 2D parameterisation significantly reduces the memory and computation requirements of the field, while introducing minimal overhead in collision detection tests. The interior of the mesh is defined implicitly for the entire domain. Importantly, it maps well to the hardware rasteriser of the GPU. Radial fields are much more application-specific than traditional distance fields. For these applications - such as collision detection with articulated characters-however, the benefits are substantial.

Real-time interactive systems such as virtual environments have high performance requirements, and profiling is a key part of the optimisation process to meet them. Traditional techniques based on metadata and static analysis have difficulty following causality in asynchronous systems. In this paper we explore a new technique for such systems. Timestamped samples of the system state are recorded at instrumentation points at runtime. These are assembled into a graph, and edges between dependent samples recovered. This approach minimises the invasiveness of the instrumentation, while retaining high accuracy. We describe how our instrumentation can be implemented natively in common environments, how its output can be processed into a graph describing causality, and how heterogeneous data sources can be incorporated into this to maximise the scope of the profiling. Across three case studies, we demonstrate the efficacy of this approach, and how it supports a variety of metrics for comprehensively bench-marking distributed virtual environments.

We describe a novel method for searching 3D model collections using free-form sketches within a virtual environment as queries. As opposed to traditional Sketch Retrieval, our queries are drawn directly onto an example model. Using immersive virtual reality the user can express their query through a sketch that demonstrates the desired structure, color and texture. Unlike previous sketch-based retrieval methods, users remain immersed within the environment without relying on textual queries or 2D projections which can disconnect the user from the environment. We show how a convolutional neural network (CNN) can create multi-view representations of colored 3D sketches. Using such a descriptor representation, our system is able to rapidly retrieve models and in this way, we provide the user with an interactive method of navigating large object datasets. Through a preliminary user study we demonstrate that by using our VR 3D model retrieval system, users can perform quick and intuitive search. Using our system users can rapidly populate a virtual environment with specific models from a very large database, and thus the technique has the potential to be broadly applicable in immersive editing systems.

In virtual reality a real walking interface limits the extent of a virtual environment to our local walkable space. As local spaces are specific to each user, sharing a virtual environment with others for collaborative work or games becomes complicated. It is not clear which user's walkable space to prefer, or whether that space will be navigable for both users.

The consumption of 360-degree videos with head-mounted displays (HMDs) is increasing rapidly. A large number of HMD users watch 360-degree videos at home, often on non-swivel seats; however videos are frequently designed to require the user to turn around. This work explores how the difference in users' chair type might influence their viewing experience. A between-subject experiment was conducted with 41 participants. Three chair conditions were used: fixed, half-swivel and full-swivel. A variety of measures were explored using eye-tracking, questionnaires, tasks and semi-structured interviews. Results suggest that the fixed and half-swivel chairs discouraged exploration for certain videos compared with the full-swivel chair. Additionally, participants in the fixed chair had worse spatial awareness and greater concern about missing something for certain video than those in the full-swivel chair. No significant differences were found in terms of incidental memory, general engagement and simulator sickness among the three chair conditions. Furthermore, thematic analysis of post-experiment interviews revealed four themes regarding the restrictive chairs: physical discomfort, difficulty following moving objects, reduced orientation and guided attention. Based on the findings, practical implications, limitations and future work are discussed.

Now that high-end consumer phones can support immersive virtual reality, we ask whether social virtual reality is a promising medium for supporting distributed groups of users. We undertook an exploratory in-the-wild study using Samsung Gear VR headsets to see how existing social groups that had become geographically dispersed could use VR for collaborative activities. The study showed a strong propensity for users to feel present and engaged with group members. Users were able to bring group behaviors into the virtual world. To overcome some technical limitations, they had to create novel forms of interaction. Overall, the study found that users experience a range of emotional states in VR that are broadly similar to those that they would experience face-to-face in the same groups. The study highlights the transferability of existing social group dynamics in VR interactions but suggests that more work would need to be done on avatar representations to support some intimate conversations.

Rendering accurate and convincing virtual content into mixed reality (MR) scenes requires detailed illumination information about the real environment. In existing MR systems, this information is often captured using light probes [1, 8, 9, 17, 19--21], or by reconstructing the real environment as a preprocess [31, 38, 54]. We present a method for capturing and updating a HDR radiance map of the real environment and tracking camera motion in real time using a self-contained camera system, without prior knowledge about the real scene. The method is capable of producing plausible results immediately and improving in quality as more of the scene is reconstructed. We demonstrate how this can be used to render convincing virtual objects whose illumination changes dynamically to reflect the changing real environment around them.

Coarse building mass models are now routinely generated at scales ranging from individual buildings to whole cities. Such models can be abstracted from raw measurements, generated procedurally, or created manually. However, these models typically lack any meaningful geometric or texture details, making them unsuitable for direct display. We introduce the problem of automatically and realistically decorating such models by adding semantically consistent geometric details and textures. Building on the recent success of generative adversarial networks (GANs), we propose F ranken GAN, a cascade of GANs that creates plausible details across multiple scales over large neighborhoods. The various GANs are synchronized to produce consistent style distributions over buildings and neighborhoods. We provide the user with direct control over the variability of the output. We allow him/her to interactively specify the style via images and manipulate style-adapted sliders to control style variability. We test our system on several large-scale examples. The generated outputs are qualitatively evaluated via a set of perceptual studies and are found to be realistic, semantically plausible, and consistent in style.

A large number of imaging and computer graphics applications require localized information on the visibility of image distortions. Existing image quality metrics are not suitable for this task as they provide a single quality value per image. Existing visibility metrics produce visual difference maps, and are specifically designed for detecting just noticeable distortions but their predictions are often inaccurate. In this work, we argue that the key reason for this problem is the lack of large image collections with a good coverage of possible distortions that occur in different applications. To address the problem, we collect an extensive dataset of reference and distorted image pairs together with user markings indicating whether distortions are visible or not. We propose a statistical model that is designed for the meaningful interpretation of such data, which is affected by visual search and imprecision of manual marking. We use our dataset for training existing metrics and we demonstrate that their performance significantly improves. We show that our dataset with the proposed statistical model can be used to train a new CNN-based metric, which outperforms the existing solutions. We demonstrate the utility of such a metric in visually lossless JPEG compression, super-resolution and watermarking.

360° images and video have become extremely popular formats for immersive displays, due in large part to the technical ease of content production. While many experiences use a single camera viewpoint, an increasing number of experiences use multiple camera locations. In such multi-view 360° media (MV360M) systems, a visual effect is required when the user transitions from one camera location to another. This effect can take several forms, such as a cut or an image-based warp, and the choice of effect may impact many aspects of the experience, including issues related to enjoyment and scene understanding. To investigate the effect of transition types on immersive MV360M experiences, a repeated-measures experiment was conducted with 31 participants. Wearing a head-mounted display, participants explored four static scenes, for which multiple 360° images and a reconstructed 3D model were available. Three transition types were examined: teleport, a linear move through a 3D model of the scene, and an image-based transition using a Möbius transformation. The metrics investigated included spatial awareness, users' movement profiles, transition preference and the subjective feeling of moving through the space. Results indicate that there was no significant difference between transition types in terms of spatial awareness, while significant differences were found for users' movement profiles, with participants taking 1.6 seconds longer to select their next location following a teleport transition. The model and Möbius transitions were significantly better in terms of creating the feeling of moving through the space. Preference was also significantly different, with model and teleport transitions being preferred over Möbius transitions. Our results indicate that trade-offs between transitions will require content creators to think carefully about what aspects they consider to be most important when producing MV360M experiences.

As we explore the use of consumer virtual reality technology for training applications, there is a need to evaluate its validity compared to more traditional training formats. In this paper, we present a study that compares the effectiveness of virtual training and physical training for teaching a bimanual assembly task. In a between-subjects experiment, 60 participants were trained to solve three 3D burr puzzles in one of six conditions comprised of virtual and physical training elements. In the four physical conditions, training was delivered via paper- and video-based instructions, with or without the physical puzzles to practice with. In the two virtual conditions, participants learnt to assemble the puzzles in an interactive virtual environment, with or without 3D animations showing the assembly process. After training, we conducted immediate tests in which participants were asked to solve a physical version of the puzzles. We measured performance through success rates and assembly completion testing times. We also measured training times as well as subjective ratings on several aspects of the experience. Our results show that the performance of virtually trained participants was promising. A statistically significant difference was not found between virtual training with animated instructions and the best performing physical condition (in which physical blocks were available during training) for the last and most complex puzzle in terms of success rates and testing times. Performance in retention tests two weeks after training was generally not as good as expected for all experimental conditions. We discuss the implications of the results and highlight the validity of virtual reality systems in training.

When rendering virtual objects in a mixed reality application, it is helpful to have access to an environment map that captures the appearance of the scene from the perspective of the virtual object. It is straightforward to render virtual objects into such maps, but capturing and correctly rendering the real components of the scene into the map is much more challenging. This information is often recovered from physical light probes, such as reflective spheres or fisheye cameras, placed at the location of the virtual object in the scene. For many application areas, however, real light probes would be intrusive or impractical. Ideally, all of the information necessary to produce detailed environment maps could be captured using a single device. We introduce a method using an RGBD camera and a small fisheye camera, contained in a single unit, to create environment maps at any location in an indoor scene. The method combines the output from both cameras to correct for their limited field of view and the displacement from the virtual object, producing complete environment maps suitable for rendering the virtual content in real time. Our method improves on previous probeless approaches by its ability to recover high-frequency environment maps. We demonstrate how this can be used to render virtual objects which shadow, reflect and refract their environment convincingly.

The proliferation of head-mounted displays (HMD) in the market means that cinematic virtual reality (CVR) is an increasingly popular format. We explore several metrics that may indicate advantages and disadvantages of CVR compared to traditional viewing formats such as TV. We explored the consumption of panoramic videos in three different display systems: a HMD, a SurroundVideo+ (SV+), and a standard 16:9 TV. The SV+ display features a TV with projected peripheral content. A between-groups experiment of 63 participants was conducted, in which participants watched panoramic videos in one of these three display conditions. Aspects examined in the experiment were spatial awareness, narrative engagement, enjoyment, memory, fear, attention, and a viewer's concern about missing something. Our results indicated that the HMD offered a significant benefit in terms of enjoyment and spatial awareness, and our SV+ display offered a significant improvement in enjoyment over traditional TV. We were unable to confirm the work of a previous study that showed incidental memory may be lower in a HMD over a TV. Drawing attention and a viewer's concern about missing something were also not significantly different between display conditions. It is clear that passive media viewing consists of a complex interplay of factors, such as the media itself, the characteristics of the display, as well as human aspects including perception and attention. While passive media viewing presents many challenges for evaluation, identifying a number of broadly applicable metrics will aid our understanding of these experiences, and allow the creation of better, more engaging CVR content and displays.

Video see-through augmented reality (VSAR) is an effective way of combing real and virtual scenes for head-mounted human computer interfaces. In this paper we present the AR-Rift 2 system, a cost-effective prototype VSAR system based around the Oculus Rift CV1 head-mounted display (HMD). Current consumer camera systems however typically have latencies far higher than the rendering pipeline of current consumer HMDs. They also have lower update rate than the display. We thus measure the latency of the video and implement a simple image-warping method to ensure smooth movement of the video.

We aim to further explore the transfer of spatial knowledge from virtual to real spaces. Based on previous research on spatial memory in immersive virtual reality (VR) we ran a study that looked at the effect of three locomotion techniques (joystick, pointing-and-teleporting and walking-in-place) on object location learning and recall. Participants were asked to learn the location of a virtual object in a virtual environment (VE). After a short period of time they were asked to recall the location by placing a real version of the object in the real-world equivalent environment. Results indicate that the average placement error, or distance between original and recalled object location, is approximately 20cm for all locomotion technique conditions. This result is similar to the outcome of a previous study on spatial memory in VEs that used real walking. We report this unexpected finding and suggest further work on spatial memory in VR by recommending the replication of this study in different environments and using objects with a wider diversity of properties, including varying sizes and shapes.

Properly handling occlusion between real and virtual objects is an important property for any mixed reality (MR) system. Existing methods have typically required known geometry of the real objects in the scene, either specified manually, or reconstructed using a dense mapping algorithm. This limits the situations in which they can be applied. Modern RGBD cameras are cheap and widely available, but the depth information they provide is typically too noisy and incomplete to use directly to provide quality results.

The term big data occurs more frequently now than ever before. A large number of fields and subjects, ranging from everyday life to traditional research fields (i.e., geography and transportation, biology and chemistry, medicine and rehabilitation), involve big data problems. The popularizing of various types of network has diversified types, issues, and solutions for big data more than ever before. In this paper, we review recent research in data types, storage models, privacy, data security, analysis methods, and applications related to network big data. Finally, we summarize the challenges and development of big data to predict current and future trends.

Modern virtual reality simulations require a constant high-frame rate from the rendering engine. They may also require very low latency and stereo images. Previous rendering engines for virtual reality applications have exploited spatial and temporal coherence by using image-warping to re-use previous frames or to render a stereo pair at lower cost than running the full render pipeline twice. However these previous approaches have shown artifacts or have not scaled well with image size. We present a new image-warping algorithm that has several novel contributions: an adaptive grid generation algorithm for proxy geometry for image warping; a low-pass hole-filling algorithm to address un-occlusion; and support for transparent surfaces by efficiently ray casting transparent fragments stored in per-pixel linked lists of an A-Buffer. We evaluate our algorithm with a variety of challenging test cases. The results show that it achieves better quality image-warping than state-of-the-art techniques and that it can support transparent surfaces effectively. Finally, we show that our algorithm can achieve image warping at rates suitable for practical use in a variety of applications on modern virtual reality equipment.

Immersive displays for virtual reality systems can be roughly classified into spatially immersive displays (similar to CAVE-like displays or large-screen simulators) or head-mounted displays. The former type is usually static in spatial configuration and configured to support a small group of users. The latter supports only a single user. We propose a new class of actuated, reconfigurable display that can support both small groups and individual users: in particular we suggest a robotic display that can change shape. The display can change shape to support different usage conditions, and can also move rapidly to give a larger apparent field of view for an individual user. We explore the potential advantages of a display that can move independently from its user(s), and we present a prototype that demonstrates some of the potential use scenarios.

The use of a self-avatar inside an immersive virtual reality system has been shown to have important effects on presence, interaction and perception of space. Based on studies from linguistics and cognition, in this paper we demonstrate that a self-avatar may aid the participant's cognitive processes while immersed in a virtual reality system. In our study participants were asked to memorise pairs of letters, perform a spatial rotation exercise and then recall the pairs of letters. In a between-subject factor they either had an avatar or not, and in a within-subject factor they were instructed to keep their hands still or not. We found that participants who both had an avatar and were allowed to move their hands had significantly higher letter pair recall. There was no significant difference between the other three conditions. Further analysis showed that participants who were allowed to move their hands, but could not see the self-avatar, usually didn't move their hands or stopped moving their hands after a short while. We argue that an active self-avatar may alleviate the mental load of doing the spatial rotation exercise and thus improve letter recall. The results are further evidence of the importance of an appropriate self-avatar representation in immersive virtual reality.

Latency is detrimental to interactive systems, especially pseudo-physical systems that emulate real-world behaviour. It prevents users from making quick corrections to their movement, and causes their experience to deviate from their expectations. Latency is a result of the processing and transport delays inherent in current computer systems. As such, while a number of studies have hypothesized that any latency will have a degrading effect, few have been able to test this for latencies less than ∼ 50 ms. In this study we investigate the effects of latency on pointing and steering tasks. We design an apparatus with a latency lower than typical interactive systems, using it to perform interaction tasks based on Fitts's law and the Steering law. We find evidence that latency begins to affect performance at ∼ 16 ms, and that the effect is non-linear. Further, we find latency does not affect the various components of an aiming motion equally. We propose a three stage characterisation of pointing movements with each stage affected independently by latency. We suggest that understanding how users execute movement is essential for studying latency at low levels, as high level metrics such as total movement time may be misleading.

Latency - the delay between a user's action and the response to this action - is known to be detrimental to virtual reality. Latency is typically considered to be a discrete value characterising a delay, constant in time and space - but this characterisation is incomplete. Latency changes across the display during scan-out, and how it does so is dependent on the rendering approach used. In this study, we present an ultra-low latency real-time ray-casting renderer for virtual reality, implemented on an FPGA. Our renderer has a latency of ~1 ms from 'tracker to pixel'. Its frameless nature means that the region of the display with the lowest latency immediately follows the scan-beam. This is in contrast to frame-based systems such as those using typical GPUs, for which the latency increases as scan-out proceeds. Using a series of high and low speed videos of our system in use, we confirm its latency of ~1 ms. We examine how the renderer performs when driving a traditional sequential scan-out display on a readily available HMO, the Oculus Rift OK2. We contrast this with an equivalent apparatus built using a GPU. Using captured human head motion and a set of image quality measures, we assess the ability of these systems to faithfully recreate the stimuli of an ideal virtual reality system - one with a zero latency tracker, renderer and display running at 1 kHz. Finally, we examine the results of these quality measures, and how each rendering approach is affected by velocity of movement and display persistence. We find that our system, with a lower average latency, can more faithfully draw what the ideal virtual reality system would. Further, we find that with low display persistence, the sensitivity to velocity of both systems is lowered, but that it is much lower for ours.

Consumer virtual reality systems are now becoming widely available. We report on a study on presence and embodiment within virtual reality that was conducted 'in the wild', in that data was collected from devices owned by consumers in uncontrolled settings, not in a traditional laboratory setting. Users of Samsung Gear VR and Google Cardboard devices were invited by web pages and email invitation to download and run an app that presented a scenario where the participant would sit in a bar watching a singer. Each participant saw one of eight variations of the scenario: with or without a self-avatar; singer inviting the participant to tap along or not; singer looking at the participant or not. Despite the uncontrolled situation of the experiment, results from an in-app questionnaire showed tentative evidence that a self-avatar had a positive effect on self-report of presence and embodiment, and that the singer inviting the participant to tap along had a negative effect on self-report of embodiment. We discuss the limitations of the study and the platforms, and the potential for future open virtual reality experiments.

Reconfigurable hardware has been used before for low latency image synthesis. These are typically low level implementations with tight vertical integration. For example the apparatus of both Regan et al and Ng et al had the tracker driven by the same device performing the rendering. Reconfigurable hardware combined with the dataflow programming model can make application specific rendering hardware cost effective. Our sprite renderer has comparable scope to both prior examples, but our dataflow graph can be adapted to other use cases with an effort comparable to GPU shader programming.

Abstract Telepresence involves the use of virtual reality technology to facilitate apparent physical participation in distant events, including potentially performing tasks, while creating a sense of being in that location. Traditionally, such systems are asymmetric in nature where only one side (participant) is “teleported” to the remote location. In this manuscript, the authors explore the possibility of symmetric three‐dimensional telepresence where both sides (participants) are “teleported” simultaneously to each other's location; the overarching concept of symmetric telepresence in virtual environments is extended to telepresence robots in physical environments. Two identical physical humanoid robots located in UK and the USA serve as surrogates while performing a transcontinental shared collaborative task. The actions of these surrogate robots are driven by capturing the intent of the participants controlling them in either location. Participants could communicate verbally but could not see the other person or the remote location while performing the task. The effectiveness of gesturing along with other observations during this preliminary experiment is presented. Results reveal that the symmetric robotic telepresence allowed participants to use and understand gestures in cases where they would otherwise have to describe their actions verbally. Copyright © 2015 John Wiley & Sons, Ltd.

Gaze, attention, and eye contact are important aspects of face to face communication, but some subtleties can be lost in videoconferencing because participants look at a single planar image of the remote user. We propose a low-cost cylindrical videoconferencing system that preserves gaze direction by providing perspective-correct images for multiple viewpoints around a conference table. We accomplish this by using an array of cameras to capture a remote person, and an array of projectors to present the camera images onto a cylindrical screen. The cylindrical screen reflects each image to a narrow viewing zone. The use of such a situated display allows participants to see the remote person from multiple viewing directions. We compare our system to three alternative display configurations. We demonstrate the effectiveness of our system by showing it allows multiple participants to simultaneously tell where the remote person is placing their gaze.

We report on two experiments that investigate the influence of display type and viewing angle on how people place their trust during avatar-mediated interaction. By monitoring advice seeking behavior, our first experiment demonstrates that if participants observe an avatar at an oblique viewing angle on a flat display, they are less able to discriminate between expert and non-expert advice than if they observe the avatar face-on. We then introduce a novel spherical display and a ray-traced rendering technique that can display an avatar that can be seen correctly from any viewing direction. We expect that a spherical display has advantages over a flat display because it better supports non-verbal cues, particularly gaze direction, since it presents a clear and undistorted viewing aspect at all angles. Our second experiment compares the spherical display to a flat display. Whilst participants can discriminate expert advice regardless of display, a negative bias towards the flat screen emerges at oblique viewing angles. This result emphasizes the ability of the spherical display to be viewed qualitatively similarly from all angles. Together the experiments demonstrate how trust can be altered depending on how one views the avatar.

Non-photorealistic rendering (NPR) has been shown as a powerful way to enhance both visual coherence and immersion in augmented reality (AR). However, it has only been evaluated in idealized pre-rendered scenarios with handheld AR devices. In this paper we investigate the use of NPR in an immersive, stereoscopic, wide field-of-view head-mounted video see-through AR display. This is a demanding scenario, which introduces many real-world effects including latency, tracking failures, optical artifacts and mismatches in lighting. We present the AR-Rift, a low-cost video see-through AR system using an Oculus Rift and consumer webcams. We investigate the themes of consistency and immersion as measures of psychophysical non-mediation. An experiment measures discernability and presence in three visual modes: conventional (unprocessed video and graphics), stylized (edge-enhancement) and virtualized (edge-enhancement and color extraction). The stylized mode results in chance-level discernability judgments, indicating successful integration of virtual content to form a visually coherent scene. Conventional and virutalized rendering bias judgments towards correct or incorrect respectively. Presence as it may apply to immersive AR, and which, measured both behaviorally and subjectively, is seen to be similarly high over all three conditions.

Abstract We present a novel tool for reverse engineering of modeling histories from consecutive 3D files based on a timeline abstraction. Although a timeline interface is commonly used in 3D modeling packages for animations, it has not been used on geometry manipulation before. Unlike previous visualization methods that require instrumentation of editing software, our approach does not rely on pre‐recorded editing instructions. Instead, each stand‐alone 3D file is treated as a keyframe of a construction flow from which the editing provenance is reverse engineered. We evaluate this tool on six complex 3D sequences created in a variety of modeling tools by different professional artists and conclude that it provides useful means of visualizing and understanding the editing history. A comparative user study suggests the tool is well suited for this purpose.

Latency of interactive computer systems is a product of the processing, transport and synchronisation delays inherent to the components that create them. In a virtual environment (VE) system, latency is known to be detrimental to a user's sense of immersion, physical performance and comfort level. Accurately measuring the latency of a VE system for study or optimisation, is not straightforward. A number of authors have developed techniques for characterising latency, which have become progressively more accessible and easier to use. In this paper, we characterise these techniques. We describe a simple mechanical simulator designed to simulate a VE with various amounts of latency that can be finely controlled (to within 3ms). We develop a new latency measurement technique called Automated Frame Counting to assist in assessing latency using high speed video (to within 1ms). We use the mechanical simulator to measure the accuracy of Steed's and Di Luca's measurement techniques, proposing improvements where they may be made. We use the methods to measure latency of a number of interactive systems that may be of interest to the VE engineer, with a significant level of confidence. All techniques were found to be highly capable however Steed's Method is both accurate and easy to use without requiring specialised hardware.

We present PanoInserts: a novel teleconferencing system that uses smartphone cameras to create a surround representation of meeting places. We take a static panoramic image of a location into which we insert live videos from smartphones. We use a combination of marker- and image-based tracking to position the video inserts within the panorama, and transmit this representation to a remote viewer. We conduct a user study comparing our system with fully-panoramic video and conventional webcam video conferencing for two spatial reasoning tasks. Results indicate that our system performs comparably with fully-panoramic video, and better than webcam video conferencing in tasks that require an accurate surrounding representation of the remote space. We discuss the representational properties and usability of varying video presentations, exploring how they are perceived and how they influence users when performing spatial reasoning tasks.

Within mobile mixed reality experiences, we would like to engage the user's head and hands for interaction. However, this requires the use of multiple tracking systems. These must be aligned, both as part of initial system setup and to counteract inter-tracking system drift that can accumulate over time. Traditional approaches to alignment use obtrusive procedures that introduce explicit constraints between the different tracking systems. These can be highly disruptive for the user's experience. In this paper, we propose another type of information which can be exploited to effect alignment: the behaviour of the user. The crucial insight is that user behaviours - such as selection through pointing - introduce implicit constraints between tracking systems. These constraints can be used as the user continually interacts with the system to infer alignment without the need for disruptive procedures. We call this concept behaviour-aware sensor fusion. We introduce two different interaction techniques-the redirected pointing technique and the yaw fix technique - to illustrate this concept. Pilot experiments show that behaviour-aware sensor fusion can increase ease of use and speed of interaction in exemplar mixed-reality interaction tasks.

Visual masking is a technique whereby the rapid presentation of an image can render another image unavailable to conscious perception. Many of its parameters have been studied but the stimuli have traditionally tended to be two dimensional in form, including those used in masked priming experiments. Recently, however, studies have begun to look at visual masking within stereoscopic, virtual environments (VEs). This paper outlines two experiments that alter various properties of visually masked, virtual objects. The first experiment found a significant interaction between surface-colour saturation and texture fluency. The second found an exposure effect modulating affective, visually masked stimuli. Using visually masked, affect-laden stimuli within a VE holds out the promise of creating a potent way to alter affective processes and behaviour. The potential of such effects is widespread, including use within psychotherapeutic interventions, training scenarios and video games.

In the BEAMING project we have been extending the scope of collaborative mixed reality to include the representation of users in multiple modalities, including augmented reality, situated displays and robots. A single user (a visitor) uses a high-end virtual reality system (the transporter) to be virtually teleported to a real remote location (the destination). The visitor may be tracked in several ways including emotion and motion capture. We reconstruct the destination and the people within it (the locals). In achieving this scenario, BEAMING has integrated many heterogeneous systems. In this paper, we describe the design and key implementation choices in the Beaming Scene Service (BSS), which allows the various processes to coordinate their behaviour. The core of the system is a light-weight shared object repository that allows loose coupling between processes with very different requirements (e.g. embedded control systems through to mobile apps). The system was also extended to support the notion of presence awareness. We demonstrate two complex applications built with the BSS.

This paper explores body ownership and control of an 'extended' humanoid avatar that features a distinct and flexible tail-like appendage protruding from its coccyx. Thirty-two participants took part in a between-groups study to puppeteer the avatar in an immersive CAVE™ -like system. Participants' body movement was tracked, and the avatar's humanoid body synchronously reflected this motion. However, sixteen participants experienced the avatar's tail moving around randomly and asynchronous to their own movement, while the other participants experienced a tail that they could, potentially, control accurately and synchronously through hip movement. Participants in the synchronous condition experienced a higher degree of body ownership and agency, suggesting that visuomotor synchrony enhanced the probability of ownership over the avatar body despite of its extra-human form. Participants experiencing body ownership were also more likely to be more anxious and attempt to avoid virtual threats to the tail and body. The higher task performance of participants in the synchronous condition indicates that people are able to quickly learn how to remap normal degrees of bodily freedom in order to control virtual bodies that differ from the humanoid form. We discuss the implications and applications of extended humanoid avatars as a method for exploring the plasticity of the brain's representation of the body and for gestural human-computer interfaces.

An emerging form of telecollaboration utilizes situated or mobile displays at a physical destination to virtually represent remote visitors. An example is a personal telepresence robot, which acts as a physical proxy for a remote visitor, and uses cameras and microphones to capture its surroundings, which are transmitted back to the visitor. We propose the use of spherical displays to represent telepresent visitors at a destination. We suggest that the use of such 360 degree displays in a telepresence system has two key advantages: it is possible to understand the identity of the visitor from any viewpoint; and with suitable graphical representation, it is possible to tell where the visitor is looking from any viewpoint. In this paper, we investigate how to optimally represent a visitor as an avatar on a spherical display by evaluating how varying representations are able to accurately convey head gaze.

In the context of scientific data analysis, we propose to compare a remote collaborative manipulation technique with a single user manipulation technique. The manipulation task consists in positioning a clipping plane in order to perform cross-sections of scientific data that show several points of interest located inside these data. For the remote collaborative manipulation, we have chosen to use the 3-hand manipulation technique proposed by Aguerreche et al., which is very suitable with a remote manipulation of a plane. We ran two experiments to compare the two manipulation techniques with some participants located in two different countries. These experiments has shown that the remote collaborative manipulation technique was significantly more efficient than the single user manipulation when the 3 points of interest were far apart inside the scientific data and, consequently, when the manipulation task was more difficult and required more precision. When the 3 points of interest were close together, there was not significant difference between the two manipulation techniques.

Additive manufacturing (3D printing) is commonly used to produce physical models for a wide variety of applications, from archaeology to design. While static models are directly supported, it is desirable to also be able to print models with functional articulations, such as a hand with joints and knuckles, without the need for manual assembly of joint components. Apart from having to address limitations inherent to the printing process, this poses a particular challenge for articulated models that should be posable: to allow the model to hold a pose, joints need to exhibit internal friction to withstand gravity, without their parts fusing during 3D printing. This has not been possible with previous printable joint designs. In this paper, we propose a method for converting 3D models into printable, functional, non-assembly models with internal friction. To this end, we have designed an intuitive work-flow that takes an appropriately rigged 3D model, automatically fits novel 3D-printable and posable joints, and provides an interface for specifying rotational constraints. We show a number of results for different articulated models, demonstrating the effectiveness of our method.

Increasingly, the computer graphics applications we use at work and play are exploiting and supporting real-time interaction in networking. Our course will introduce attendees to recent advances and best practice in the networking of graphics applications. We take a broad view of networked graphics, including the domains of network games, virtual reality and networked simulations. We start by noting why networked graphics applications have different requirements on the network as compared to "normal" applications. We then pay particular attention to issues of latency, bandwidth and scalability. The course is very timely; networking has been acknowledged in SIGGRAPH's own materials as important for application domains ranging from games through training to visualisation. Most existing tutorial material was developed before the age of commonly available broadband and wireless technologies. These technologies have enabled different types of environment and technology strategies, from massive social worlds with thousands of users through to fast-paced games on peer-to-peer networks. Furthermore, with the resurgence of thin-client systems such as Onlive™, the field is ripe for further innovation.

ABSTRACT Autonomous virtual characters (AVCs) are becoming more prevalent both for real‐time interaction and also as digital actors in film and TV production. AVCs require believable virtual human animations, accompanied by natural attention generation, and thus the software that controls the AVCs needs to model when and how to interact with the objects and other characters that exist in the virtual environment. This paper models automatic attention behaviour using a saliency model that generates plausible targets for combined gaze and head motions. The model was compared with the default behaviour of the Second Life (SL) system in an object observation scenario while it was compared with real actors' behaviour in a conversation scenario. Results from a study run within the SL system demonstrate a promising attention model that is not just believable and realistic but also adaptable to varying task, without any prior knowledge of the virtual scene. Copyright © 2011 John Wiley & Sons, Ltd.

Abstract We present a method for simulating local water waves caused by obstacles in water streams for real‐time graphics applications. Given a low‐resolution water surface and velocity field, our method is able to decorate the input water surface with high resolution detail for the animated waves around obstacles. We construct and animate a vector representation of the waves. It is then converted to feature‐aligned meshes for capturing the surfaces of the waves. Results demonstrate that our method has the benefits of real‐time performance and easy controllability. The method also fits well into a state‐of‐the‐art river animation system. Copyright © 2011 John Wiley & Sons, Ltd.

The conveyance and recognition of affect and emotion partially determine how people interact with others and how they carry out and perform in their day-to-day activities. Hence, it is becoming necessary to endow technology with the ability to recognize users' affective states to increase the technologies' effectiveness. This paper makes three contributions to this research area. First, we demonstrate recognition models that automatically recognize affective states and affective dimensions from non-acted body postures instead of acted postures. The scenario selected for the training and testing of the automatic recognition models is a body-movement-based video game. Second, when attributing affective labels and dimension levels to the postures represented as faceless avatars, the level of agreement for observers was above chance level. Finally, with the use of the labels and affective dimension levels assigned by the observers as ground truth and the observers' level of agreement as base rate, automatic recognition models grounded on low-level posture descriptions were built and tested for their ability to generalize to new observers and postures using random repeated subsampling validation. The automatic recognition models achieve recognition percentages comparable to the human base rates as hypothesized.

The four papers in this special section are expanded versions of the four best papers from the IEEE Virtual Reality (VR) Proceedings.

The success of visual telecommunication systems depends on their ability to transmit and display users' natural nonverbal behavior. While video-mediated communication (VMC) is the most widely used form of interpersonal remote interaction, avatar-mediated communication (AMC) in shared virtual environments is increasingly common. This paper presents two experiments investigating eye tracking in AMC. The first experiment compares the degree of social presence experienced in AMC and VMC during truthful and deceptive discourse. Eye tracking data (gaze, blinking, and pupil size) demonstrates that oculesic behavior is similar in both mediation types, and uncovers systematic differences between truth telling and lying. Subjective measures show users' psychological arousal to be greater in VMC than AMC. The second experiment demonstrates that observers of AMC can more accurately detect truth and deception when viewing avatars with added oculesic behavior driven by eye tracking. We discuss implications for the design of future visual telecommunication media interfaces.

We present a novel integration of a brain-computer interface (BCI) with a multi-touch surface. BCIs based on the P300 paradigm often use a visual stimulus of a flashing character to elicit an event related potential in the brain's EEG signal. Traditionally, P300-based BCI paradigms use a grid layout of visual targets, commonly an alphabet, and allow users to select targets using their thoughts. In our new system a multi-touch table senses objects placed upon its surface and the system can highlight the objects on the table by flashing an area of light around them. This allows us to construct a P300-based BCI that uses a user-assembled collection of objects as targets, rather than a pre-determined grid layout. An experiment shows that our new paradigm works just as well as the traditional paradigms, thus highlighting the potential for BCIs to be integrated in a broader range of situations.

The rubber hand illusion is a simple illusion where participants can be induced to report and behave as if a rubber hand is part of their body. The induction is usually done by an experimenter tapping both a rubber hand prop and the participant's real hand: the touch and visual feedback of the taps must be synchronous and aligned to some extent. The illusion is usually tested by several means including a physical threat to the rubber hand. The response to the threat can be measured by galvanic skin response (GSR): those that have the illusion showed a marked rise in GSR. Based on our own and reported experiences with immersive virtual reality (IVR), we ask whether a similar illusion is induced naturally within IVR? Does the participant report and behave as if the virtual arm is part of their body? We show that participants in a HMD-based IVR who see a virtual body can experience similar responses to threats as those in comparable rubber hand illusion experiments. We show that these responses can be negated by replacing the virtual body with an abstract cursor representing the hand, and that the responses are stable under some gradual forced distortion of tracker space so that proprioceptive and visual information are not matched.

Abstract When compared to gaze, animation of the eyelids has been largely overlooked in the computer graphics literature. Eyelid movement plays an important part both in conveying accurate gaze direction and in improving the visual appearance of virtual characters. Eyelids have two major motion components: lid saccades that follow the vertical rotation the eyes, and blinking. Derived from literature in ophthalmology and psychology, this paper presents parametric models for both motion types, and emphasizes their dynamic temporal behaviour. Experimental validation classifies model‐generated animation as similar to that encoded from expensive motion captured data, and significantly exceeding linearly interpolated animation. Copyright © 2010 John Wiley & Sons, Ltd.

Immersive virtual environments (IVEs) allow participants to interact with their 3D surroundings using natural hand gestures. Previous work shows that the addition of haptic feedback cues improves performance on certain 3D tasks. However, we believe this is not true for all situations. Depending on the difficulty of the task, we suggest that we should expect differences in the ballistic movement of our hands when presented with different types of haptic force-feedback conditions. We investigated how hard, soft and no haptic force-feedback responses, experienced when in contact with the surface of an object, affected user performance on a task involving selection of multiple targets. To do this, we implemented a natural egocentric selection interaction technique by integrating a two-handed large-scale force-feedback device in to a CAVETM-like IVE system. With this, we performed a user study where we show that participants perform selection tasks best when interacting with targets that exert soft haptic force-feedback cues. For targets that have hard and no force-feedback properties, we highlight certain associated hand movement that participants make under these conditions, that we hypothesise reduce their performance.

Eye gaze is an important and widely studied non-verbal resource in co-located social interaction. When we attempt to support tele-presence between people, there are two main technologies that can be used today: video-conferencing (VC) and collaborative virtual environments (CVEs). In VC, one can observe eye-gaze behaviour but practically the targets of eye-gaze are only correct if the participants remain relatively still. We attempt to support eye-gaze behaviour in an unconstrained manner by integrating eye-trackers into an Immersive CVE (ICVE) system. This paper aims to show that while both ICVE and VC allow people to discern being looked at and what else is looked at, when someone gazes into their space from another location, ICVE alone can continue to do this as people move. The conditions of aligned VC, ICVE, eye-gaze enabled ICVE and co-location are compared. The impact of factors of alignment, lighting, resolution, and perspective distortion are minimised through a set of pilot experiments, before a formal experiment records results for optimal settings. Results show that both VC and ICVE support eye-gaze in constrained situations, but only ICVE supports movement of the observer. We quantify the mis-judgements that are made and discuss how our findings might inform research into supporting eye-gaze through interpolated free viewpoint video based methods.

In order to increase a user's sense of presence in an artificial environment some researchers propose a gradual transition from reality to the virtual world instead of immersing users into the virtual world directly. One approach is to start the VR experience in a virtual replica of the physical space to accustom users to the characteristics of VR, e.g., latency, reduced field of view or tracking errors, in a known environment. Although this procedure is already applied in VR demonstrations, until now it has not been verified whether the usage of such a transitional environment - as transition between real and virtual environment - increases someone's sense of presence. We have observed subjective, physiological and behavioral reactions of subjects during a fully-immersive flight phobia experiment under two different conditions: the virtual flight environment was displayed immediately, or subjects visited a transitional environment before entering the virtual flight environment. We have quantified to what extent a gradual transition to the VE via a transitional environment increases the level of presence. We have found that subjective responses show significantly higher scores for the user's sense of presence, and that subjects' behavioral reactions change when a transitional environment is shown first. Considering physiological reactions, no significant difference could be found.

In face-to-face collaboration, eye gaze is used both as a bidirectional signal to monitor and indicate focus of attention and action, as well as a resource to manage the interaction. In remote interaction supported by immersive collaborative virtual environments (ICVEs), embodied avatars representing and controlled by each participant share a virtual space. We report on a study designed to evaluate methods of avatar eye gaze control during an object-focused puzzle scenario performed between three networked CAVEtrade-like systems. We compare tracked gaze, in which avatars' eyes are controlled by head-mounted mobile eye trackers worn by participants, to a gaze model informed by head orientation for saccade generation, and static gaze featuring non-moving eyes. We analyse task performance, subjective user experience, and interactional behaviour. While not providing statistically significant benefit over static gaze, tracked gaze is observed as the highest performing condition. However, the gaze model resulted in significantly lower task performance and increased error rate.

Complex interactions occur in virtual reality systems, requiring the modelling of next-generation attention models to obtain believable virtual human animations. This paper presents a saliency model that is neither domain nor task specific, which is used to animate the gaze of virtual characters. A critical question is addressed: What types of saliency attract attention in virtual environments and how can they be weighted to drive an avatar's gaze? Saliency effects were measured as a function of their total frequency. Scores were then generated for each object in the field of view within each frame to determine the most salient object within the virtual environment. This paper compares the resulting saliency gaze model to tracked gaze, in which avatars' eyes are controlled by head-mounted mobile eye-trackers worn by human subjects, random gaze model informed by head-orientation for saccade generation, and static gaze featuring non-moving centered eyes. Results from the evaluation experiment and graphical analysis demonstrate a promising saliency gaze model that is not just believable and realistic but also target-relevant and adaptable to varying tasks. Furthermore, the saliency model does not use any prior knowledge of the content or description of the virtual scene.

In this paper we profiled the performance of two types of 3D selections tasks: selection of one target and the selection of two targets. We designed an Immersive Virtual Environment (IVE) to evaluate any differences that may exist, and understand the underlying human behaviour by recording the hand movements' participants made when asked to select a series of 3D objects. To do this, we implemented a natural virtual hand-like interaction technique that participants could control using a large-scale force-feedback device placed into a CAVE™-like IVE system. We also investigated the interaction of no, soft and hard haptic force-feedback responses in addition to three target sizes on user performance. From the results obtained, we show distinct differences in the movement time taken when participants used their right hand to select one target in comparison to the selection of two targets.

Abstract For efficient collaboration between participants, eye gaze is seen as being critical for interaction. Video conferencing either does not attempt to support eye gaze (e.g. AcessGrid) or only approximates it in round table conditions (e.g. life size telepresence). Immersive collaborative virtual environments represent remote participants through avatars that follow their tracked movements. By additionally tracking people's eyes and representing their movement on their avatars, the line of gaze can be faithfully reproduced, as opposed to approximated. This paper presents the results of initial work that tested if the focus of gaze could be more accurately gauged if tracked eye movement was added to that of the head of an avatar observed in an immersive VE. An experiment was conducted to assess the difference between user's abilities to judge what objects an avatar is looking at with only head movements being displayed, while the eyes remained static, and with eye gaze and head movement information being displayed. The results from the experiment show that eye gaze is of vital importance to the subjects correctly identifying what a person is looking at in an immersive virtual environment. This is followed by a description of the work that is now being undertaken following the positive results from the experiment. We discuss the integration of an eye tracker more suitable for immersive mobile use and the software and techniques that were developed to integrate the user's real‐world eye movements into calibrated eye gaze in an immersive virtual world. This is to be used in the creation of an immersive collaborative virtual environment supporting eye gaze and its ongoing experiments. Copyright © 2009 John Wiley & Sons, Ltd.

The three papers in this special section are expanded versions of the three best papers from the IEEE VR 2008 proceedings.

Participants' eye-gaze is generally not captured or represented in immersive collaborative virtual environment (ICVE) systems. We present EyeCVE, which uses mobile eye-trackers to drive the gaze of each participant's virtual avatar, thus supporting remote mutual eye-contact and awareness of others' gaze in a perceptually unfragmented shared virtual workspace. We detail trials in which participants took part in three-way conferences between remote CAVE™ systems linked via EyeCVE. Eye-tracking data was recorded and used to evaluate interaction, confirming the system's support for the use of gaze as a communicational and management resource in multiparty conversational scenarios. We point toward subsequent investigation of eye-tracking in ICVEs for enhanced remote social-interaction and analysis.

A desktop tool for replay and analysis of gaze-enhanced multiparty virtual collaborative sessions is described. We linked three CAVETM-like environments, creating a multiparty collaborative virtual space where avatars are animated with 3D gaze as well as head and hand motions in real time. Log files are recorded for subsequent playback and analysis using the proposed software tool. During replaying the user can rotate the viewpoint and navigate in the simulated 3D scene. The playback mechanism relies on multiple distributed log files captured at every site. This structure enables an observer to experience latencies of movement and information transfer for every site as this is important for conversation analysis. Playback uses an event-replay algorithm, modified to allow fast traversal of the scene by selective rendering of nodes, and to simulate fast random access. The toolpsilas analysis module can show each participantpsilas 3D gaze points and areas where gaze has been concentrated.

Eye gaze is an important conversational resource that until now could only be supported across a distance if people were rooted to the spot. We introduce EyeCVE, the worldpsilas first tele-presence system that allows people in different physical locations to not only see what each other are doing but follow each otherpsilas eyes, even when walking about. Projected into each space are avatar representations of remote participants, that reproduce not only body, head and hand movements, but also those of the eyes. Spatial and temporal alignment of remote spaces allows the focus of gaze as well as activity and gesture to be used as a resource for non-verbal communication. The temporal challenge met was to reproduce eye movements quick enough and often enough to interpret their focus during a multi-way interaction, along with communicating other verbal and non-verbal language. The spatial challenge met was to maintain communicational eye gaze while allowing free movement of participants within a virtually shared common frame of reference. This paper reports on the technical and especially temporal characteristics of the system.

In collaborative virtual environments, the visual representation of avatars has been shown to be an important determinant of participant behaviour and response. We explored the influence of varying conditions of eye-representation in our high-fidelity avatar by measuring how accurately people can identify the avatar's point-of- regard (direction of gaze), together with subjective authenticity assessments of the avatar's behaviour and visual representation. The first of two variables investigated was socket-deformation, which is to say that our avatar's eyelids, eyebrows and surrounding areas morphed realistically depending on eye-rotation. The second was vergence of our avatar's eyes to the exact point-of-regard. Our results suggest that the two variables significantly influence the accuracy of point-of-regard identification. This accuracy is highly dependent on the combination of viewing-angle and the point-of-regard itself. We found that socket-deformation in particular has a highly positive impact on the perceived authenticity of our avatar's overall appearance, and when judging just the eyes. However, despite favourable subjective ratings, overall performance during the point-of-regard identification task was actually worse with the highest quality avatar. This provides more evidence that as we move forward to using higher fidelity avatars, there will be a tradeoff between supporting realism of representation and supporting the actual communicative task.

One of the critical determinants of the effectiveness and usability of interactive graphics simulations is the latency with which visual updates can be made based on input from interaction devices. High latency can diminish performance and can lead to simulator sickness. We demonstrate a new method for measuring latency using a standard video camera. The method is simple to configure, sensitive and rapid to use. This is in contrast to previous methods which required specialized equipment, were laborious or could only determine gross changes in latency. We attach a tracker to a pendulum and move a simulated image on the screen using the tracker positions. We video both the pendulum and simulated image together, and fit two sine curves, one to centre of motion of pendulum and one to the centre of motion of the simulated image. From the phase difference between these two sine curves we can determine latency changes significantly less than the frame rate of the camera. We demonstrate the method by comparing the latency of a two different systems for a CAVE™-like display.

The four papers in this special section focus on the field of virtual reality. The papers are summarized here.

This paper presents the quantitative and qualitative findings from an experiment designed to evaluate a developing model of affective postures for full-body virtual characters in immersive virtual environments (IVEs). Forty-nine participants were each requested to explore a virtual environment by asking two virtual characters for instructions. The participants used a CAVE-like system to explore the environment. Participant responses and their impression of the virtual characters were evaluated through a wide variety of both quantitative and qualitative methods. Combining a controlled experimental approach with various data-collection methods provided a number of advantages such as providing a reason to the quantitative results. The quantitative results indicate that posture plays an important role in the communication of affect by virtual characters. The qualitative findings indicated that participants attribute a variety of psychological states to the behavioral cues displayed by virtual characters. In addition, participants tended to interpret the social context portrayed by the virtual characters in a holistic manner. This suggests that one aspect of the virtual scene colors the perception of the whole social context portrayed by the virtual characters. We conclude by discussing the importance of designing holistically congruent virtual characters especially in immersive settings.

Many augmented reality (AR) applications use marker-based vision tracking systems to recover camera pose by detecting one or more planar landmarks. However, most of these systems do not interactively quantify the accuracy of the pose they calculate. Instead, the accuracy of these systems is either ignored, assumed to be a fixed value, or determined using error tables (constructed in an off-line ground-truthed process) along with a run-time interpolation scheme. The validity of these approaches are questionable as errors are strongly dependent on the intrinsic and extrinsic camera parameters and scene geometry. In this paper we present an algorithm for predicting the statistics of marker tracker error in real-time. Based on the scaled spherical simplex unscented transform (SSSUT), the algorithm is applied to the augmented reality toolkit plus (ARToolKitPlus). The results are validated using precision off-line photogrammetric techniques.

Abstract Previous works have shown that for characters with obvious articulation, animations comprising the blending of skeletal bone transformations, significantly reduce the computation and data requirements, permitting much greater visual detail. However increasingly, animators and engine designers must produce animations that can run on platforms with quite different runtime capabilities. They may simply author multiple skeletons and sets of animations, or reduce the temporal or spatial detail to achieve acceptable frame rates. In this paper we show how these skeletal skinning methods can be extended to support a continuous level of animation detail. We show that by manipulating the skeletal hierarchy and skinning parameters we can throttle the computational load of a character model in real‐time according to its position in a scene and any hardware constraints. Our method is compatible with additional geometric level of detail (LoD) methods. Copyright © 2007 John Wiley & Sons, Ltd.

In collaborative situations, eye gaze is a critical element of behavior which supports and fulfills many activities and roles. In current computer-supported collaboration systems, eye gaze is poorly supported. Even in a state-of-the-art video conferencing system such as the access grid, although one can see the face of the user, much of the communicative power of eye gaze is lost. This article gives an overview of some preliminary work that looks towards integrating eye gaze into an immersive collaborative virtual environment and assessing the impact that this would have on interaction between the users of such a system. Three experiments were conducted to assess the efficacy of eye gaze within immersive virtual environments. In each experiment, subjects observed on a large screen the eye-gaze behavior of an avatar. The eye-gaze behavior of that avatar had previously been recorded from a user with the use of a head-mounted eye tracker. The first experiment was conducted to assess the difference between users' abilities to judge what objects an avatar is looking at with only head gaze being viewed and also with eye- and head-gaze data being displayed. The results from the experiment show that eye gaze is of vital importance to the subjects, correctly identifying what a person is looking at in an immersive virtual environment. The second experiment examined whether a monocular or binocular eye-tracker would be required. This was examined by testing subjects' ability to identify where an avatar was looking from their eye direction alone, or by eye direction combined with convergence. This experiment showed that convergence had a significant impact on the subjects' ability to identify where the avatar was looking. The final experiment looked at the effects of stereo and mono-viewing of the scene, with the subjects being asked to identify where the avatar was looking. This experiment showed that there was no difference in the subjects' ability to detect where the avatar was gazing. This is followed by a description of how the eye-tracking system has been integrated into an immersive collaborative virtual environment and some preliminary results from the use of such a system.

The increasing power and ubiquity of mobile phones mean that a visitor to a city now carries with them a device capable of giving quite detailed guiding and routing information. Whilst there has been a lot of studies of text and map based guiding applications for mobile devices, in this paper we want to propose and give an initial exploratory study of a guiding system that utilises photographs. These photographs are not explicitly taken with the intention of using them subsequently for giving route directions; rather they are extracted from existing geo-tagged photo collections from mobile phones. A user of our system sees a route description as text and a map that refers to a series of photographs. The main contribution of this paper is in demonstrating this concept and testing it in an exploratory between-subjects experiment. The experiment shows that presenting the right photographs certainly can help with particular types of routing instruction for users not familiar with an area. For example, in unusual situation where the user has to walk through a specific gate or path, photographs provide information and reassurance about the navigation decision.

This paper presents a study of the fine grain physiological responses of participants to an immersive virtual simulation of an urban environment. The analysis of differences in participant responses at various stages of the experiment (baseline recordings, training, first half and second half of the urban simulation) are examined in detail. It was found that participants typically show a stress response during the training phase and a stress response towards the end of the simulation of the urban experience.There is also some evidence that variations in the level of visual realism based the texture strategy used was associated with changes in mental stress.

Some tasks vital to many mixed and augmented reality systems are either too time consuming or complex to be carried out whilst the system is active. 3D scene modelling and labelling are two such tasks commonly performed by skilled operators in an off-line initialisation phase. Because this phase sometimes needs specialist software and/or expertise it can be a considerable limiting factor for new mixed reality system developers. If a mixed reality system is to operate in real-time, where artificial graphics are woven into real world live images, the way in which these off-line processes are tackled is critical. In this paper we propose a flexible new approach that reduces the time spent during the off-line initialisation phase by adopting an on-line interactive primitive modelling technique. Our solution combines two existing and freely available packages, the Augmented Reality Toolkit Plus (ARToolKitPlus) and the Mixed Reality Toolkit (MRT), to enable rapid interactive modelling over live video using a freely moving camera. As a demonstration we show how these can be used to rapidly seed an object appearance-based tracking algorithm.

Collaboration at a distance has long been a research goal of distributed virtual environments.A number of recent technologies, including immersive projection technology systems (IPTs) and head-mounted displays (HMDs), promise a new generation of technologies that are more intuitive to use than desktop-based systems.This paper presents an experiment that compares collaboration in five different settings.Pairs collaborated on the same puzzle-solving task using one of: an IPT connected to another IPT, an IPT connected to an HMD, an IPT connected to a desktop system, two connected desktop systems, or face-to-face collaboration with real objects.The findings demonstrate the benefits of using immersive technologies, and show the advantages of using symmetrical settings for better performance.Some usability problems of the different distributed settings are addressed, as well as factors such as "presence" and "copresence" and how these contribute to the participants' overall experiences.

We present a scalable implementation of a network partitioning scheme that we have called frontier sets. Frontier sets build on the notion of a potentially visible set (PVS) [1][22]. In a PVS, a world is sub-divided into cells and for each cell all the other cells that can be seen are computed. In contrast, a frontier set considers pairs of cells, A and B. For eac`h pair, it lists two sets of cells, FAB and FBA. By definition, from no cell in FAB is any cell in FBA visible and vice-versa. Our initial use of frontier sets has been to enable scalability in distributed networking. In this paper we build on previous work by showing how to avoid pre-computing frontier sets. Our previous algorithm, required O(N 3) space in the number of cells, to store pre-computed frontier sets. Our new algorithm precomputes an enhanced potentially visible set that requires only O(N 2) space and then computes frontiers only as needed. Network simulations using code based on the Quake II engine show that frontiers have significant promise and may allow a new class of scalable peer-to-peer game infrastructures to emerge.

Many mixed-reality systems require real-time composition of virtual objects with real video. Such composition requires some description of the virtual and real scene geometries and calibration information for the real camera. Once these descriptions are available, they can be used to perform many types of visual simulation including virtual object placement, occlusion culling, texture extraction, collision detection and reverse and reillumination methods. In this paper we present a demonstration where we rapidly register prefabricated virtual models to a videoed scene. Using this registration information we were able to augment animated virtual avatars to create a novel mixed reality system. Rather than build a single monolithic system, we briefly introduce our lightweight modelling tool, the Mixed-Reality Toolkit (MRT) which enables rapid reconfiguration of scene objects without performing a full reconstruction. We also generalise to outline some initial requirements for a Mixed Reality Modelling Language (MRML). Copyright 2005 ACM.

Movements of interfaces can be analyzed in terms of whether they are expected, sensed, and desired. Expected movements are those that users naturally perform; sensed are those that can be measured by a computer; and desired movements are those that are required by a given application. We show how a systematic comparison of expected, sensed, and desired movements, especially with regard to how they do not precisely overlap, can reveal potential problems with an interface and also inspire new features. We describe how this approach has been applied to the design of three interfaces: pointing flashlights at walls and posters in order to play sounds; the Augurscope II, a mobile augmented reality interface for outdoors; and the Drift Table, an item of furniture that uses load sensing to control the display of aerial photographs. We propose that this approach can help to build a bridge between the analytic and inspirational approaches to design and can help designers meet the challenges raised by a diversification of sensing technologies and interface forms, increased mobility, and an emerging focus on technologies for everyday life.

Successfully staging a mixed reality game in which online players are chased through a virtual city by runners located in the real world requires extensive orchestration work. An ethnographic study shows how this concerted achievement extends beyond the control room to the runners on the street. This, in turn, suggests the need to 'decentralize' orchestration and develop support for collaboration 'on the ground'. The study leads to design proposals for orchestration interfaces for mobile experiences that augment situational awareness and surreptitious monitoring among mobile participants and support troubleshooting in situations where participants are disconnected or are unable to access positioning systems such as GPS.

A man walks into a bar and looks around. A bartender looks up and smiles quickly. The man notices, smiles back and heads over towards the bar. A very simple interaction between two people just took place, but the punch line of this example from a recent experiment in our CAVE-like system is that the bartender is a simulated agent. Never the less the man's response appears to be genuine: it seems that he acts to some extent, even if its only subconsciously, as if the bartender is real. Two friends meet in a collaborative virtual environment (CVE). Both are using CAVE-like immersive virtual environments which give them a 1-1 scale view of the world. They are both quite tall, but one comments that the other is looking quite short today. They laugh about this and then get down to the task in hand. The “shortness” is actually true in the virtual sense and is due to a tracker calibration problem in one of the CAVE-like systems. These two small examples highlight what a complex phenomena a CVE is. Today tens of thousands of people regularly use simple versions of such systems through online game technology. But their experience of interacting with other people is limited by the crude interfaces they must use. This talk investigates how good communication at a distance can be using the best available technology. We will look at the qualities of inter-personal communication that are possible in CVEs using a range of examples from recent experiments and trials. We will then analyze how distributed system technology supports and hinders various aspects of this

In this paper we discuss the use of models of space in the building of mixed-reality systems. By model of space we mean a geometric or symbolic description associated with a physical space. We outline several types of model that exist, how they are surveyed and authored, how they are represented to the users and how they are supported by middleware and sensors. We show that systems often contain numerous models of space and we discuss the issues in maintaining or reifying assumptions about transformations between models. We illustrate these ideas by describing the implementation of a collaborative mixed-reality system that allows users to experience a museum in three modalities: physically colocated visitor with personal digital assistant guide, virtual reality visitor and Web visitor.

Future teleconferencing may enhance communication between remote people by supporting non-verbal communication within an unconstrained space where people can move around and share the manipulation of artefacts. By linking walk-in displays with a Collaborative Virtual Environment (CVE) platform we are able to physically situate a distributed team in a spatially organised social and information context. We have found this to demonstrate unprecedented naturalness in the use of space and body during non-verbal communication and interaction with objects.However, relatively little is known about how people interact through this technology, especially while sharing the manipulation of objects. We observed people engaged in such a task while geographically separated across national boundaries. Our analysis is organised into collaborative scenarios, that each requires a distinct balance of social human communication with consistent shared manipulation of objects.Observational results suggest that walk-in displays do not suffer from some of the important drawbacks of other displays. Previous trials have shown that supporting natural non-verbal communication, along with responsive and consistent shared object manipulation, is hard to achieve. To better understand this problem, we take a close look at how the scenario impacts on the characteristics of event traffic. We conclude by suggesting how various strategies might reduce the consistency problem for particular scenarios.

Abstract This paper describes a behavioural model used to simulate realistic eye‐gaze behaviour and body animations for avatars representing participants in a shared immersive virtual environment (IVE). The model was used in a study designed to explore the impact of avatar realism on the perceived quality of communication within a negotiation scenario. Our eye‐gaze model was based on data and studies carried out on the behaviour of eye‐gaze during face‐to‐face communication. The technical features of the model are reported here. Information about the motivation behind the study, experimental procedures and a full analysis of the results obtained are given in [ 17 ].

Museums attract increasing numbers of online visitors along with their conventional physical visitors. This paper presents a study of a mixed reality system that allows web, virtual reality and physical visitors to share a museum visit together in real time. Our system allows visitors to share their location and orientation, communicate over a voice channel, and jointly navigate around a shared information space. Results from a study of 34 users of the system show that visiting with the system was highly interactive and retained many of the attractions of a traditional shared exhibition visit. Specifically, users could navigate together, collaborate around objects and discuss exhibits. These findings have implications for non-museum settings, in particular how location awareness is a powerful resource for collaboration, and how 'hybrid objects' can support collaboration at-a-distance.

This paper presents an experiment designed to investigate the impact of visual and behavioral realism in avatars on perceived quality of communication in an immersive virtual environment. Participants were paired by gender and were randomly assigned to a CAVE‘-like system or a head-mounted display. Both were represented by a humanoid avatar in the shared 3D environment. The visual appearance of the avatars was either basic and genderless (like a "match-stick" figure), or more photorealistic and gender-specific. Similarly, eye gaze behavior was either random or inferred from voice, to reflect different levels of behavioral realism. Our comparative analysis of 48 post-experiment questionnaires confirms earlier findings from non-immersive studies using semi-photorealistic avatars, where inferred gaze significantly outperformed random gaze. However responses to the lower-realism avatar are adversely affected by inferred gaze, revealing a significant interaction effect between appearance and behavior. We discuss the importance of aligning visual and behavioral realism for increased avatar effectiveness.

This study examines pairs of subjects who used networked immersive projection technology systems to collaborate on five tasks over an extended period of time (210+ minutes). The aim was to compare zero history and mutual history partners, to examine how their experience changed over time, and compare their experience of different tasks. Analysis yields a number of interesting findings for these comparisons. Overall, the study shows that users could collaborate effectively over an extended period of time, but that understanding the intentions and activities of the other person remained a hindrance.

We investigate several techniques that partition a crowded virtual environment into regions that can be managed by separate servers or mapped onto different multicast groups. When constructing a partitioning, we attempt to minimize overhead of the partitioning with respect to network management, whilst maintaining a bound on the number of entities that are mapped to any particular server or group.We compare several partitioning schemes: quad tree, k-d tree unconstrained, k-d tree constrained, and region growing. With our simulations of a crowded virtual environment modelled on a part of central London, we find that the region growing technique give the best overall results.

The research community is challenged to investigate the factors that make virtual reality technologies effective and productive. Realising the goals of virtual reality systems and harnessing them to successful applications can be accomplished only by employing robust metrics and formal experimentation. Subsequently, the quality of interfaces and platforms implemented could be assessed.

A study by Slater, et al., [1995] indicated that naive subjects in an immersive virtual environment experience a higher subjective sense of presence when they locomote by walking-in-place (virtual walking) than when they push-button-fly (along the floor plane).We replicated their study, adding real walking as a third condition.Our study confirmed their findings.We also found that real walking is significantly better than both virtual walking and flying in ease (simplicity, straightforwardness, naturalness) as a mode of locomotion.The greatest difference in subjective presence was between flyers and both kinds of walkers.In addition, subjective presence was higher for real walkers than virtual walkers, but the difference was statistically significant only in some models.Follow-on studies show virtual walking can be substantially improved by detecting footfalls with a head accelerometer.As in the Slater study, subjective presence significantly correlated with subjects' degree of association with their virtual bodies (avatars).This, our strongest statistical result, suggests that substantial potential presence gains can be had from tracking all limbs and customizing avatar appearance.An unexpected by-product was that real walking through our enhanced version of Slater's visual-cliff virtual environment (Figure 1) yielded a strikingly compelling virtual experience-the strongest we and most of our visitors have yet experienced.The most needed system improvement is the substitution of wireless technology for all links to the user.

We present an experiment that investigates the behaviour of small groups of participants in a wide-area distributed collaborative virtual environment (CVE). This is the third and largest study in a series of experiments that have examined trios of participants carrying out a highly collaborative puzzle-solving task. The results reproducing those of earlier studies suggest a positive relationship between place-presence and co-presence, between co-presence and group accord, with evidence supporting the notion that immersion confers leadership advantage.

In virtual reality simulations the speed of rendering is vitally important. One of the techniques for controlling the frame rate is the assignment of different levels of detail for each object within a scene. The most well-known level of detail assignment algorithms are the Funkhouser[1] algorithm and the algorithm where the level of detail is assigned with respect to the distance of the object from the viewer.

Travel can be a stressful experience and it is an activity that is difficult to prepare for in advance. Although maps, routes and landmarks can be memorised, travellers do not get much sense of the spatial layout of the destination and can easily get confused when they arrive. There is little doubt that virtual environments techniques can assist in such situations, by, for example, providing walkthroughs of virtual cityscapes to effect route learning.

Current virtual environment (VE) systems employ a number of techniques for navigation such & walking on a plane, or flying through space.When exploring complex environments such as building interiors, the navigable spaces might include bridges, steps and slopes, and in some applications it is desirable to keep the participant at a realistic height above the ground.The techniques described in this paper can track the surface point which the participant is above for a tiny computational cost.

This paper describes the creation of a Western shootout animation for a BBC2 television production. This provides a case study describing the interaction between the TV production team and the animators. The animation was produced by explicitly programming transformations applied to nodes in hierarchical models of simple human characters. A subsequent project has attempted to provide character animation making use of the human body tracking and immersion afforded by virtual reality systems. In the new system a human actor can enter into a virtual environment (VE), act out a role, resulting in a file containing a script. While the script is replayed the actor can enter again into the VE, and interact with the first virtual character, and so on for subsequent replays. The paper describes a first version of this system, and provides a critical examination of the use of VR for creating animations.>

Abstract Many applications now demand interaction with visualizations of 3D scenes and data sets. Current flat 2D displays are limited in their capacity to provide this not only by the display technology but the interaction metaphors and devices used. The Desktop Bat is a device that has 5 degrees of freedom whilst retaining the simplicity of use o fa mouse. To use it for general 3D interaction several metaphors were created for the tasks of navigation and cursor manipulation and a set of experiments were conducted to determine which metaphors were the most efficient in use. Of these metaphors, a velocity control metaphor was the best for navigation and a metaphor that applied rotations and translations relative to the eyepoint coordinate system was best for object control.

This article presents an interactive technique for moving through an immersive virtual environment (or “virtual reality”). The technique is suitable for applications where locomotion is restricted to ground level. The technique is derived from the idea that presence in virtual environments may be enhanced the stronger the match between proprioceptive information from human body movements and sensory feedback from the computer-generated displays. The technique is an attempt to simulate body movements associated with walking. The participant “walks in place” to move through the virtual environment across distances greater than the physical limitations imposed by the electromagnetic tracking devices. A neural network is used to analyze the stream of coordinates from the head-mounted display, to determine whether or not the participant is walking on the spot. Whenever it determines the walking behavior, the participant is moved through virtual space in the direction of his or her gaze. We discuss two experimental studies to assess the impact on presence of this method in comparison to the usual hand-pointing method of navigation in virtual reality. The studies suggest that subjective rating of presence is enhanced by the walking method provided that participants associate subjectively with the virtual body provided in the environment. An application of the technique to climbing steps and ladders is also presented.