This paper presents a mixed reality (MR) system that results from the integration of a telepresence system and an application to improve collaborative space exploration. The system combines free viewpoint video with immersive projection technology to support nonverbal communication (NVC), including eye gaze, interpersonal distance, and facial expression. Importantly, these features can be interpreted together as people move around the simulation, maintaining a natural social distance. The application is a simulation of Mars, within which the collaborators must come to agreement over; for example, where the Rover should land and go. The first contribution is the creation of an MR system supporting contextualization of NVC. Two technological contributions are prototyping a technique to subtract a person from a background that may contain physical objects and/or moving images and a lightweight texturing method for multiview rendering, which provides balance in terms of visual and temporal quality. A practical contribution is the demonstration of pragmatic approaches to sharing space between display systems of distinct levels of immersion. A research tool contribution is a system that allows comparison of conventional authored and video-based reconstructed avatars, within an environment that encourages exploration and social interaction. Aspects of system quality, including the communication of facial expression and end-to-end latency are reported.

An immersive virtual environment is the ideal platform for the planning and training of on-orbit servicing missions, as it provides a flexible and safe environment. In such kind of virtual assembly simulation, grasping virtual objects is one of the most common and natural interactions. However, unlike grasping objects in the real world, it is a non-trivial task in virtual environments, where the primary feedback is visual only. A lot of research investigated ways to provide haptic feedback, such as force, vibrational and electrotactile feedback. Such devices, however, are usually uncomfortable and hard to integrate in projection-based immersive YR systems. In this paper, we present a novel, small and lightweight electro-tactile feedback device, specifically designed for immersive virtual environments. It consists of a small tactor with eight electrodes for each finger and a signal generator attached to the user's hand or arm. Our device can be easily integrated with an existing optical finger tracking system. The study presented in this paper assesses the feasibility and usability of the interaction device. An experiment was conducted in a repeated measures design using the electrotactile feedback modality as independent variable. As benchmark, we chose three typical assembly tasks of a YR simulation for satellite on-orbit servicing missions, including pressing a button, switching a lever switch, and pulling a module from its slot. Results show that electrotactile feedback improved the user's grasping in our virtual on-orbit servicing scenario. The task completion time was significantly lower for all three tasks and the precision of the user's interaction was higher. The workload reported by the participants was significantly lower when using electrotactile feedback. Additionally, users were more confident with their performance while completing the tasks with electrotactile feedback. We describe the device, outline the user study and report the results.

We introduce the collaborative telepresence workspaces for SPACE operation and science that are under development in the European research project CROSS DRIVE. The vision is to give space mission controllers and scientists the impression of “beaming” to the surface of Mars, along with simulations of the environment and equipment, to step out together where a robot has or may move. We briefly overview the design and describe the state of the demonstrator. The contribution of the publication is to give an example of how collaborative Virtual Reality research is being taken up in space science.

The growth of space debris is becoming a serious problem. There is an urgent need for mitigation measures based on maintenance, repair and de-orbiting technologies. Our video presents a virtual reality framework in which robotic maintenance tasks of satellites can be simulated interactively. The two key components of this framework are a realistic virtual reality simulation and an immersive interaction device. The peculiarity of the virtual reality simulation is the combination of a physics engine based on Bullet with an extremely efficient haptic rendering algorithm inspired by an enhanced version of the Voxmap-Pointshell Algorithm. A central logic module controls all states and objects in the virtual world. To enable the human operator an optimal immersion into the virtual environment, the DLR bimanual haptic device is used as interaction device. Equipped with two light-weight robot arms, this device is able to provide realistic haptic feedback at both human hands, while covering the major part of human operator's workspace. The applicability of this system is enhanced by additional force sensors, active hand interfaces with an additional degree of freedom, smart safety technologies and intuitive robot data augmentation. Our platform can be used for verification or training purposes of robotic systems interacting in space environments.

We present results of a user study in which we compared three interaction methods for manipulating deformable objects in immersive virtual environments. The task was to control a virtual robot hand removing a thin foil cover from a satellite in an on-orbit servicing training simulator. The lack of haptic feedback placed a high challenge on the user when trying to apply the right force for grasping the foil without losing grip or damaging it. We compared the intuitiveness and effectiveness of using a tracked joystick, finger distance measurement, and a novel prototype enabling direct force input through pinching.

This paper outlines the development of a real-time interactive application for the analysis, training and programming of on-orbit servicing tasks within a virtual reality environment. The main challenges put on the system are the real-time simulation of the realistic dynamic and kinematic behavior of satellite components and additionally integrate interaction through a bimanual haptic interface, as well as enable tele-operation of a robot. We give an overview of the application, describe the real-time challenges and outline our approach and proposed system structure.

Latency in a communication system can result in confusing a conversation through loss of causality as people exchange verbal and non-verbal nuances. This paper compares true end-to-end latencies across an immersive virtual environment and a video conference link using the same approach to measure both. Our approach is to measure end-to-end latency through filming the movements of a participant and their remote representation through synchronised cameras. We also compare contemporary and traditional immersive display and capture devices, whilst also measuring event latency taken from log files. We compare an immersive collaborative virtual environment to a video conference as both attempt to reproduce different aspects of the face-to- face meeting, the former favouring appearance and the latter attention. Results inform not only the designers of both approaches but also set the requirements for future developments for 3D video which has the potential to faithfully reproduce both appearance and attention.

Eye gaze is an important and widely studied non-verbal resource in co-located social interaction. When we attempt to support tele-presence between people, there are two main technologies that can be used today: video-conferencing (VC) and collaborative virtual environments (CVEs). In VC, one can observe eye-gaze behaviour but practically the targets of eye-gaze are only correct if the participants remain relatively still. We attempt to support eye-gaze behaviour in an unconstrained manner by integrating eye-trackers into an Immersive CVE (ICVE) system. This paper aims to show that while both ICVE and VC allow people to discern being looked at and what else is looked at, when someone gazes into their space from another location, ICVE alone can continue to do this as people move. The conditions of aligned VC, ICVE, eye-gaze enabled ICVE and co-location are compared. The impact of factors of alignment, lighting, resolution, and perspective distortion are minimised through a set of pilot experiments, before a formal experiment records results for optimal settings. Results show that both VC and ICVE support eye-gaze in constrained situations, but only ICVE supports movement of the observer. We quantify the mis-judgements that are made and discuss how our findings might inform research into supporting eye-gaze through interpolated free viewpoint video based methods.

In face-to-face collaboration, eye gaze is used both as a bidirectional signal to monitor and indicate focus of attention and action, as well as a resource to manage the interaction. In remote interaction supported by immersive collaborative virtual environments (ICVEs), embodied avatars representing and controlled by each participant share a virtual space. We report on a study designed to evaluate methods of avatar eye gaze control during an object-focused puzzle scenario performed between three networked CAVEtrade-like systems. We compare tracked gaze, in which avatars' eyes are controlled by head-mounted mobile eye trackers worn by participants, to a gaze model informed by head orientation for saccade generation, and static gaze featuring non-moving eyes. We analyse task performance, subjective user experience, and interactional behaviour. While not providing statistically significant benefit over static gaze, tracked gaze is observed as the highest performing condition. However, the gaze model resulted in significantly lower task performance and increased error rate.

Abstract For efficient collaboration between participants, eye gaze is seen as being critical for interaction. Video conferencing either does not attempt to support eye gaze (e.g. AcessGrid) or only approximates it in round table conditions (e.g. life size telepresence). Immersive collaborative virtual environments represent remote participants through avatars that follow their tracked movements. By additionally tracking people's eyes and representing their movement on their avatars, the line of gaze can be faithfully reproduced, as opposed to approximated. This paper presents the results of initial work that tested if the focus of gaze could be more accurately gauged if tracked eye movement was added to that of the head of an avatar observed in an immersive VE. An experiment was conducted to assess the difference between user's abilities to judge what objects an avatar is looking at with only head movements being displayed, while the eyes remained static, and with eye gaze and head movement information being displayed. The results from the experiment show that eye gaze is of vital importance to the subjects correctly identifying what a person is looking at in an immersive virtual environment. This is followed by a description of the work that is now being undertaken following the positive results from the experiment. We discuss the integration of an eye tracker more suitable for immersive mobile use and the software and techniques that were developed to integrate the user's real‐world eye movements into calibrated eye gaze in an immersive virtual world. This is to be used in the creation of an immersive collaborative virtual environment supporting eye gaze and its ongoing experiments. Copyright © 2009 John Wiley & Sons, Ltd.

Participants' eye-gaze is generally not captured or represented in immersive collaborative virtual environment (ICVE) systems. We present EyeCVE, which uses mobile eye-trackers to drive the gaze of each participant's virtual avatar, thus supporting remote mutual eye-contact and awareness of others' gaze in a perceptually unfragmented shared virtual workspace. We detail trials in which participants took part in three-way conferences between remote CAVE™ systems linked via EyeCVE. Eye-tracking data was recorded and used to evaluate interaction, confirming the system's support for the use of gaze as a communicational and management resource in multiparty conversational scenarios. We point toward subsequent investigation of eye-tracking in ICVEs for enhanced remote social-interaction and analysis.

A desktop tool for replay and analysis of gaze-enhanced multiparty virtual collaborative sessions is described. We linked three CAVETM-like environments, creating a multiparty collaborative virtual space where avatars are animated with 3D gaze as well as head and hand motions in real time. Log files are recorded for subsequent playback and analysis using the proposed software tool. During replaying the user can rotate the viewpoint and navigate in the simulated 3D scene. The playback mechanism relies on multiple distributed log files captured at every site. This structure enables an observer to experience latencies of movement and information transfer for every site as this is important for conversation analysis. Playback uses an event-replay algorithm, modified to allow fast traversal of the scene by selective rendering of nodes, and to simulate fast random access. The toolpsilas analysis module can show each participantpsilas 3D gaze points and areas where gaze has been concentrated.

Eye gaze is an important conversational resource that until now could only be supported across a distance if people were rooted to the spot. We introduce EyeCVE, the worldpsilas first tele-presence system that allows people in different physical locations to not only see what each other are doing but follow each otherpsilas eyes, even when walking about. Projected into each space are avatar representations of remote participants, that reproduce not only body, head and hand movements, but also those of the eyes. Spatial and temporal alignment of remote spaces allows the focus of gaze as well as activity and gesture to be used as a resource for non-verbal communication. The temporal challenge met was to reproduce eye movements quick enough and often enough to interpret their focus during a multi-way interaction, along with communicating other verbal and non-verbal language. The spatial challenge met was to maintain communicational eye gaze while allowing free movement of participants within a virtually shared common frame of reference. This paper reports on the technical and especially temporal characteristics of the system.

Collaborative Virtual Environments (CVE) are a form of telecommunication technology that bring together co-located or remote, participants within a spatial social and information context. Collaboration occurs between people and often around shared objects. Fruitful cooperation is helped by natural and intuitive ways of communicating and sharing, for which responsiveness and consistency are leading factors. Many CVEs maximise local responsiveness through a process of localisation and database replication, increasing responsiveness at the cost of lowering consistency. This is acceptable provided the application does not require the shared manipulation of objects. Those that do, require consistency control that provide sufficient synchronisation, ordering and update control, whilst maximising concurrence and thus the responsiveness of the system. This paper describes the major issues and principles of consistency control and demonstrates how we have applied many of these principles in three CVEs.

Many teamwork tasks in the real world require the shared manipulation of objects, both sequentially and concurrently. Collaborative virtual environments (CVE) bring remote people together in an interactive, spatial social and information context that is distributed over a network. Supporting highly collaborative tasks in a natural manner requires a thourough understanding of event traffic as distinct forms of shared object manipulation are bound to diverse real-time and consistency constraints. This position paper summarises our findings during user trials around the shared manipulation of objects within linked immersive displays and introduce a CVE system with a flexible event-handling framework that allows task-oriented concictency control.

Future teleconferencing may enhance communication between remote people by supporting non-verbal communication within an unconstrained space where people can move around and share the manipulation of artefacts. By linking walk-in displays with a Collaborative Virtual Environment (CVE) platform we are able to physically situate a distributed team in a spatially organised social and information context. We have found this to demonstrate unprecedented naturalness in the use of space and body during non-verbal communication and interaction with objects.However, relatively little is known about how people interact through this technology, especially while sharing the manipulation of objects. We observed people engaged in such a task while geographically separated across national boundaries. Our analysis is organised into collaborative scenarios, that each requires a distinct balance of social human communication with consistent shared manipulation of objects.Observational results suggest that walk-in displays do not suffer from some of the important drawbacks of other displays. Previous trials have shown that supporting natural non-verbal communication, along with responsive and consistent shared object manipulation, is hard to achieve. To better understand this problem, we take a close look at how the scenario impacts on the characteristics of event traffic. We conclude by suggesting how various strategies might reduce the consistency problem for particular scenarios.