This paper presents the system architecture of a collaborative virtual environment in which distributed multidisciplinary teams involved in space exploration activities come together and explore areas of scientific interest of a planet for future missions. The aim is to reduce the current challenges of distributed scientific and engineering meetings that prevent the exploitation of their collaborative potential, as, at present, expertise, tools and datasets are fragmented. This paper investigates the functional characteristics of a software framework that addresses these challenges following the design science research methodology in the context of the space industry and research. An implementation of the proposed architecture and a validation process with end users, based on the execution of different use cases, are described. These use cases cover relevant aspects of real science analysis and operation, including planetary data visualization, as the system aims at being used in future European missions. This validation suggests that the system has the potential to enhance the way space scientists will conduct space science research in the future.

This paper presents a mixed reality (MR) system that results from the integration of a telepresence system and an application to improve collaborative space exploration. The system combines free viewpoint video with immersive projection technology to support nonverbal communication (NVC), including eye gaze, interpersonal distance, and facial expression. Importantly, these features can be interpreted together as people move around the simulation, maintaining a natural social distance. The application is a simulation of Mars, within which the collaborators must come to agreement over; for example, where the Rover should land and go. The first contribution is the creation of an MR system supporting contextualization of NVC. Two technological contributions are prototyping a technique to subtract a person from a background that may contain physical objects and/or moving images and a lightweight texturing method for multiview rendering, which provides balance in terms of visual and temporal quality. A practical contribution is the demonstration of pragmatic approaches to sharing space between display systems of distinct levels of immersion. A research tool contribution is a system that allows comparison of conventional authored and video-based reconstructed avatars, within an environment that encourages exploration and social interaction. Aspects of system quality, including the communication of facial expression and end-to-end latency are reported.

We introduce the collaborative telepresence workspaces for SPACE operation and science that are under development in the European research project CROSS DRIVE. The vision is to give space mission controllers and scientists the impression of “beaming” to the surface of Mars, along with simulations of the environment and equipment, to step out together where a robot has or may move. We briefly overview the design and describe the state of the demonstrator. The contribution of the publication is to give an example of how collaborative Virtual Reality research is being taken up in space science.

The aim of our experiment is to determine if eye-gaze can be estimated from a virtuality human: to within the accuracies that underpin social interaction; and reliably across gaze poses and camera arrangements likely in every day settings. The scene is set by explaining why Immersive Virtuality Telepresence has the potential to meet the grand challenge of faithfully communicating both the appearance and the focus of attention of a remote human participant within a shared 3D computer-supported context. Within the experiment n=22 participants rotated static 3D virtuality humans, reconstructed from surround images, until they felt most looked at. The dependent variable was absolute angular error, which was compared to that underpinning social gaze behaviour in the natural world. Independent variables were 1) relative orientations of eye, head and body of captured subject; and 2) subset of cameras used to texture the form. Analysis looked for statistical and practical significance and qualitative corroborating evidence. The analysed results tell us much about the importance and detail of the relationship between gaze pose, method of video based reconstruction, and camera arrangement. They tell us that virtuality can reproduce gaze to an accuracy useful in social interaction, but with the adopted method of Video Based Reconstruction, this is highly dependent on combination of gaze pose and camera arrangement. This suggests changes in the VBR approach in order to allow more flexible camera arrangements. The work is of interest to those wanting to support expressive meetings that are both socially and spatially situated, and particular those using or building Immersive Virtuality Telepresence to accomplish this. It is also of relevance to the use of virtuality humans in applications ranging from the study of human interactions to gaming and the crossing of the stage line in films and TV.

This paper reports a GPU based, projective texturing approach to reconstructing the human form, from multiple images, at a quality and frame rate close to high end video conferencing. The ultimate aim is to support spatially grounded, non-verbal communication through a video based medium. This will, we hope, enable us to balance image quality and update rate to deliver highly realistic and dynamic 3D human representations that offer the visual quality of high end video conferencing with the spatial and temporal characteristics of immersive virtual environments. The output of this will enhance communication by enabling a remote actor to be realistically projected into another person's local space, projected into an extension of the local space, or projected into a shared virtual space. This extends previous work by incorporating texture into the reconstructed form and evaluating the optimized process within our established simulation system.

Would a building be more appealing if it had interactive elements? Would that encourage people to visit it again, spend more time in it and promote it to others? Or would interactivity be a trivial presence in a building? Our previous study showed that people could concentrate better when surrounded by moving patterns that give an impression of moving walls. Whether people are more likely to return to architecture that is interactive and the appeal of interactive architecture over a non interactive counterpart has not been tested before. Before building such architecture we use simulation to gain an initial understanding. While our first study was carried out in a large surround screen projection environment, this study makes use of second life as a place where people can visit at will. Two identical buildings were designed for the experiment with one of them having interactive elements. They were then placed in an online virtual environment that is accessible by the public and they were monitored for four months. Initial results indicate that people prefer the interactive building over the non-interactive one. Appeal was measured by number of visitors, number of visits, time spent in each visit, number of groups that visited and returned to each building. We also used feedback provided by a separate focus group as well. The results demonstrate the importance of interactive elements in generating more interest and traffic for buildings and events.

We present an analysis of the requirement for input synchronisation in a multi-camera 3D reconstruction system for real-time applications such as telepresence. Synchronisation of the cameras at the acquisition stage is universally used to ensure the images feeding the reconstruction algorithm were taken at the same time. However, this requirement adds delays to the reconstruction pipeline, therefore increasing the end to end latency of the system. While this has not been a significant problem for many of the applications of 3D reconstruction, it is for its application to tele-presence. Furthermore, synchronising the firing of cameras adds much financial cost to the system. Using real camera images of moving humans, we study the effect removing synchronisation has on the output reconstructed model over a range of camera configurations and relative frame delays. From this we determine the synchronisation requirements for a 3D reconstruction telepresence system in terms of the maximum time between camera frames that gives rise to acceptable results.

We present the use of commodity depth based cameras to control viewpoint from passive unencumbered head tracking. This allowed participants to maintain correct perspective while playing an immersive computer ball game, without the need to wear devices, wires or markers. While viewpoint update is more generally useful than stereoscopy it has traditionally used tracking equipment of expense and complexity unsuitable for home use. The current explosion of depth based cameras into the home game market could provide a cheaper, simpler and more elegant solution. While updating viewpoint from a depth camera has been demonstrated it has not previously been studied in a multi-player or fast moving game or undergone accurate measures of latency, task performance or user experience. We integrated low cost commodity hardware including depth based cameras, computers, projectors and stereo glasses, computer games software, a laptop and large projection surfaces. System latency and user performance and experience were measured across a number of viewpoint conditions. System latency of the viewpoint and hand tracking were measured by analysing high speed video footage. The number of goals scored and saved gave a measure of task performance. Subjective impressions of experience were taken from questionnaire and post interview.

In this paper we examine how closely together images to be used in 3D reconstruction are captured, when acquisition is started using a software-based trigger delivered to multiple computers on a network. In addition we compare a software triggered pull system to a push system. Synchronisation is a key component in 3D reconstruction systems and can also be one of the most problematic. In a shape-from-silhouette based system, images are captured at the same moment to ensure the position of the object is consistent in all frames. Differences in position between captured frames results in deformations such as holes and missing or extended features. By capturing images of a projected clock, we can tell with around 10ms granularity, how closely together images are captured. This gives us an indication of whether software capture triggers delivered over a network can be used for image acquisition in a 3D reconstruction system, instead of hardware synchronised capture.

A feature of standard video-mediated Communication systems (VMC) is that participants see into each other's spaces from the viewpoint of a camera. Consequently, participants' capacity to use the spatially-based resources that exist in co-located settings (eg the production and comprehension of pointing and eye gaze direction) can be compromised. Whilst positioning cameras close to displays, or switching or interpolating between multiple cameras to provide appropriately aligned views can reduce this problem, an alternative paradigm is the use of immersive projection technology to locate participants within an immersive collaborative virtual environment (ICVE), in which remote participants appear as 3D graphical representations. Two approaches toward representation of remote participants in ICVEs have been studied: embodied avatars animated using participants' tracked body motion, and vision-based techniques that reconstruct 3D models from multiple streams of live video input. Drawing on empirical evaluations of an avatar-based ICVE system that both captures and displays eye-movement, together with an examination of previous research into gaze, we provide a specification of gaze practices and the cues used in the perception of gaze that should be supported in ICVEs. We delineate some of the challenges for vision-based ICVE and discuss the potential for combining different approaches in the development of such systems.

We present a method for reconstruction of the visual hull (VH) of an object in real-time from multiple video streams. A state of the art polyhedral reconstruction algorithm is accelerated by implementing it for parallel execution on a multi-core graphics processor (GPU). The time taken to reconstruct the VH is measured for both the accelerated and non-accelerated implementations of the algorithm, over a range of image resolutions and number of cameras. The results presented are of relevance to researchers in the field of 3D reconstruction at interactive frame rates (real-time), for applications such as telepresence.

What level of synchronization is necessary between images from multiple cameras in order to realistically reconstruct a moving human in 3D? Live reconstruction of the human form, from cameras surrounding the subject, could bridge the gap between video conferencing and Immersive Collaborative Virtual Environments (ICVEs). Video conferencing faithfully reproduces what someone looks like whereas ICVE faithfully reproduces what they look at. While 3D video has been demonstrated in tele-immersion prototypes, the visual/temporal quality has been way below what has become acceptable in video conferencing. Managed synchronization of the acquisition stage is universally used today to ensure multiple images feeding the reconstruction algorithm were taken at the same time. However, this inevitably increases latency and jitter. We measure the temporal characteristics of the capture stage and the impact of inconsistency on the reconstruction algorithm this feeds. This gives us both input and output characteristics for synchronization. From this we determine whether frame synchronization of multiple camera video streams actually needs to be delivered for 3D reconstruction, and if not what level of temporal divergence is acceptable across the captured image frames.

Would a room in which the walls appear to come to life, be a pleasant place and would it help or hinder concentration? On the doorstep of life-like architecture we use virtual reality to begin to answer these questions. We are brought to this doorstep through the potential convergence of emerging adaptive, reactive and organic architecture and the approaches that have made life like virtual agents both engaging and useful. The scene is set by introducing the concept of Adaptive Appraisive Architecture, in which a building could appear to exhibit life like appearance and behaviour. While being impressive would such a building be pleasant and useful? Before building the physical structure or coding the artificial intelligence this paper measures the impact of being within a room with moving walls on experience and performance. To do this we gave test subjects two jigsaw puzzles and placed them within a life size simulation where the walls move then remain static. The impact of this difference is measured on experience through questionnaire and post interview. The impact on performance is measured in terms of the amount of the puzzle solved. The relevance of the results are not constrained to adaptive architecture as they add to the body of knowledge that relate distractions to concentration.

Latency in a communication system can result in confusing a conversation through loss of causality as people exchange verbal and non-verbal nuances. This paper compares true end-to-end latencies across an immersive virtual environment and a video conference link using the same approach to measure both. Our approach is to measure end-to-end latency through filming the movements of a participant and their remote representation through synchronised cameras. We also compare contemporary and traditional immersive display and capture devices, whilst also measuring event latency taken from log files. We compare an immersive collaborative virtual environment to a video conference as both attempt to reproduce different aspects of the face-to- face meeting, the former favouring appearance and the latter attention. Results inform not only the designers of both approaches but also set the requirements for future developments for 3D video which has the potential to faithfully reproduce both appearance and attention.

Eye gaze is an important and widely studied non-verbal resource in co-located social interaction. When we attempt to support tele-presence between people, there are two main technologies that can be used today: video-conferencing (VC) and collaborative virtual environments (CVEs). In VC, one can observe eye-gaze behaviour but practically the targets of eye-gaze are only correct if the participants remain relatively still. We attempt to support eye-gaze behaviour in an unconstrained manner by integrating eye-trackers into an Immersive CVE (ICVE) system. This paper aims to show that while both ICVE and VC allow people to discern being looked at and what else is looked at, when someone gazes into their space from another location, ICVE alone can continue to do this as people move. The conditions of aligned VC, ICVE, eye-gaze enabled ICVE and co-location are compared. The impact of factors of alignment, lighting, resolution, and perspective distortion are minimised through a set of pilot experiments, before a formal experiment records results for optimal settings. Results show that both VC and ICVE support eye-gaze in constrained situations, but only ICVE supports movement of the observer. We quantify the mis-judgements that are made and discuss how our findings might inform research into supporting eye-gaze through interpolated free viewpoint video based methods.

In face-to-face collaboration, eye gaze is used both as a bidirectional signal to monitor and indicate focus of attention and action, as well as a resource to manage the interaction. In remote interaction supported by immersive collaborative virtual environments (ICVEs), embodied avatars representing and controlled by each participant share a virtual space. We report on a study designed to evaluate methods of avatar eye gaze control during an object-focused puzzle scenario performed between three networked CAVEtrade-like systems. We compare tracked gaze, in which avatars' eyes are controlled by head-mounted mobile eye trackers worn by participants, to a gaze model informed by head orientation for saccade generation, and static gaze featuring non-moving eyes. We analyse task performance, subjective user experience, and interactional behaviour. While not providing statistically significant benefit over static gaze, tracked gaze is observed as the highest performing condition. However, the gaze model resulted in significantly lower task performance and increased error rate.

Participants' eye-gaze is generally not captured or represented in immersive collaborative virtual environment (ICVE) systems. We present EyeCVE, which uses mobile eye-trackers to drive the gaze of each participant's virtual avatar, thus supporting remote mutual eye-contact and awareness of others' gaze in a perceptually unfragmented shared virtual workspace. We detail trials in which participants took part in three-way conferences between remote CAVE™ systems linked via EyeCVE. Eye-tracking data was recorded and used to evaluate interaction, confirming the system's support for the use of gaze as a communicational and management resource in multiparty conversational scenarios. We point toward subsequent investigation of eye-tracking in ICVEs for enhanced remote social-interaction and analysis.

A desktop tool for replay and analysis of gaze-enhanced multiparty virtual collaborative sessions is described. We linked three CAVETM-like environments, creating a multiparty collaborative virtual space where avatars are animated with 3D gaze as well as head and hand motions in real time. Log files are recorded for subsequent playback and analysis using the proposed software tool. During replaying the user can rotate the viewpoint and navigate in the simulated 3D scene. The playback mechanism relies on multiple distributed log files captured at every site. This structure enables an observer to experience latencies of movement and information transfer for every site as this is important for conversation analysis. Playback uses an event-replay algorithm, modified to allow fast traversal of the scene by selective rendering of nodes, and to simulate fast random access. The toolpsilas analysis module can show each participantpsilas 3D gaze points and areas where gaze has been concentrated.

Eye gaze is an important conversational resource that until now could only be supported across a distance if people were rooted to the spot. We introduce EyeCVE, the worldpsilas first tele-presence system that allows people in different physical locations to not only see what each other are doing but follow each otherpsilas eyes, even when walking about. Projected into each space are avatar representations of remote participants, that reproduce not only body, head and hand movements, but also those of the eyes. Spatial and temporal alignment of remote spaces allows the focus of gaze as well as activity and gesture to be used as a resource for non-verbal communication. The temporal challenge met was to reproduce eye movements quick enough and often enough to interpret their focus during a multi-way interaction, along with communicating other verbal and non-verbal language. The spatial challenge met was to maintain communicational eye gaze while allowing free movement of participants within a virtually shared common frame of reference. This paper reports on the technical and especially temporal characteristics of the system.

In collaborative situations, eye gaze is a critical element of behavior which supports and fulfills many activities and roles. In current computer-supported collaboration systems, eye gaze is poorly supported. Even in a state-of-the-art video conferencing system such as the access grid, although one can see the face of the user, much of the communicative power of eye gaze is lost. This article gives an overview of some preliminary work that looks towards integrating eye gaze into an immersive collaborative virtual environment and assessing the impact that this would have on interaction between the users of such a system. Three experiments were conducted to assess the efficacy of eye gaze within immersive virtual environments. In each experiment, subjects observed on a large screen the eye-gaze behavior of an avatar. The eye-gaze behavior of that avatar had previously been recorded from a user with the use of a head-mounted eye tracker. The first experiment was conducted to assess the difference between users' abilities to judge what objects an avatar is looking at with only head gaze being viewed and also with eye- and head-gaze data being displayed. The results from the experiment show that eye gaze is of vital importance to the subjects, correctly identifying what a person is looking at in an immersive virtual environment. The second experiment examined whether a monocular or binocular eye-tracker would be required. This was examined by testing subjects' ability to identify where an avatar was looking from their eye direction alone, or by eye direction combined with convergence. This experiment showed that convergence had a significant impact on the subjects' ability to identify where the avatar was looking. The final experiment looked at the effects of stereo and mono-viewing of the scene, with the subjects being asked to identify where the avatar was looking. This experiment showed that there was no difference in the subjects' ability to detect where the avatar was gazing. This is followed by a description of how the eye-tracking system has been integrated into an immersive collaborative virtual environment and some preliminary results from the use of such a system.

Dead reckoning is widely employed as an entity update packet reduction technique in distributed interactive applications (DIAs). Such techniques reduce network bandwidth consumption and thus limit the effects of network latency on the consistency of networked simulations. A key component of the dead reckoning method is the underlying error threshold metric, as this directly determines when an entity update packet is to be sent between local and remote users. The most common metric is the spatial threshold, which is simply based on the distance between a local user's actual position and their predicted position. Other, recently proposed, metrics include the time-space threshold and the hybrid threshold, both of which are summarised within. This paper investigates the issue of user movement in relation to dead reckoning and each of the threshold metrics. In particular the relationship between the curvature of movement, the various threshold metrics and absolute consistency is studied. Experimental live trials across the Internet allow a comparative analysis of how users behave when different threshold metrics are used with varying degrees of curvature. The presented results provide justification for the use of a hybrid threshold approach when dead reckoning is employed in DIAs

For efficient collaboration between participants, eye gaze is seen as being critical for interaction. Teleconferencing systems such as the AcessGrid allow users to meet across geographically disparate rooms but as of now there seems no substitute for face to face meetings. This paper gives an overview of some preliminary work that looks towards integrating eye gaze into an immersive collaborative virtual environment and assessing the impact that this would have on interaction between the users of such a system. An experiment was conducted to assess the difference between users' abilities to judge what objects an avatar is looking at with only head gaze being viewed and also with eye and head gaze data being displayed. The results from the experiment show that eye gaze is of vital importance to the subjects correctly identifying what a person is looking at in an immersive virtual environment. This is followed by a description of how the eye tracking system has been integrated into an immersive collaborative virtual environment and some preliminary results from the use of such a system

Human-to-human interaction across distributed applications requires that sufficient consistency be maintained among participants in the face of network characteristics such as latency and limited bandwidth. Techniques and approaches for reducing bandwidth usage can minimize network delays by reducing the network traffic and therefore better exploiting available bandwidth. However, these approaches induce inconsistencies within the level of human perception. Dead reckoning is a well-known technique for reducing the number of update packets transmitted between participating nodes. It employs a distance threshold for deciding when to generate update packets. This paper questions the use of such a distance threshold in the context of absolute consistency and it highlights a major drawback with such a technique. An alternative threshold criterion based on time and distance is examined and it is compared to the distance only threshold. A drawback with this proposed technique is also identified and a hybrid threshold criterion is then proposed. However, the trade-off between spatial and temporal inconsistency remains.

Summary form only given. The grid, or grid computing, provides a new and unrivalled technology for large scale distributed simulation as it enables collaboration and the use of distributed computing resources. Last year at DS-RT 2004 a panel was convened to consider the impact of the grid on distributed simulation. Four members presented their views of this area and together they tried to identify the main research issues involved in applying grid technology to distributed simulation and the key future challenges that need to be solved to achieve this goal. These challenges included not only technical ones, but also social ones such as management methodology and the development of standards. This year we revisit this fast changing technology and ask the questions: 1) What major changes has grid computing had over the past year?; 2) How can distributed simulation and related applications benefit from grid computing?; 3) What are the barriers to this?; 4) Are there alternative new technologies that would be a better ROI for distributed simulation than grid computing?.

Collaborative Virtual Environments (CVE) are a form of telecommunication technology that bring together co-located or remote, participants within a spatial social and information context. Collaboration occurs between people and often around shared objects. Fruitful cooperation is helped by natural and intuitive ways of communicating and sharing, for which responsiveness and consistency are leading factors. Many CVEs maximise local responsiveness through a process of localisation and database replication, increasing responsiveness at the cost of lowering consistency. This is acceptable provided the application does not require the shared manipulation of objects. Those that do, require consistency control that provide sufficient synchronisation, ordering and update control, whilst maximising concurrence and thus the responsiveness of the system. This paper describes the major issues and principles of consistency control and demonstrates how we have applied many of these principles in three CVEs.

Many teamwork tasks in the real world require the shared manipulation of objects, both sequentially and concurrently. Collaborative virtual environments (CVE) bring remote people together in an interactive, spatial social and information context that is distributed over a network. Supporting highly collaborative tasks in a natural manner requires a thourough understanding of event traffic as distinct forms of shared object manipulation are bound to diverse real-time and consistency constraints. This position paper summarises our findings during user trials around the shared manipulation of objects within linked immersive displays and introduce a CVE system with a flexible event-handling framework that allows task-oriented concictency control.

Future teleconferencing may enhance communication between remote people by supporting non-verbal communication within an unconstrained space where people can move around and share the manipulation of artefacts. By linking walk-in displays with a Collaborative Virtual Environment (CVE) platform we are able to physically situate a distributed team in a spatially organised social and information context. We have found this to demonstrate unprecedented naturalness in the use of space and body during non-verbal communication and interaction with objects.However, relatively little is known about how people interact through this technology, especially while sharing the manipulation of objects. We observed people engaged in such a task while geographically separated across national boundaries. Our analysis is organised into collaborative scenarios, that each requires a distinct balance of social human communication with consistent shared manipulation of objects.Observational results suggest that walk-in displays do not suffer from some of the important drawbacks of other displays. Previous trials have shown that supporting natural non-verbal communication, along with responsive and consistent shared object manipulation, is hard to achieve. To better understand this problem, we take a close look at how the scenario impacts on the characteristics of event traffic. We conclude by suggesting how various strategies might reduce the consistency problem for particular scenarios.

Social interaction is a key driving force behind any team activity. Real time closely coupled interaction, where we quickly see the effect of our actions on others, is an important feature of social interaction. For example, smiling and returning a smile, shaking hands and passing a gift or business card are some of the most important interactions in the real world. Collaborative virtual environments (CVE) offer the potential for social interaction between geographically distributed groups. Closely coupled interaction is, however, very difficult in present CVE systems. This is because this medium lags behind real world communication in terms of representation, consistency and responsiveness. This paper takes a psychological perspective, describing how the primary forms of human communication in the real world map to those in the virtual. We discuss how each form of communication relates to the feeling of co-presence, giving real examples of behaviour observed in an example application, the virtual gazebo. We present detailed results from user evaluation, focusing on the perceived importance of these influences on collaboration.