Foveated rendering adapts the image synthesis process to the user’s gaze. By exploiting the human visual system’s limitations, in particular in terms of reduced acuity in peripheral vision, it strives to deliver high-quality visual experiences at very reduced computational, storage, and transmission costs. Despite the very substantial progress made in the past decades, the solution landscape is still fragmented, and several research problems remain open. In this work, we present an up-to-date integrative view of the domain from the point of view of the rendering methods employed, discussing general characteristics, commonalities, differences, advantages, and limitations. We cover, in particular, techniques based on adaptive resolution, geometric simplification, shading simplification, chromatic degradation, as well spatio-temporal deterioration. Next, we review the main areas where foveated rendering is already in use today. We finally point out relevant research issues and analyze research trends.

Rendering for Head Mounted Displays (HMD) causes a doubled computational effort, since serving the human stereopsis requires the creation of one image for the left and one for the right eye. The difference in this image pair, called binocular disparity, is an important cue for depth perception and the spatial arrangement of surrounding objects. Findings in the context of the human visual system (HVS) have shown that especially in the near range of an observer, binocular disparities have a high significance. But as with rising distance the disparity converges to a simple geometric shift, also the importance as depth cue exponentially declines. In this paper, we exploit this knowledge about the human perception by rendering objects fully stereoscopic only up to a chosen distance and monoscopic, from there on. By doing so, we obtain three distinct images which are synthesized to a new hybrid stereoscopic image pair, which reasonably approximates a conventionally rendered stereoscopic image pair. The method has the potential to reduce the amount of rendered primitives easily to nearly 50 % and thus, significantly lower frame times. Besides of a detailed analysis of the introduced formal error and how to deal with occurring artifacts, we evaluated the perceived quality of the VR experience during a comprehensive user study with nearly 50 participants. The results show that the perceived difference in quality between the shown image pairs was generally small. An in-depth analysis is given on how the participants reached their decisions and how they subjectively rated their VR experience.

Interactive user interfaces in head-mounted Augmented Reality environments are not always projected onto a physical surface. However, operating such free-floating interfaces by touch gestures is challenging because they do not provide haptic feedback. Considering a pointing gesture, in this work we present a user study evaluating the benefits of increasing the arm position sense for operating non-haptic interface. Our findings confirm that haptic feedback is required and show that an increased arm sense compensates for the lack of haptic feedback. The results suggest that applying 0.3 times of the pointing arm’s weight significantly speeds up direct object selection for free-floating interfaces. We also show that the correction phase of the underlying pointing movement is affected by boosting the arm sense.

The depth images from RGB-D cameras contain a substantial amount of artifacts such as holes and flickering. Moreover, for fast moving objects in successive frames, we perceive ghosting artifacts on the depth images. Hence, the poor quality of the depth images limits them to be used in various applications. Here, we propose a gradient based spatial and temporal method of depth enhancement (gSMOOTH) using least median of squares, which deals with these artifacts. For each depth pixel over a sequence of frames, we look for invalid or unstable or drastically changed depth values and use our approach to replace those values with stable and more feasible depth values. Our approach removes the ghosting artifacts and flickering, and attenuates the amount of temporal noise significantly in real time. We conduct experiments with our own- and reference datasets and evaluate our method against reference methods. Experimental results show improvements for both static and dynamic scenes.

Large, high-resolution displays demonstrated their effectiveness in lab settings for cognitively demanding tasks in single user and collaborative scenarios. The effectiveness is mostly reached through inherent displays» properties - large display real estate and high resolution - that allow for visualization of complex datasets, and support of group work and embodied interaction. To raise users» efficiency, however, more sophisticated user support in the form of advanced user interfaces might be needed. For that we need profound understanding of how large, tiled displays impact users work and behavior. We need to extract behavioral patterns for different tasks and data types. This paper reports on study results of how users, while working collaboratively, process spatially fixed items on large, tiled displays. The results revealed a recurrent pattern showing that users prefer to process documents column wise rather than row wise or erratic.

Supported by their large size and high resolution, display walls suit well for different collaboration types. However, in order to foster instead of impede collaboration processes, interaction techniques need to be carefully designed, taking into regard the possibilities and limitations of the display size, and their effects on human perception and performance. In this paper we investigate the impact of visual distractors (which, for instance, might be caused by other collaborators' input) in peripheral vision on short-term memory and attention. The distractors occur frequently when multiple users collaborate in large wall display systems and may draw attention away from the main task, as such potentially affecting performance and cognitive load. Yet, the effect of these distractors is hardly understood. Gaining a better understanding thus may provide valuable input for designing more effective user interfaces. In this article, we report on two interrelated studies that investigated the effect of distractors. Depending on when the distractor is inserted in the task performance sequence, as well as the location of the distractor, user performance can be disturbed: we will show that distractors may not affect short term memory, but do have an effect on attention. We will closely look into the effects, and identify future directions to design more effective interfaces.

In this paper, we modeled eye movements on tiled Large High-Resolution Displays (LHRD) as a Markov decision process(MDP). We collected eye movements of users participated with free viewing task experiments on LHRD. We have examined two different inverse reinforcement learning algorithms. The presented approach used information about the possible eye movement positions. We showed that it is possible to automatically extract reward function based on effective features from user eye movement behaviors using IRL. We found that, the Itti and HoG features shows highest positive reward weights for both algorithms. Most interestingly, we found that the reward function was able to extract expert behavior information that fulfill to predict eye movements. This is valuable information for estimating the internal states of users such as in, for example, intention recognition, to adapt visual interfaces, or to place important information.

We propose a new method to fill missing or invalid values in depth images generated from the Kinect depth sensor. To fill the missing depth values, we use a robust least median of squares (LMedS) approach. We apply our method for telepresence environments, where Kinects are used very often for reconstructing the captured scene in 3D. We introduce a modified 1D LMedS approach for efficient traversal of consecutive image frames. Our approach solves the unstable nature of depth values in static scenes that is perceived as flickering. We obtain very good result both for static and moving objects inside a scene.

Telepresence systems use 3D techniques to create a more natural human-centered communication over long distances. This work concentrates on the analysis of latency in telepresence systems where acquisition and rendering are distributed. Keeping latency low is important to immerse users in the virtual environment. To better understand latency problems and to identify the source of such latency, we focus on the decomposition of system latency into sub-latencies. We contribute a model of latency and show how it can be used to estimate latencies in a complex telepresence dataflow network. To compare the estimates with real latencies in our prototype, we modify two common latency measurement methods. This presented methodology enables the developer to optimize the design, find implementation issues and gain deeper knowledge about specific sources of latency.

The performance of Augmented Reality (AR) direct object selection coded outside of the human egocentric body frame of reference is decreased under short-term altered gravity. Therefore an adequate countermeasure is required. This paper presents the results of a proof-of-concept (POC) study to investigate the impact of simulated hypergravity on target's size and distance. For gravity-dependent resizing and -positioning we used Hooke's law for the target deformation. The POC study was divided in two experiments, whereby hypergravity was induced by a long-arm human centrifuge and by weights attached to subjects' dominant arm. The study showed that at higher gravity levels larger target size and larger distance between the targets led to increased performance.

We present our current state in developing and testing of Augmented Reality supported spaceflight procedures for intra-vehicular payload activities. Our vision is to support the ground team and the flight crew to author and operate easily AR guidelines without programming and AR knowledge. For visualization of the procedural instructions using an HMD, 3D registered visual aids are overlaid onto the payload model operated by additional voice control. Embedded informational resources (e.g., images and videos) are provided through a mobile tangible user interface. In a pilot study that was performed at the ESA European Astronaut Centre by application domain experts, we evaluated the performance, workload and acceptance by comparing our AR system with the conventional method of displaying PDF documents of the procedure.

Understanding information and gaining insights about information usually means interacting with the displayed information. Utilizing the new capabilities of smart meeting rooms, visual outputs of different information representation applications are presented according to the user's needs. In this paper we present smart interaction management. This interaction approach enables the users to interact with all displayed views, utilizing the novel capabilities of these environments while still being able to interact with applications in a conventional manner using local devices. We further show two use cases demonstrating typical applications of our approach in such multi-display environments: (1) to modify the arrangement and layout of views and (2) to interact with the displayed information within a view.

Systems for immersive videoconferencing utilizing LHRDs face the problem of generating large amounts of data. A solution to reduce this data is essential to realize such systems. We present different possible approaches to do this and give an overview where these are applicable. This provides the reader with a first guide for design decisions when implementing such systems.

How do users of virtual environments perceive virtual space? Many experiments have explored this question, but most of these have used head-mounted immersive displays. This paper reports an experiment that studied large-screen immersive displays at medium-field distances of 2 to 15 meters. The experiment measured ego-centric depth judgments in a CAVE, a tiled display wall, and a real-world outdoor field as a control condition. We carefully modeled the outdoor field to make the three environments as similar as possible. Measuring egocentric depth judgments in large-screen immersive displays requires adapting new measurement protocols; the experiment used timed imagined walking, verbal estimation, and triangulated blind walking. We found that depth judgments from timed imagined walking and verbal estimation were very similar in all three environments. However, triangulated blind walking was accurate only in the out-door field; in the large-screen immersive displays it showed under-estimation effects that were likely caused by insufficient physical space to perform the technique. These results suggest using timed imagined walking as a primary protocol for assessing depth perception in large-screen immersive displays. We also found that depth judgments in the CAVE were more accurate than in the tiled display wall, which suggests that the peripheral scenery offered by the CAVE is helpful when perceiving virtual space.

We propose an acceleration scheme for many-body dynamic collision detection at interactive rates. We use the Velocity-Aligned Discrete Oriented Politope (VADOP), a tight bounding volume representation that offers fast update rates and which is particularly suitable for applications with many fast-moving objects. The axes selection that determines the shape of our bounding volumes is based on spherical coverings. We demonstrate that we can robustly detect collisions that are missed by pseudo-dynamic collision detection schemes, with even greater performance due to substantial collision pruning by our bounding volumes.

We present an event-based scheduling method for time-critical collision detection that meets real-time constraints by balancing and prioritizing computation spent on inter-section tests without starvation. We test each potentially colliding pair of objects at a different frequency, with un-bounded temporal resolution. We show that believability is preserved by adaptively prioritizing intersection tests to re-duce errors in collision detection, using information about the objects and scene. Through the combination of kinetic sweep and prune with stride scheduling we continuously interleave rendering, broad phase collision pruning, nar-row phase intersection testing, and collision response. Our method accrues no per-frame overhead and is interruptible at any point in collision detection, even the broad phase. 1

We introduce a system that adds a foveal inset to large-scale projection displays. The effective resolution of the foveal inset projection is higher than the original display resolution, allowing the user to see more details and finer features in large data sets. The foveal inset is generated by projecting a high-resolution image onto a mirror mounted on a pan-tilt unit (PTU) that is controlled by the user with a laser pointer. Our implementation is based on Chromium and supports many OpenGL applications without modifications.

Continued advances in display hardware, computing power, networking, and rendering algorithms have all converged to dramatically improve large high-resolution display capabilities. We present a survey on prior research with large high-resolution displays. In the hardware configurations section we examine systems including multi-monitor workstations, reconfigurable projector arrays, and others. Rendering and the data pipeline are addressed with an overview of current technologies. We discuss many applications for large high-resolution displays such as automotive design, scientific visualization, control centers, and others. Quantifying the effects of large high-resolution displays on human performance and other aspects is important as we look toward future advances in display technology and how it is applied in different situations. Interacting with these displays brings a different set of challenges for HCI professionals, so an overview of some of this work is provided. Finally, we present our view of the top ten greatest challenges in large highresolution displays.

In the late 90’s the emergence of high performance 3D commodity graphics cards opened the way to use PC clusters for high performance Virtual Reality (VR) applications. Today PC clusters are broadly used to drive multi projector immersive environments. In this paper, we survey the different approaches that have been developed to use PC clusters for VR applications. We review the most common software tools that enable to take advantage of the power of clusters. We also discuss some new trends.

We introduce the 3D warp brush, a method for interactive shape modeling in a immersive virtual reality environment. 3D warp brushes are implicitly-defined tools that operate on triangle meshes. We combine the efficiency of explicit mesh representations with implicit modeling operators. The area of influence of a 3D warp brush can be of arbitrary shape since it has an associated distance field. We define different warp functions including drag, explode, and whittle. A unique feature of our framework is the ability to convert meshes into 3D warp brushes at run time. The use of a Responsive Workbench and two-handed interaction allows the user to exploit the full potential of the modeling system by intuitive and easy modification of a base surface into a desired shape. We present models, which have been created and modified using 3D warp brushes, to demonstrate the usefulness of our framework.

Much work has been done on the development of laser pointers as interaction devices. Typically a camera captures images of a display surface and extracts a laser pointer dot location. This location is processed and used as a cursor position. While the current literature well explains such a system, we feel that some important practical concerns have gone unaddressed. We discuss the design of such a tracking system, focusing on key practical implementation details. In particular we present a robust and efficient dot detection algorithm that allows us to use our system under a variety lighting conditions, and allows us to reduce the amount of image parsing required to find a laser position by an order of magnitude.

We present a distributed scene graph architecture for use in the blue-c, a novel collaborative immersive virtual environment. We extend the widely used OpenGL Performer toolkit to provide a distributed scene graph maintaining full synchronization down to vertex and texel level. We propose a synchronization scheme including customizable, relaxed locking mechanisms. We demonstrate the functionality of our toolkit with two prototype applications in our high-performance virtual reality and visual simulation environment.

Abstract We present the 3D Video Recorder, a system capable of recording, processing, and playing three‐dimensional video from multiple points of view. We first record 2D video streams from several synchronized digital video cameras and store pre‐processed images to disk. An off‐line processing stage converts these images into a time‐varying 3D hierarchical point‐based data structure and stores this 3D video to disk. We show how we can trade‐off 3D video quality with processing performance and devise efficient compression and coding schemes for our novel 3D video representation. A typical sequence is encoded at less than 7 Mbps at a frame rate of 8.5 frames per second. The 3D video player decodes and renders 3D videos from hard‐disk in real‐time, providing interaction features known from common video cassette recorders, like variable‐speed forward and reverse, and slow motion. 3D video playback can be enhanced with novel 3D video effects such as freeze‐and‐rotate and arbitrary scaling. The player builds upon point‐based rendering techniques and is thus capable of rendering high‐quality images in real‐time. Finally, we demonstrate the 3D Video Recorder on multiple real‐life video sequences. ACM CSS: I.3.2 Computer Graphics—Graphics Systems, I.3.5 Computer Graphics—Computational Geometry and Object Modelling, I.3.7 Computer Graphics—Three‐Dimensional Graphics and Realism

We present blue-c , a new immersive projection and 3D video acquisition environment for virtual design and collaboration. It combines simultaneous acquisition of multiple live video streams with advanced 3D projection technology in a CAVE™-like environment, creating the impression of total immersion. The blue-c portal currently consists of three rectangular projection screens that are built from glass panels containing liquid crystal layers. These screens can be switched from a whitish opaque state (for projection) to a transparent state (for acquisition), which allows the video cameras to "look through" the walls. Our projection technology is based on active stereo using two LCD projectors per screen. The projectors are synchronously shuttered along with the screens, the stereo glasses, active illumination devices, and the acquisition hardware. From multiple video streams, we compute a 3D video representation of the user in real time. The resulting video inlays are integrated into a networked virtual environment. Our design is highly scalable, enabling blue-c to connect to portals with less sophisticated hardware.

We present the 3D video recorder, a system capable of recording, processing, and playing three-dimensional video from multiple points of view. We first record 2D video streams from several synchronized digital video cameras and store pre-processed images to disk. An off-line processing stage converts these images into a time-varying three-dimensional hierarchical point-based data structure and stores this 3D video to disk. We show how we can trade-off 3D video quality with processing performance and devise efficient compression and coding schemes for our novel 3D video representation. A typical sequence is encoded at less than 7 megabits per second at a frame rate of 8.5 frames per second. The 3D video player decodes and renders 3D videos from hard-disk in real-time, providing interaction features known from common video cassette recorders, like variable-speed forward and reverse, and slow motion. 3D video playback can be enhanced with novel 3D video effects such as freeze-and-rotate and arbitrary scaling. The player builds upon point-based rendering techniques and is thus capable of rendering high-quality images in real-time. Finally, we demonstrate the 3D video recorder on multiple real-life video sequences.

We present a spatialized audio rendering system for the use in immersive virtual environments. The system is optimized for rendering a sufficient number of dynamically moving sound sources in multi-speaker environments using off-the-shelf audio hardware. Based on simplified physics-based models, we achieve a good trade-off between audio quality, spatial precision, and performance. Convincing acoustic room simulation is accomplished by integrating standard hardware reverberation devices as used in the professional audio and broadcast community. We elaborate on important design principles for audio rendering as well as on practical implementation issues. Moreover, we describe the integration of the audio rendering pipeline into a scene graph-based virtual reality toolkit.

We present JAPE, a flexible prototyping system to support the design of a new advanced collaborative virtual environment. We describe the utilization of different hard- and software components to quickly build a flexible, yet powerful test bed for application and algorithm development. These components include a 3-D rendering toolkit, live video acquisition, speech transmission, and the control of tracking and interaction devices. To facilitate the simultaneous design of applications and algorithms that take advantage of unique features of new collaborative virtual environments, we provide the developer with a flexible prototyping toolkit which emulates the functionality of the final system. The applicability of JAPE is demonstrated with several prototype applications and algorithms.

The paper describes some fundamental issues for robust implementations of progressively refined tetrahedralizations generated through sequences of edge collapses. We address the definition of appropriate cost functions and explain on various tests which are necessary to preserve the consistency of the mesh when collapsing edges. Although considered a special case of progressive simplicial complexes (J. Popovic and H. Hoppe, 1997), the results of our method are of high practical importance and can be used in many different applications, such as finite element meshing, scattered data interpolation, or rendering of unstructured volume data.

The paper presents a framework for multiresolution compression and geometric reconstruction of arbitrarily dimensioned data designed for distributed applications. Although being restricted to uniform sampled data, the versatile approach enables the handling of a large variety of real world elements. Examples include nonparametric, parametric and implicit lines, surfaces or volumes, all of which are common to large scale data sets. The framework is based on two fundamental steps: compression is carried out by a remote server and generates a bit-stream transmitted over the underlying network. Geometric reconstruction is performed by the local client and renders a piecewise linear approximation of the data. More precisely, the compression scheme consists of a newly developed pipeline starting from an initial B-spline wavelet precoding. The fundamental properties of wavelets allow progressive transmission and interactive control of the compression gain by means of global and local oracles. In particular the authors discuss the problem of oracles in semiorthogonal settings and propose sophisticated oracles to remove unimportant coefficients. In addition, geometric constraints such as boundary lines can be compressed in a lossless manner and are incorporated into the resulting bit-stream. The reconstruction pipeline performs a piecewise adaptive linear approximation of data using a fast and easy to use point removal strategy which works with any subsequent triangulation technique.

We present a method for adaptive surface meshing and triangulation which controls the local level of detail of the surface approximation by local spectral estimates. These estimates are determined by a wavelet representation of the surface data. The basic idea is to decompose the initial data set by means of an orthogonal or semi orthogonal tensor product wavelet transform (WT) and to analyze the resulting coefficients. In surface regions, where the partial energy of the resulting coefficients is low, the polygonal approximation of the surface can be performed with larger triangles without losing too much fine grain details. However, since the localization of the WT is bound by the Heisenberg principle, the meshing method has to be controlled by the detail signals rather than directly by the coefficients. The dyadic scaling of the WT stimulated us to build an hierarchical meshing algorithm which transforms the initially regular data grid into a quadtree representation by rejection of unimportant mesh vertices. The optimum triangulation of the resulting quadtree cells is carried out by selection from a look up table. The tree grows recursively as controlled by detail signals which are computed from a modified inverse WT. In order to control the local level of detail, we introduce a new class of wavelet space filters acting as "magnifying glasses" on the data. We show that our algorithm performs a low algorithmic complexity, so that surface meshing can be achieved at interactive rates, such as required by flight simulators, however, other applications are possible as well.

Presents a new method for adaptive surface meshing and triangulation which controls the local level-of-detail of the surface approximation by local spectral estimates. These estimates are determined by a wavelet representation of the surface data. The basic idea is to decompose the initial data set by means of an orthogonal or semi-orthogonal tensor product wavelet transform (WT) and to analyze the resulting coefficients. In surface regions where the partial energy of the resulting coefficients is low, the polygonal approximation of the surface can be performed with larger triangles without losing too much fine-grain detail. However, since the localization of the WT is bound by the Heisenberg principle, the meshing method has to be controlled by the detail signals rather than directly by the coefficients. The dyadic scaling of the WT stimulated us to build a hierarchical meshing algorithm which transforms the initially regular data grid into a quadtree representation by rejection of unimportant mesh vertices. The optimum triangulation of the resulting quadtree cells is carried out by selection from a look-up table. The tree grows recursively, as controlled by the detail signals, which are computed from a modified inverse WT. In order to control the local level-of-detail, we introduce a new class of wavelet space filters acting as "magnifying glasses" on the data.