We introduce a new approach for reconstruction and novel view synthesis of unbounded real-world scenes. In contrast to previous methods using either volumetric fields, grid-based models, or discrete point cloud proxies, we pro-pose a hybrid scene representation, which implicitly encodes the geometry in a continuous octree-based probability field and view-dependent appearance in a multi-resolution hash grid. This allows for extraction of arbitrary explicit point clouds, which can be rendered using rasterization. In doing so, we combine the benefits of both worlds and retain favorable behavior during optimization: Our novel implicit point cloud representation and differentiable bilinear rasterizer enable fast rendering while preserving the fine geometric detail captured by volumetric neural fields. Furthermore, this representation does not depend on priors like structure-from-motion point clouds. Our method achieves state-of-the-art image quality on common benchmarks. Furthermore, we achieve fast inference at interactive frame rates, and can convert our trained model into a large, explicit point cloud to further enhance performance.

Abstract With the advent of holographic near‐eye displays, the need for rendering algorithms that output holograms instead of color images emerged. These holograms usually encode phase maps that alter the phase of coherent light sources such that images result from diffraction effects. While common approaches rely on translating the output of traditional rendering systems to holograms in a post processing step, we instead developed a rendering system that can directly output a phase map to a Spatial Light Modulator (SLM). Our hardware‐ray‐traced sparse point distribution, and depth mapping enable rapid hologram generation, allowing for high‐quality time‐multiplexed holography for real‐time content. Additionally, our system is compatible with foveated rendering which enables further performance optimizations.

Abstract Ray‐axis aligned bounding box intersection tests play a crucial role in the runtime performance of many rendering applications, driven not by complexity but mainly by the volume of tests required. While existing solutions were believed to be pretty much optimal in terms of runtime on current hardware, our paper introduces a new intersection test requiring fewer arithmetic operations compared to all previous methods. By transforming the ray we eliminate the need for one third of the traditional bounding‐slab tests and achieve a speed enhancement of approximately 13.8% or 10.9%, depending on the compiler. We present detailed runtime analyses in various scenarios.

Abstract 3D Gaussian Splats (3DGS) have proven a versatile rendering primitive, both for inverse rendering as well as real‐time exploration of scenes. In these applications, coherence across camera frames and multiple views is crucial, be it for robust convergence of a scene reconstruction or for artifact‐free fly‐throughs. Recent work started mitigating artifacts that break multi‐view coherence, including popping artifacts due to inconsistent transparency sorting and perspective‐correct outlines of (2D) splats. At the same time, real‐time requirements forced such implementations to accept compromises in how transparency of large assemblies of 3D Gaussians is resolved, in turn breaking coherence in other ways. In our work, we aim at achieving maximum coherence, by rendering fully perspective‐correct 3D Gaussians while using a high‐quality approximation of accurate blending, hybrid transparency, on a per‐pixel level, in order to retain real‐time frame rates. Our fast and perspectively accurate approach for evaluation of 3D Gaussians does not require matrix inversions, thereby ensuring numerical stability and eliminating the need for special handling of degenerate splats, and the hybrid transparency formulation for blending maintains similar quality as fully resolved per‐pixel transparencies at a fraction of the rendering costs. We further show that each of these two components can be independently integrated into Gaussian splatting systems. In combination, they achieve up to 2× higher frame rates, 2× faster optimization, and equal or better image quality with fewer rendering artifacts compared to traditional 3DGS on common benchmarks.

Abstract Dynamic reconstruction and spatiotemporal novel‐view synthesis of non‐rigidly deforming scenes recently gained increased attention. While existing work achieves impressive quality and performance on multi‐view or teleporting camera setups, most methods fail to efficiently and faithfully recover motion and appearance from casual monocular captures. This paper contributes to the field by introducing a new method for dynamic novel view synthesis from monocular video, such as casual smartphone captures. Our approach represents the scene as a dynamic neural point cloud, an implicit time‐conditioned point distribution that encodes local geometry and appearance in separate hash‐encoded neural feature grids for static and dynamic regions. By sampling a discrete point cloud from our model, we can efficiently render high‐quality novel views using a fast differentiable rasterizer and neural rendering network. Similar to recent work, we leverage advances in neural scene analysis by incorporating data‐driven priors like monocular depth estimation and object segmentation to resolve motion and depth ambiguities originating from the monocular captures. In addition to guiding the optimization process, we show that these priors can be exploited to explicitly initialize our scene representation to drastically improve optimization speed and final image quality. As evidenced by our experimental evaluation, our dynamic point cloud model not only enables fast optimization and real‐time frame rates for interactive applications, but also achieves competitive image quality on monocular benchmark sequences. Our code and data are available online https://moritzkappel.github.io/projects/dnpc/ .

Abstract In this paper we propose SPaGS, a high‐quality, real‐time free‐viewpoint rendering approach from 360‐degree panoramic images. While existing methods building on Neural Radiance Fields or 3D Gaussian Splatting have difficulties to achieve real‐time frame rates and high‐quality results at the same time, SPaGS combines the advantages of an explicit 3D Gaussian‐based scene representation and ray casting‐based rendering to attain fast and accurate results. Central to our new approach is the exact calculation of axis‐aligned bounding boxes for spherical images that significantly accelerates omnidirectional ray casting of 3D Gaussians. We also present a new dataset consisting of ten real‐world scenes recorded with a drone that incorporates both calibrated 360‐degree panoramic images as well as perspective images captured simultaneously, i.e., with the same flight trajectory. Our evaluation on this new dataset as well as established benchmarks demonstrates that SPaGS excels over state‐of‐the‐art methods in terms of both rendering quality and speed.

Abstract We present Splatshop, a highly optimized toolbox for interactive editing (selection, deletion, painting, transformation, …) of 3D Gaussian Splatting models. Utilizing a comprehensive collection of heuristic approaches, we carefully balance between exact and fast rendering to enable precise editing without sacrificing real‐time performance. Our experiments confirm that Splatshop achieves these goals for scenes with up to 100 million primitives. We also show how our proposed pipeline can be extended for use with head‐mounted displays. As such, Splatshop is the first VR‐capable editor for large‐scale 3D Gaussian Splatting models and a step towards a “Photoshop for Gaussian Splatting.”

Several studies have examined the potential of immersive technologies, such as Virtual Reality (VR), to enhance the effectiveness of memorization. However, existing research has not specifically focused on individuals with Attention Deficit Hyperactivity Disorder (ADHD), and only a limited number of studies have examined the effectiveness of Memory Palace (MP) as a memorization aid for this population, who often experience working memory impairments. To address this gap, we recruited participants with an official diagnosis of ADHD and conducted an experiment in which we investigated the impact of both the Traditional MP technique and a VR version to assess their impact on a memorization task. Our findings indicate that the effectiveness of a VR-based MP might be influenced by prior experience with VR systems, the level of familiarity with the virtual environment, and the MP technique. Nevertheless, our results demonstrate that the MP technique has the potential to enhance recall performance in individuals diagnosed with ADHD.

The reconstruction and novel view synthesis of dynamic scenes recently gained increased attention. As reconstruction from large-scale multi-view data involves immense memory and computational requirements, recent benchmark datasets provide collections of single monocular views per timestamp sampled from multiple (virtual) cameras. We refer to this form of inputs asmonocularizeddata. Existing work shows impressive results for synthetic setups and forward-facing real-world data, but is often limited in the training speed and angular range for generating novel views. This paper addresses these limitations and proposes a new method for full$360^{\circ}$inward-facing novel view synthesis of non-rigidly deforming scenes. At the core of our method are: 1) An efficient deformation module that decouples the processing of spatial and temporal information for accelerated training and inference; and 2) A static module representing the canonical scene as a fast hash-encoded neural radiance field. In addition to existing synthetic monocularized data, we systematically analyze the performance on real-world inward-facing scenes using a newly recorded challenging dataset sampled from a synchronized large-scale multi-view rig. In both cases, our method is significantly faster than previous methods, converging in less than 7 minutes and achieving real-time framerates at 1K resolution, while obtaining a higher visual accuracy for generated novel views. Our code and dataset are available online:https://github.com/MoritzKappel/MoNeRF.

A major factor resulting in cybersickness is the feeling of self-motion experienced when viewing a moving scene in Virtual Reality (VR). Current research indicates that this effect is largely created by motion in the periphery. To discover why this is the case, we investigate the influence of temporal frequency and eccentricity of a stimulus on the magnitude of perceived velocity in the periphery. Based on the perception of two-dimensional stimuli on a wide field-of-view display, we build a model to predict the scaling factor by which the perceived velocity of visual patterns deviates from the physical velocity. Further, our exploratory findings indicate no impact of gaze type on the results, suggesting our model works for both fixation and smooth pursuit scenarios. In an additional pilot study in Virtual Reality (VR), we test the accuracy of the model to predict unnoticeable object motion adaptation in 3D virtual worlds and find positive indications for a similar effect.

Virtual reality has ushered in a revolutionary era of immersive content perception. However, a persistent challenge in dynamic environments is the occurrence of cybersickness arising from a conflict between visual and vestibular cues. Prior techniques have demonstrated that limiting illusory self-motion, so-called vection, by blurring the peripheral part of images, introducing tunnel vision, or altering the camera path can effectively reduce the problem. Unfortunately, these methods often alter the user's experience with visible changes to the content. In this paper, we propose a new technique for reducing vection and combating cybersickness by subtly lowering the screen-space speed of objects in the user's peripheral vision. The method is motivated by our hypothesis that small modifications to the objects' velocity in the periphery and geometrical distortions in the peripheral vision can remain unnoticeable yet lead to reduced vection. This paper describes the experiments supporting this hypothesis and derives its limits. Furthermore, we present a method that exploits these findings by introducing subtle, screen-space geometrical distortions to animation frames to counteract the motion contributing to vection. We implement the method as a realtime post-processing step that can be integrated into existing rendering frameworks. The final validation of the technique and comparison to an alternative approach confirms its effectiveness in reducing cybersickness.

In this paper we investigate the use of electrical muscle stimulation (EMS) to trigger eye blinks for instant hand redirection in virtual reality (VR). With the rapid development of VR technology and increasing user expectations for realistic experiences, maintaining a seamless match between real and virtual objects becomes crucial for immersive interactions. However, hand movements are fast and sometimes unpredictable, increasing the need for instantaneous redirection. We introduce EMS to the field of hand redirection in VR through precise stimulation of the eyelid muscles. By exploiting the phenomenon of change blindness through natural eye blinks, our novel stimulation model achieves instantaneous, imperceptible hand redirection without the need for eye tracking. We first empirically validate the efficiency of our EMS model in eliciting full eye closure. In a second experiment, we demonstrate the feasibility of using such a technique for seamless instantaneous displacement in VR and its particular impact for hand redirection. Among other factors, our analysis also delves into the under-explored domain of gender influence on hand redirection techniques, revealing significant gender-based performance disparities.

Abstract In this paper, we tackle the challenging problem of rendering real‐world 360° panorama videos that support full 6 degrees‐of‐freedom (DoF) head motion from a prerecorded omnidirectional stereo (ODS) video. In contrast to recent approaches that create novel views for individual panorama frames, we introduce a video‐specific temporally‐consistent multi‐sphere image (MSI) scene representation. Given a conventional ODS video, we first extract information by estimating framewise descriptive feature maps. Then, we optimize the global MSI model using theory from recent research on neural radiance fields. Instead of a continuous scene function, this multi‐sphere image (MSI) representation depicts colour and density information only for a discrete set of concentric spheres. To further improve the temporal consistency of our results, we apply an ancillary refinement step which optimizes the temporal coherency between successive video frames. Direct comparisons to recent baseline approaches show that our global MSI optimization yields superior performance in terms of visual quality. Our code and data will be made publicly available.

In this paper, we propose a wavelet-based video codec specifically designed for VR displays that enables real-time playback of high-resolution 360° videos. Our codec exploits the fact that only a fraction of the full 360° video frame is visible on the display at any time. To load and decode the video viewport-dependently in real time, we make use of the wavelet transform for intra- as well as inter-frame coding. Thereby, the relevant content is directly streamed from the drive, without the need to hold the entire frames in memory. With an average of 193 frames per second at 8192 × 8192 -pixel full-frame resolution, the conducted evaluation demonstrates that our codec's decoding performance is up to 272% higher than that of the state-of-the-art video codecs H.265 and AV1 for typical VR displays. By means of a perceptual study, we further illustrate the necessity of high frame rates for a better VR experience. Finally, we demonstrate how our wavelet-based codec can also directly be used in conjunction with foveation for further performance increase.

In this paper, we investigate the perceived personalities of face swaps and how they relate to the personalities of the real people used to create the synthetic individuals' appearance and movements. Given that face swaps have become nearly indistinguishable from real humans, they offer a promising direction for the fast creation of realistic avatars. To investigate the usability of face swaps as avatars, we perform an experiment assessing their personality on the Five-Factor Model, their eeriness and appeal, as well as effects due to familiarity with the original individuals. Our results indicate that face swaps are perceived similarly to real humans and are affected by familiarity. Furthermore, we find a stronger influence of the body and movements on the perceived synthetic personality, especially for their extroversion and conscientiousness.

We present a novel framework for the evaluation of eye tracking data in portrait videos including the automatic generation of customized areas of interest (AOIs) based on facial landmarks. In contrast to previous work, our framework allows the user to flexibly create AOIs by grouping the detected landmarks. Moreover, their shape and size can be modified to better fit both the research question and the precision of the eye tracker. The framework can be used as an integrated solution to not only generate AOIs but also to evaluate viewing behavior like the overall fixation times, the similarity of scanpaths, and the number of saccades between AOIs. Other functionalities include the visualization of gaze paths and the creation of heatmaps. We demonstrate the benefits of our framework and user-defined AOI layouts via an exemplary application, i.e., the investigation of face swapping artifacts.

In this paper we propose omnidirectional galvanic vestibular stimulation (GVS) to mitigate cybersickness in virtual reality applications. One of the most accepted theories indicates that Cybersickness is caused by the visually induced impression of ego motion while physically remaining at rest. As a result of this sensory mismatch, people associate negative symptoms with VR and sometimes avoid the technology altogether. To reconcile the two contradicting sensory perceptions, we investigate GVS to stimulate the vestibular canals behind our ears with low-current electrical signals that are specifically attuned to the visually displayed camera motion. We describe how to calibrate and generate the appropriate GVS signals in real-time for pre-recorded omnidirectional videos exhibiting ego-motion in all three spatial directions. For validation, we conduct an experiment presenting real-world 360° videos shot from a moving first-person perspective in a VR head-mounted display. Our findings indicate that GVS is able to significantly reduce discomfort for cybersickness-susceptible VR users, creating a deeper and more enjoyable immersive experience for many people.

In this paper, we report on perceptual experiments indicating that there are distinct and quantitatively measurable differences in the way we visually perceive genuine versus face-swapped videos.

Video-based human motion transfer creates video animations of humans following a source motion. Current methods show remarkable results for tightly-clad subjects. However, the lack of temporally consistent handling of plausible clothing dynamics, including fine and high-frequency details, significantly limits the attainable visual quality. We address these limitations for the first time in the literature and present a new framework which performs high-fidelity and temporally-consistent human motion transfer with natural pose-dependent non-rigid deformations, for several types of loose garments. In contrast to the previous techniques, we perform image generation in three subsequent stages: synthesizing human shape, structure, and appearance. Given a monocular RGB video of an actor, we train a stack of recurrent deep neural networks that generate these intermediate representations from 2D poses and their temporal derivatives. Splitting the difficult motion transfer problem into subtasks that are aware of the temporal motion context helps us to synthesize results with plausible dynamics and pose-dependent detail. It also allows artistic control of results by manipulation of individual framework stages. In the experimental results, we significantly outperform the state-of-the-art in terms of video realism. The source code is available at https://graphics.tu-bs.de/publications/kappel2020high-fidelity.

We investigate the brain’s subliminal response to fake portrait videos using electroencephalography, with a special emphasis on the viewer’s familiarity with the depicted individuals.Deepfake videos are increasingly becoming popular but, while they are entertaining, they can also pose a threat to society. These face-swapped videos merge physiognomy and behaviour of two different individuals, both strong cues used for recognizing a person. We show that this mismatch elicits different brain responses depending on the viewer’s familiarity with the merged individuals.Using EEG, we classify perceptual differences of familiar and unfamiliar people versus their face-swapped counterparts. Our results show that it is possible to discriminate fake videos from genuine ones when at least one face-swapped actor is known to the observer. Furthermore, we indicate a correlation of classification accuracy with level of personal engagement between participant and actor, as well as with the participant’s familiarity with the used dataset.

Computed Tomography is increasingly employed for non-destructive evaluation, with the aim of reconstructing a surface mesh of a scanned object from radiographic projections. State-of-the-art algorithms first reconstruct a voxel grid and then extract a surface mesh using existing meshing algorithms, often leading to stair-like aliasing artifacts along the grid axes, due to the grid’s orientation-dependent resolution. We circumvent such artifacts in filtered backprojection reconstructions by optimizing the mesh’s vertex positions using information taken directly from the projections, rather than from a voxel grid. We show that our approach reduces stair artifacts both visibly and measurably, at relatively little additional computational cost. Our method can be tied into existing mesh extraction algorithms and removes stair artifacts almost entirely.

We present an efficient and effective computational frame-work for the inverse rendering problem of reconstructing the 3D shape of a piece of glass from its caustic image. Our approach is motivated by the needs of 3D glass printing, a nascent additive manufacturing technique that promises to revolutionize the production of optics elements, from lightweight mirrors to waveguides and lenses. One important problem is the reliable control of the manufacturing process by inferring the printed 3D glass shape from its caustic image. Towards this goal, we propose a novel general-purpose reconstruction algorithm based on differentiable light propagation simulation followed by a regularization scheme that takes the deposited glass volume into account. This enables incorporating arbitrary measurements of caustics into an efficient reconstruction framework. We demonstrate the effectiveness of our method and establish the influence of our hyperparameters using several sample shapes and parameter configurations.

We present a new pipeline to enable head-motion parallax in omnidirectional stereo (ODS) panorama video rendering using a neural depth decoder. While recent ODS panorama cameras record short-baseline horizontal stereo parallax to offer the impression of binocular depth, they do not support the necessary translational degrees-of-freedom (DoF) to also provide for head-motion parallax in virtual reality (VR) applications.

Abstract The peak of virtual reality offers new exciting possibilities for the creation of media content but also poses new challenges. Some areas of interest might be overlooked because the visual content fills up a large portion of viewers' visual field. Moreover, this content is available in 360° around the viewer, yielding locations completely out of sight, making, for example, recall or storytelling in cinematic Virtual Reality (VR) quite difficult. In this paper, we present an evaluation of Stereo Inverse Brightness Modulation for effective and subtle guidance of participants' attention while navigating dynamic virtual environments. The used technique exploits the binocular rivalry effect from human stereo vision and was previously shown to be effective in static environments. Moreover, we propose an extension of the method for successful guidance towards target locations outside the initial visual field. We conduct three perceptual studies, using 13 distinct panorama videos and two VR systems (a VR head mounted display and a fully immersive dome projection system), to investigate (1) general applicability to dynamic environments, (2) stimulus parameter and VR system influence, and (3) effectiveness of the proposed extension for out‐of‐sight targets. Our results prove the applicability of the method to dynamic environments while maintaining its unobtrusive appearance.

Abstract This article investigates neural and physiological correlates of simulator sickness (SS) through a controlled experiment conducted within a fully immersive dome projection system. Our goal is to establish a reliable, objective, and in situ measurable predictive indicator of SS. SS is a problem common to all types of visual simulators consisting of motion sickness‐like symptoms that may be experienced while and after being exposed to a dynamic, immersive visualization. It leads to ethical concerns and impaired validity of simulator‐based research. Due to the popularity of virtual reality devices, the number of people exposed to this problem is increasing and, therefore, it is crucial to find reliable predictors of this condition before any symptoms appear. Despite its relevance and the several theories about its origins, SS cannot yet be quantitatively modeled and predicted. Our results indicate that, while neural correlates did not materialize, physiological measures may be a solid early indicator of oncoming SS.

In this paper, we propose a new method for attention and gaze redirection, specifically designed for immersive stereo displays. Exploiting the dual nature of stereo imagery, our stimulus is composed of complementary parts displayed for each individual eye. This attracts viewers’ attention due to induced binocular rivalry. In a perceptual study, we investigate size- and intensity-related perceptual thresholds of our stimulus for six different real-world panorama images. Our results show that a flexible parameterization allows the stimulus to be perceived even in complex surroundings. To prepare for technical innovations expected in future-generation virtual reality headsets, we used a commercially available head-mounted display as well as a high-resolution dps.

We present Octopus, a learning-based model to infer the personalized 3D shape of people from a few frames (1-8) of a monocular video in which the person is moving with a reconstruction accuracy of 4 to 5mm, while being orders of magnitude faster than previous methods. From semantic segmentation images, our Octopus model reconstructs a 3D shape, including the parameters of SMPL plus clothing and hair in 10 seconds or less. The model achieves fast and accurate predictions based on two key design choices. First, by predicting shape in a canonical T-pose space, the network learns to encode the images of the person into pose-invariant latent codes, where the information is fused. Second, based on the observation that feed-forward predictions are fast but do not always align with the input images, we predict using both, bottom-up and top-down streams (one per view) allowing information to flow in both directions. Learning relies only on synthetic 3D data. Once learned, Octopus can take a variable number of frames as input, and is able to reconstruct shapes even from a single image with an accuracy of 5mm. Results on 3 different datasets demonstrate the efficacy and accuracy of our approach.

We present a simple yet effective method to infer detailed full human body shape from only a single photograph. Our model can infer full-body shape including face, hair, and clothing including wrinkles at interactive frame-rates. Results feature details even on parts that are occluded in the input image. Our main idea is to turn shape regression into an aligned image-to-image translation problem. The input to our method is a partial texture map of the visible region obtained from off-the-shelf methods. From a partial texture, we estimate detailed normal and vector displacement maps, which can be applied to a low-resolution smooth body model to add detail and clothing. Despite being trained purely with synthetic data, our model generalizes well to real-world photographs. Numerous results demonstrate the versatility and robustness of our method.

These days convolutional neural networks (CNNs) are one of the most popular techniques for optical flow estimation yielding exceptional results on many benchmarks. However, the size of their perceptive fields are fixed and displacements not present in the training set cannot be estimated reliably. Additionally, these CNNs need to learn numerous parameters which increases memory consumption. Consequently, their application to high resolution images is impractical, as displacement and memory consumption increase prohibitively. To overcome these limitations, we present an iterative flow refinement approach that can adapt any flow estimation CNN to arbitrary resolution images. In a pyramidal approach, we perform flow estimation for image patches at multiple increasing image resolutions while matching the patches based on the flow of previous iterations. We evaluate our approach using different baselines, displacements and resolutions. The results show that flow estimators can be adapted to high resolution and even panorama images while preserving fine details and reliably handling large displacements without retraining.

We present the ICG Dome, a research facility to explore human visual perception in a high-resolution virtual environment. Current state-of-the-art VR devices still suffer from some technical limitations, like limited field of view, screen-door effect or so-called god-rays. These issues are not present or at least strongly reduced in our system, by design. Latest technology for real-time motion capture and eye tracking open up a wide range of applications.

We investigate the feasibility of combining off-the-shelf virtual reality headsets and electroencephalography. EEG is a highly sensitive tool and subject to strong distortions when exerting physical force like mounting a VR headset on top of it that twists sensors and cables. Our study compares the signal quality of EEG in VR against immersive dome environments and traditional displays using an oddball paradigm experimental design. Furthermore, we compare the signal quality of EEG when combined with a commodity VR headset without modification against a modified version that reduces physical strain on the EEG headset. Our results indicate, that it is possible to combine EEG and VR even without modification under certain conditions. VR headset customisation improves signal quality results. Additionally, display latency of the different modalities is visible on a neurological level.

We present a novel method for high detail-preserving human avatar creation from monocular video. A parameterized body model is refined and optimized to maximally resemble subjects from a video showing them from all sides. Our avatars feature a natural face, hairstyle, clothes with garment wrinkles, and high-resolution texture. Our paper contributes facial landmark and shading-based human body shape refinement, a semantic texture prior, and a novel texture stitching strategy, resulting in the most sophisticated-looking human avatars obtained from a single video to date. Numerous results show the robustness and versatility of our method. A user study illustrates its superiority over the state-of-the-art in terms of identity preservation, level of detail, realism, and overall user preference.

No abstract available.

In a concurrent electroencephalography (EEG) and eye-tracking study, we explore the specific neural responses associated with saccadic eye movements. We hypothesise that there is a distinct saccade-related neural response that occurs well before a physical saccade and that this response is different for free, natural saccades versus forced saccades. Our results show a distinct and measurable brain response approximately 200 ms before a physical saccade actually occurs. This response is distinctly different for free saccades versus forced saccades. Our results open up possibilities of predicting saccades based on neural data. This is of particular relevance for creating effective gaze guidance mechanisms within a virtual reality (VR) environment and for creating faster brain computer interfaces (BCI).

This paper describes a method to obtain accurate 3D body models and texture of arbitrary people from a single, monocular video in which a person is moving. Based on a parametric body model, we present a robust processing pipeline to infer 3D model shapes including clothed people with 4.5mm reconstruction accuracy. At the core of our approach is the transformation of dynamic body pose into a canonical frame of reference. Our main contribution is a method to transform the silhouette cones corresponding to dynamic human silhouettes to obtain a visual hull in a common reference frame. This enables efficient estimation of a consensus 3D shape, texture and implanted animation skeleton based on a large number of frames. Results on 4 different datasets demonstrate the effectiveness of our approach to produce accurate 3D models. Requiring only an RGB camera, our method enables everyone to create their own fully animatable digital double, e.g., for social VR applications or virtual try-on for online fashion shopping.

In this paper we investigate the efficiency of five different image-space post-processing gaze guidance techniques adapted for immersive environments, probing our peripheral vision's sensitivity to different stimuli embedded in complex, real-world panorama still images. We conducted an extensive perceptual study for the commercially available HTC Vive Head-Mounted Display that is equipped with a high-quality eye tracking system to monitor the subjects' gaze direction and saccades in real-time. Despite evaluation reveals no outstanding winner, the local magnification effect by Dorr et al. performed the best with up to 30 % target-directed saccades within the first second.

We propose a GPU-based approach to accelerate filtered backprojection (FBP)-type computed tomography (CT) algorithms by adaptively reconstructing only relevant regions of the object at full resolution. In industrial applications, the object's insensitivity to radiation as well as lack of inner motion allow for high-resolution scans. The large amounts of recorded data, however, pose serious challenges as the computational cost of CT reconstruction scales quartically with resolution. To ensure real-time reconstruction (i.e. faster processing than projection acquisition) for high-resolution scans, our method skips below-threshold voxels and monotonous regions inside the object. Our approach is able to speed up the reconstruction process by a factor of up to 13 while simultaneously reducing memory requirements by a factor of up to 71.

We investigate the efficiency of five different gaze guidance techniques for immersive environments, probing our peripheral vision's sensitivity to different stimuli embedded in complex, real-world panorama still images. We conducted extensive user studies for a commercially available headset as well as in a custom-built dome projection environment. The dome enables us to create true 360° visual immersion at high-resolution, akin to what may be expected of future-generation VR headsets. Evaluation with high-quality eye tracking shows that local luminance modulation as proposed by Bailey et al. is the most effective technique, eliciting saccades to the target region with up to 40 % success rate within the first second.

In this article, we evaluate various image-space modulation techniques that aim to unobtrusively guide viewers’ attention. While previous evaluations mainly target desktop settings, we examine their applicability to ultrawide field of view immersive environments, featuring technical characteristics expected for future-generation head-mounted displays. A custom-built, high-resolution immersive dome environment with high-precision eye tracking is used in our experiments. We investigate gaze guidance success rates and unobtrusiveness of five different techniques. Our results show promising guiding performance for four of the tested methods. With regard to unobtrusiveness we find that—while no method remains completely unnoticed—many participants do not report any distractions. The evaluated methods show promise to guide users’ attention also in a wide field of virtual environment applications, e.g., virtually guided tours or field operation training.

Immersive displays allow presentation of rich video content over a wide field of view. We present a method to boost visual importance for a selected - possibly invisible - scene part in a cluttered virtual environment. This desirable feature enables to unobtrusively guide the gaze direction of a user to any location within the immersive 360° surrounding. Our method is based on subtle gaze direction which did not include head rotations in previous work. For covering the full 360° environment and wide field of view, we contribute an approach for dynamic stimulus positioning and shape variation based on eccentricity to compensate for visibility differences across the visual field. Our approach is calibrated in a perceptual study for a head-mounted display with binocular eye tracking. An additional study validates the method within an immersive visual search task.

There is a continuous effort by animation experts to create increasingly realistic and more human-like digital characters. However, as virtual characters become more human they risk evoking a sense of unease in their audience. This sensation, called the Uncanny Valley effect, is widely acknowledged both in the popular media and scientific research but empirical evidence for the hypothesis has remained inconsistent. In this paper, we investigate the neural responses to computer-generated faces in a cognitive neuroscience study. We record brain activity from participants (N = 40)} using electroencephalography (EEG) while they watch videos of real humans and computer-generated virtual characters. Our results show distinct differences in neural responses for highly realistic computer-generated faces such as Digital Emily compared with real humans. These differences are unique only to agents that are highly photorealistic, i.e. the `uncanny' response. Based on these specific neural correlates we train a support vector machine~(SVM) to measure the probability of an uncanny response for any given computer-generated character from EEG data. This allows the ordering of animated characters based on their level of `uncanniness'.

Advances in computer graphics enable us to create digital images of astonishing complexity and realism. However, processing resources are still a limiting factor. Hence, many costly but desirable aspects of realism are often not accounted for, including global illumination, accurate depth of field and motion blur, spectral effects, etc. especially in real-time rendering. At the same time, there is a strong trend towards more pixels per display due to larger displays, higher pixel densities or larger fields of view. Further observable trends in current display technology include more bits per pixel (high dynamic range, wider color gamut/fidelity), increasing refresh rates (better motion depiction), and an increasing number of displayed views per pixel (stereo, multi-view, all the way to holographic or lightfield displays). These developments cause significant unsolved technical challenges due to aspects such as limited compute power and bandwidth. Fortunately, the human visual system has certain limitations, which mean that providing the highest possible visual quality is not always necessary. In this report, we present the key research and models that exploit the limitations of perception to tackle visual quality and workload alike. Moreover, we present the open problems and promising future research targeting the question of how we can minimize the effort to compute and display only the necessary pixels while still offering a user full visual experience.

This study compares three popular modalities for analyzing perceived video quality; user ratings, eye tracking, and EEG. We contrast these three modalities for a given video sequence to determine if there is a gap between what humans consciously see and what we implicitly perceive . Participants are shown a video sequence with different artifacts appearing at specific distances in their field of vision; near foveal, middle peripheral, and far peripheral. Our results show distinct differences between what we saccade to (eye tracking), how we consciously rate video quality, and our neural responses (EEG data). Our findings indicate that the measurement of perceived quality depends on the specific modality used.

With ever-increasing display resolution for wide field-of-view displays—such as head-mounted displays or 8k projectors—shading has become the major computational cost in rasterization. To reduce computational effort, we propose an algorithm that only shades visible features of the image while cost-effectively interpolating the remaining features without affecting perceived quality. In contrast to previous approaches we do not only simulate acuity falloff but also introduce a sampling scheme that incorporates multiple aspects of the human visual system: acuity, eye motion, contrast (stemming from geometry, material or lighting properties), and brightness adaptation. Our sampling scheme is incorporated into a deferred shading pipeline to shade the image's perceptually relevant fragments while a pull-push algorithm interpolates the radiance for the rest of the image. Our approach does not impose any restrictions on the performed shading. We conduct a number of psycho-visual experiments to validate scene- and task-independence of our approach. The number of fragments that need to be shaded is reduced by 50 % to 80 %. Our algorithm scales favorably with increasing resolution and field-of-view, rendering it well-suited for head-mounted displays and wide-field-of-view projection.

Bright glare in nighttime situations strongly decreases human contrast perception. Nighttime simulations therefore require a way to realistically depict contrast perception of the user. Due to the limited luminance of popular as well as specialized high-dynamic range displays, physical adaptation of the human eye cannot yet be replicated in a physically correct manner in a simulation environment. To overcome this limitation, we propose a method to emulate the adaptation in nighttime glare situations using a perception-based model. We implemented a postprocessing tone mapping algorithm that simulates the corresponding contrast reduction effect for a night-driving simulation with glares from oncoming vehicles headlights. During glare, tone mapping reduces image contrast in accordance with the incident veiling luminance. As the glare expires, the contrast starts to normalize smoothly over time. The conversion of glare parameters and elapsed time into image contrast during the readaptation phase is based on extensive user studies carried out first in a controlled laboratory setup. Additional user studies have then been conducted in field tests to ensure validity of the derived time-dependent tone-mapping function and to verify transferability onto real-world traffic scenarios.

Order selection of autoregressive processes is an active research topic in time series analysis, and the development and evaluation of automatic order selection criteria remains a challenging task for domain experts. We propose a visual analytics approach, to guide the analysis and development of such criteria. A flexible synthetic model generator-combined with specialized responsive visualizations-allows comprehensive interactive evaluation. Our fast framework allows feedback-driven development and fine-tuning of new order selection criteria in real-time. We demonstrate the applicability of our approach in three use-cases for two general as well as a real-world example.

High-quality stereo and optical flow maps are essential for a multitude of tasks in visual media production, e.g. virtual camera navigation, disparity adaptation or scene editing. Rather than estimating stereo and optical flow separately, scene flow is a valid alternative since it combines both spatial and temporal information and recently surpassed the former two in terms of accuracy. However, since automated scene flow estimation is non-accurate in a number of situations, resulting rendering artifacts have to be corrected manually in each output frame, an elaborate and time-consuming task. We propose a novel workflow to edit the scene flow itself, catching the problem at its source and yielding a more flexible instrument for further processing. By integrating user edits in early stages of the optimization, we allow the use of approximate scribbles instead of accurate editing, thereby reducing interaction times. Our results show that editing the scene flow improves the quality of visual results considerably while requiring vastly less editing effort.

Immersion is the ultimate goal of head-mounted displays (HMD) for Virtual Reality (VR) in order to produce a convincing user experience. Two important aspects in this context are motion sickness, often due to imprecise calibration, and the integration of a reliable eye tracking. We propose an affordable hard- and software solution for drift-free eye-tracking and user-friendly lens calibration within an HMD. The use of dichroic mirrors leads to a lean design that provides the full field-of-view (FOV) while using commodity cameras for eye tracking. Our prototype supports personalizable lens positioning to accommodate for different interocular distances. On the software side, a model-based calibration procedure adjusts the eye tracking system and gaze estimation to varying lens positions. Challenges such as partial occlusions due to the lens holders and eye lids are handled by a novel robust monocular pupil-tracking approach. We present four applications of our work: Gaze map estimation, foveated rendering for depth of field, gaze-contingent level-of-detail, and gaze control of virtual avatars.

Recent advances in free-viewpoint rendering techniques as well as the continued improvements of the internet network infrastructure open the door for challenging new applications. In this paper, we present a framework for interactive free-viewpoint streaming with open standards and software. Network bandwidth, encoding strategy as well as codec support for open source browsers are key constraints to be considered for our interactive streaming applications. Our framework is capable of real-time server-side rendering and interactively streaming the output by means of open source streaming. To enable viewer interaction with the free-viewpoint video rendering back-end in a standard browser, user events are captured with Javascript and transmitted using WebSockets. The rendered video is streamed to the browser using the FFmpeg free software project. This paper discusses the applicability of open source streaming and presents timing measurements for video-frame transmission over network.

We present a novel approach for compression of depth maps based on Compressed Sensing and Sparse Coding. Our proposed scheme compresses and stores the depth map, and then-during the decompression step-makes use of the readily available additional RGB information to guide the reconstruction. We introduce additional constraints to the underlying optimization problem enforcing the correctness of the RGB image in the decompression step. A comparison with established compression schemes shows that our proposed method leads to a lower error rate at high compression ratios.

We present a complete hardware and software solution for integrating binocular eye tracking into current state-of-the-art lens-based Head-mounted Displays (HMDs) without affecting the user's wide field-of-view off the display. The system uses robust and efficient new algorithms for calibration and pupil tracking and allows realtime eye tracking and gaze estimation. Estimating the relative gaze direction of the user opens the door to a much wider spectrum of virtual reality applications and games when using HMDs. We show a 3d-printed prototype of a low-cost HMD with eye tracking that is simple to fabricate and discuss a variety of VR applications utilizing gaze estimation.

Abstract Adaptive filtering techniques have proven successful in handling non‐uniform noise in Monte‐Carlo rendering approaches. A recent trend is to choose an optimal filter per pixel from a selection of non spatially‐varying filters. Nonetheless, the best filter choice is difficult to predict in the absence of a reference rendering. Our approach relies on the observation that the reconstruction error is locally smooth for a given filter. Hence, we propose to construct a dense error prediction from a small set of sparse but robust estimates. The filter selection is then formulated as a non‐local optimization problem, which we solve via graph cuts, to avoid visual artifacts due to inconsistent filter choices. Our approach does not impose any restrictions on the used filters, outperforms previous state‐of‐the‐art techniques and provides an extensible framework for future reconstruction techniques.

Abstract We present a fast reconstruction filtering method for images generated with Monte Carlo–based rendering techniques. Our approach specializes in reducing global illumination noise in the presence of depth‐of‐field effects at very low sampling rates and interactive frame rates. We employ edge‐aware filtering in the sample space to locally improve outgoing radiance of each sample. The improved samples are then distributed in the image plane using a fast, linear manifold‐based approach supporting very large circles of confusion. We evaluate our filter by applying it to several images containing noise caused by Monte Carlo–simulated global illumination, area light sources and depth of field. We show that our filter can efficiently denoise such images at interactive frame rates on current GPUs and with as few as 4–16 samples per pixel. Our method operates only on the colour and geometric sample information output of the initial rendering process. It does not make any assumptions on the underlying rendering technique and sampling strategy and can therefore be implemented completely as a post‐process filter.

In this paper, we present and investigate an approach to fast multilabel color image segmentation using convex optimization techniques. The presented model is in some ways related to the well-known Mumford-Shah model, but deviates in certain important aspects. The optimization problem has been designed with two goals in mind. The objective function should represent fundamental concepts of image segmentation, such as incorporation of weighted curve length and variation of intensity in the segmented regions, while allowing transformation into a convex concave saddle point problem that is computationally inexpensive to solve. This paper introduces such a model, the nontrivial transformation of this model into a convex-concave saddle point problem, and the numerical treatment of the problem. We evaluate our approach by applying our algorithm to various images and show that our results are competitive in terms of quality at unprecedentedly low computation times. Our algorithm allows high-quality segmentation of megapixel images in a few seconds and achieves interactive performance for low resolution images.

Many compelling video processing effects can be achieved if per-pixel depth information and 3D camera calibrations are known. However, the success of such methods is highly dependent on the accuracy of this "scene-space" information. We present a novel, sampling-based framework for processing video that enables high-quality scene-space video effects in the presence of inevitable errors in depth and camera pose estimation. Instead of trying to improve the explicit 3D scene representation, the key idea of our method is to exploit the high redundancy of approximate scene information that arises due to most scene points being visible multiple times across many frames of video. Based on this observation, we propose a novel pixel gathering and filtering approach. The gathering step is general and collects pixel samples in scene-space, while the filtering step is application-specific and computes a desired output video from the gathered sample sets. Our approach is easily parallelizable and has been implemented on GPU, allowing us to take full advantage of large volumes of video data and facilitating practical runtimes on HD video using a standard desktop computer. Our generic scene-space formulation is able to comprehensively describe a multitude of video processing applications such as denoising, deblurring, super resolution, object removal, computational shutter functions, and other scene-space camera effects. We present results for various casually captured, hand-held, moving, compressed, monocular videos depicting challenging scenes recorded in uncontrolled environments.

We propose the computation of a perceptual motion blur in videos. Our technique takes the predicted eye motion into account when watching the video. Compared to traditional motion blur recorded by a video camera our approach results in a perceptual blur that is closer to reality. This postprocess can also be used to simulate different shutter effects or for other artistic purposes. It handles real and artificial video input, is easy to compute and has a low additional cost for rendered content. We illustrate its advantages in a user study using eye tracking.

An algorithmic framework to compute sparse or minimal-TV solutions of linear systems is proposed. The framework includes both the Kaczmarz method and the linearized Bregman method as special cases and also several new methods such as a sparse Kaczmarz solver. The algorithmic framework has a variety of applications and is especially useful for problems in which the linear measurements are slow and expensive to obtain. We present examples for online compressed sensing, TV tomographic reconstruction and radio interferometry.

Reconstructing objects from monocular input video data is a challenging task. Since cues about shading and illumination are ambiguous from a single viewpoint, simultaneously distinguishing between material properties, texture, and illumination is infeasible. To properly reconstruct scene illumination a model of the object surface is mandatory. As most 3D surface reconstruction, surface refinement, and illumination reconstruction techniques rely on a diffuse object surface model, we propose a technique to robustly reconstruct surface albedo from monocular input video data and a 3D mesh representation of an object. For all frames of the video sequence we collect color samples for visible vertices under varying illumination directions and correct for possible ambient occlusion using mesh geometry information. We then use a two-staged clustering approach to recover the most plausible surface albedo for every mesh vertex. The presented approach does not require any a-priori illumination model and is evaluated using synthetic as well as real-world test sequences.

Abstract This paper introduces compressed eigenfunctions of the Laplace‐Beltrami operator on 3D manifold surfaces. They constitute a novel functional basis, called the compressed manifold basis, where each function has local support. We derive an algorithm, based on the alternating direction method of multipliers (ADMM), to compute this basis on a given triangulated mesh. We show that compressed manifold modes identify key shape features, yielding an intuitive understanding of the basis for a human observer, where a shape can be processed as a collection of parts. We evaluate compressed manifold modes for potential applications in shape matching and mesh abstraction. Our results show that this basis has distinct advantages over existing alternatives, indicating high potential for a wide range of use‐cases in mesh processing.

Abstract Light refracted by a dispersive interface leads to beautifully colored patterns that can be rendered faithfully with spectral Monte‐Carlo methods. Regrettably, results often suffer from chromatic noise or banding, requiring high sampling rates and large amounts of memory compared to renderers operating in some trichromatic color space. Addressing this issue, we introduce spectral ray differentials, which describe the change of light direction with respect to changes in the spectrum. In analogy with the classic ray and photon differentials, this information can be used for filtering in the spectral domain. Effectiveness of our approach is demonstrated by filtering for offline spectral light and path tracing as well as for an interactive GPU photon mapper based on splatting. Our results show considerably less chromatic noise and spatial aliasing while retaining good visual similarity to reference solutions with negligible overhead in the order of milliseconds.

We present a novel approach to free-viewpoint video. Our main contribution is the formulation of a hybrid approach between image morphing and depth-image based rendering. When rendering the scene from novel viewpoints, we use both dense pixel correspondences between image pairs as well as an underlying, view-dependent geometrical model. Our novel reconstruction scheme iteratively refines geometric and correspondence information. By combining the strengths of both depth and correspondence estimation, our approach enables free-viewpoint video also for challenging scenes as well as for recordings that may violate typical constraints in multiview reconstruction. For example, our method is robust against inaccurate camera calibration, asynchronous capture, and imprecise depth reconstruction. Rendering results for different scenes and applications demonstrate the versatility and robustness of our approach.

We present a new scheme for video colorization using optimization in rotation-aware Gabor feature space. Most current methods of video colorization incur temporal artifacts and prohibitive processing costs, while this approach is designed in a spatiotemporal manner to preserve temporal coherence. The parallel implementation on graphics hardware is also facilitated to achieve realtime performance of color optimization. By adaptively clustering video frames and extending Gabor filtering to optical flow computation, we can achieve real-time color propagation within and between frames. Temporal coherence is further refined through user scribbles in video frames. The experimental results demonstrate that our proposed approach is efficient in producing high-quality colorized videos.

We present a semi-automatic approach to exchange the clothes of an actor for arbitrary virtual garments in conventional monocular video footage as a postprocess. We reconstruct the actor's body shape and motion from the input video using a parameterized body model. The reconstructed dynamic 3D geometry of the actor serves as an animated mannequin for simulating the virtual garment. It also aids in scene illumination estimation, necessary to realistically light the virtual garment. An image-based warping technique ensures realistic compositing of the rendered virtual garment and the original video. We present results for eight real-world video sequences featuring complex test cases to evaluate performance for different types of motion, camera settings, and illumination conditions.

Abstract We present a comprehensive data‐driven statistical model for skin and muscle deformation of the human shoulder‐arm complex. Skin deformations arise from complex bio‐physical effects such as non‐linear elasticity of muscles, fat, and connective tissue; and vary with physiological constitution of the subjects and external forces applied during motion. Thus, they are hard to model by direct physical simulation. Our alternative approach is based on learning deformations from multiple subjects performing different exercises under varying external forces. We capture the training data through a novel multi‐camera approach that is able to reconstruct fine‐scale muscle detail in motion. The resulting reconstructions from several people are aligned into one common shape parametrization, and learned using a semi‐parametric non‐linear method. Our learned data‐driven model is fast, compact and controllable with a small set of intuitive parameters – pose, body shape and external forces, through which a novice artist can interactively produce complex muscle deformations. Our method is able to capture and synthesize fine‐scale muscle bulge effects to a greater level of realism than achieved previously. We provide quantitative and qualitative validation of our method.

Abstract Astronomical nebulae are among the most complex and visually appealing phenomena known outside the bounds of the Solar System. However, our fixed vantage point on Earth limits us to a single known view of these objects, and their intricate volumetric structure cannot be recovered directly. Recent approaches to reconstructing a volumetric 3D model use the approximate symmetry inherent to many nebulae, but require several hours of computation time on large multi‐GPU clusters. We present a novel reconstruction algorithm based on group sparsity that reaches or even exceeds the quality of prior results while taking only a fraction of the time on a conventional desktop PC, thereby enabling end users in planetariums or educational facilities to produce high‐quality content without expensive hardware or manual modeling. In principle, our approach can be generalized to other transparent phenomena with arbitrary types of user‐specified symmetries.

Display resolution is frequently exceeded by available image resolution. Recently, apparent display resolution enhancement (ADRE) techniques show how characteristics of the human visual system can be exploited to provide super-resolution on high refresh rate displays. In this paper, we address the problem of generalizing the ADRE technique to conventional videos of arbitrary content. We propose an optimization-based approach to continuously translate the video frames in such a way that the added motion enables apparent resolution enhancement for the salient image region. The optimization considers the optimal velocity, smoothness, and similarity to compute an appropriate trajectory. In addition, we provide an intuitive user interface that allows to guide the algorithm interactively and preserves important compositions within the video. We present a user study evaluating apparent rendering quality and show versatility of our method on a variety of general test scenes.

We propose a method that extracts sparse and spatially localized deformation modes from an animated mesh sequence. To this end, we propose a new way to extend the theory of sparse matrix decompositions to 3D mesh sequence processing, and further contribute with an automatic way to ensure spatial locality of the decomposition in a new optimization framework. The extracted dimensions often have an intuitive and clear interpretable meaning. Our method optionally accepts user-constraints to guide the process of discovering the underlying latent deformation space. The capabilities of our efficient, versatile, and easy-to-implement method are extensively demonstrated on a variety of data sets and application contexts. We demonstrate its power for user friendly intuitive editing of captured mesh animations, such as faces, full body motion, cloth animations, and muscle deformations. We further show its benefit for statistical geometry processing and biomechanically meaningful animation editing. It is further shown qualitatively and quantitatively that our method outperforms other unsupervised decomposition methods and other animation parameterization approaches in the above use cases.

Image Based Rendering (IBR) allows interactive scene exploration from images alone. However, despite considerable development in the area, one of the main obstacles to better quality and more realistic visualizations is the occurrence of visually disagreeable artifacts. In this paper we present a methodology to map out the perception of IBR-typical artifacts. This work presents an alternative to traditional image and video quality evaluation methods by using an EEG device to determine the implicit visual processes in the human brain. Our work demonstrates the distinct differences in the perception of different types of visual artifacts and the implications of these differences.

High quality dense image correspondence estimation between two images is an essential pre-requisite for view interpolation in visual media production. Due to the ill-posed nature of the problem, automated estimation approaches are prone to erroneous correspondences and subsequent quality degradation, e.g. in the presence of ambiguous movements that require human scene understanding to resolve. Where visually convincing results are essential, artifacts resulting from estimation errors must be repaired by hand with image editing tools. In this paper, we propose a new workflow alternative by fixing the correspondences instead of fixing the interpolated images. We combine realtime interactive correspondence display, multi-level user guidance and algorithmic subpixel precision to counteract failure cases of automated estimation algorithms. Our results show that already few interactions improve the visual quality considerably.

We present an audio retargeting technique to create custom soundtracks for movies and games from existing audio material (typically music) by automatically rearranging audio segments. Constraints can be specified to make the length of the audio fit the length of a movie scene, or to align parts of a piece of music with particular events. Existing approaches typically create soundtracks with many unnecessary and often disruptive transitions. We extend a recent analysis and resynthesis method with a novel genetic algorithm for finding an optimal succession of audio segments that minimizes the number of audible transitions and repetitions as well as the deviation from user-specified constraints. Compared to prior work, our experiments with audio examples from different musical genres show a significant improvement with respect to the optimization criteria, and the resulting soundtracks contain few, if any, noticeable transitions.

Abstract We present a new data structure for object space partitioning that can be represented completely implicitly. The bounds of each node in the tree structure are recreated at run‐time from the scene objects contained therein. By applying a presorting procedure to the geometry, only a known fraction of the geometry is needed to locate the bounding planes of any node. We evaluate the impact of the implicit bounding plane representation and compare our algorithm to a classic bounding volume hierarchy. Though the representation is completely implicit, we still achieve interactive frame rates on commodity hardware.

Abstract The scatterplot matrix (SPLOM) is a well‐established technique to visually explore high‐dimensional data sets. It is characterized by the number of scatterplots (plots) of which it consists of. Unfortunately, this number quadratically grows with the number of the data set’s dimensions. Thus, an SPLOM scales very poorly. Consequently, the usefulness of SPLOMs is restricted to a small number of dimensions. For this, several approaches already exist to explore such ‘small’ SPLOMs. Those approaches address the scalability problem just indirectly and without solving it. Therefore, we introduce a new greedy approach to manage ‘large’ SPLOMs with more than 100 dimensions. We establish a combined visualization and interaction scheme that produces intuitively interpretable SPLOMs by combining known quality measures, a pre‐process reordering and a perception‐based abstraction. With this scheme, the user can interactively find large amounts of relevant plots in large SPLOMs.

In this article we use an ElectroEncephaloGraph (EEG) to explore the perception of artifacts that typically appear during rendering and determine the perceptual quality of a sequence of images. Although there is an emerging interest in using an EEG for image quality assessment, one of the main impediments to the use of an EEG is the very low Signal-to-Noise Ratio (SNR) which makes it exceedingly difficult to distinguish neural responses from noise. Traditionally, event-related potentials have been used for analysis of EEG data. However, they rely on averaging and so require a large number of participants and trials to get meaningful data. Also, due the the low SNR ERP's are not suited for single-trial classification. We propose a novel wavelet-based approach for evaluating EEG signals which allows us to predict the perceived image quality from only a single trial. Our wavelet-based algorithm is able to filter the EEG data and remove noise, eliminating the need for many participants or many trials. With this approach it is possible to use data from only 10 electrode channels for single-trial classification and predict the presence of an artifact with an accuracy of 85%. We also show that it is possible to differentiate and classify a trial based on the exact type of artifact viewed. Our work is particularly useful for understanding how the human visual system responds to different types of degradations in images and videos. An understanding of the perception of typical image-based rendering artifacts forms the basis for the optimization of rendering and masking algorithms.

The 3D visualization of astronomical nebulae is a challenging problem since only a single 2D projection is observable from our fixed vantage point on Earth. We attempt to generate plausible and realistic looking volumetric visualizations via a tomographic approach that exploits the spherical or axial symmetry prevalent in some relevant types of nebulae. Different types of symmetry can be implemented by using different randomized distributions of virtual cameras. Our approach is based on an iterative compressed sensing reconstruction algorithm that we extend with support for position-dependent volumetric regularization and linear equality constraints. We present a distributed multi-GPU implementation that is capable of reconstructing high-resolution datasets from arbitrary projections. Its robustness and scalability are demonstrated for astronomical imagery from the Hubble Space Telescope. The resulting volumetric data is visualized using direct volume rendering. Compared to previous approaches, our method preserves a much higher amount of detail and visual variety in the 3D visualization, especially for objects with only approximate symmetry.

The way images are perceived by a human observer is becoming increasingly important in visual media, e.g. for (photo-) realistic image synthesis or memory-efficient encoding of large volumes of image/video data. Traditionally, perceived image quality is assessed by either using specially developed image quality metrics or by conducting user studies. In this paper, we investigate the use of electroencephalography as a tool for evaluating image quality. We demonstrate that the presence of artifacts reliably elicits a measurable response in the brain. We furthermore show that the reaction varies depending on the severity of artifacts, so that it may be used in order to objectively quantify image quality.

Music and sound play an important role for the emotional involvement of the spectator in visual media such as movies, video games or artistic shows. However, the generation of suitable background music and atmospheric sound matching the visual context is a tedious and typically completely manual task. For content creators, it would be desirable to be able to specify a sound example and a few key points at which certain parts are to be played, and to then automatically generate a soundtrack subject to these constraints. For interactive media such as video games, on the other hand, example-based background sound could automatically adapt to the game events. We propose a new example-based audio synthesis method for both applications which works for musical and texture-like sound examples alike. Our algorithm is evaluated on a variety of musical styles, from classical music to punk rock, and on several texture-like atmospheric sounds.

Abstract Interactive computation of global illumination is a major challenge in current computer graphics research. Global illumination heavily affects the visual quality of generated images. It is therefore a key attribute for the perception of photo‐realistic images. Path tracing is able to simulate the physical behaviour of light using Monte Carlo techniques. However, the computational burden of this technique prohibits interactive rendering times on standard commodity hardware in high‐quality. Trying to solve the Monte Carlo integration with fewer samples results in characteristic noisy images. Global illumination filtering methods take advantage of the fact that the integral for neighbouring pixels may be very similar. Averaging samples of similar characteristics in screen‐space may approximate the correct integral, but may result in visible outliers. In this paper, we present a novel path tracing pipeline based on an edge‐aware filtering method for the indirect illumination which produces visually more pleasing results without noticeable outliers. The key idea is not to filter the noisy path traced images but to use it as a guidance to filter a second image composed from characteristic scene attributes that do not contain noise by default. We show that our approach better approximates the Monte Carlo integral compared to previous methods. Since the computation is carried out completely in screen‐space it is therefore applicable to fully dynamic scenes, arbitrary lighting and allows for high‐quality path tracing at interactive frame rates on commodity hardware.

Traditional optical flow algorithms rely on consecutive short-exposed images. In this work, we make use of an additional long-exposed image for motion field estimation. Long-exposed images integrate motion information directly in the form of motion-blur. With this additional information, more robust and accurate motion fields can be estimated. In addition, the moment of occlusion can be determined. Considering the basic signal-theoretical problem in motion field estimation, we exploit the fact that long-exposed images integrate motion information to prevent temporal aliasing. A suitable image formation model relates the long-exposed image to preceding and succeeding short-exposed images in terms of dense 2D motion and per-pixel occlusion/disocclusion timings. Based on our image formation model, we describe a practical variational algorithm to estimate the motion field not only for visible image regions but also for regions getting occluded. Results for synthetic as well as real-world scenes demonstrate the validity of the approach.

We present a method for image interpolation that is able to create high-quality, perceptually convincing transitions between recorded images. By implementing concepts derived from human vision, the problem of a physically correct image interpolation is relaxed to that of image interpolation which is perceived as visually correct by human observers. We find that it suffices to focus on exact edge correspondences, homogeneous regions and coherent motion to compute convincing results. A user study confirms the visual quality of the proposed image interpolation approach. We show how each aspect of our approach increases perceived quality of the result. We compare the results to other methods and assess achievable quality for different types of scenes.

Generation of synthetic datasets is a common practice in many research areas. Such data is often generated to meet specific needs or certain conditions that may not be easily found in the original, real data. The nature of the data varies according to the application area and includes text, graphs, social or weather data, among many others. The common process to create such synthetic datasets is to implement small scripts or programs, restricted to small problems or to a specific application. In this paper we propose a framework designed to generate high dimensional datasets. Users can interactively create and navigate through multi dimensional datasets using a suitable graphical user-interface. The data creation is driven by statistical distributions based on a few user-defined parameters. First, a grounding dataset is created according to given inputs, and then structures and trends are included in selected dimensions and orthogonal projection planes. Furthermore, our framework supports the creation of complex non-orthogonal trends and classified datasets. It can successfully be used to create synthetic datasets simulating important trends as multidimensional clusters, correlations and outliers.

We present a flexible interactive 3D morpho-kinematical modeling application for astrophysics. Compared to other systems, our application reduces the restrictions on the physical assumptions, data type, and amount that is required for a reconstruction of an object's morphology. It is one of the first publicly available tools to apply interactive graphics to astrophysical modeling. The tool allows astrophysicists to provide a priori knowledge about the object by interactively defining 3D structural elements. By direct comparison of model prediction with observational data, model parameters can then be automatically optimized to fit the observation. The tool has already been successfully used in a number of astrophysical research projects.

Visual exploration of multivariate data typically requires projection onto lower dimensional representations. The number of possible representations grows rapidly with the number of dimensions, and manual exploration quickly becomes ineffective or even unfeasible. This paper proposes automatic analysis methods to extract potentially relevant visual structures from a set of candidate visualizations. Based on features, the visualizations are ranked in accordance with a specified user task. The user is provided with a manageable number of potentially useful candidate visualizations, which can be used as a starting point for interactive data analysis. This can effectively ease the task of finding truly useful visualizations and potentially speed up the data exploration task. In this paper, we present ranking measures for class-based as well as non-class-based scatterplots and parallel coordinates visualizations. The proposed analysis methods are evaluated on different data sets.

Video editing plays an important role in today's cinematic post-production: editing operations are typically applied on a keyframe basis and propagated automatically to the rest of the sequence. Thereby, small inconsistencies in correspondences used for the propagation of editing operations accumulate and need to be corrected manually. The amount of required manual interaction increases further when editing stereoscopic video sequences. In this paper, we propose an algorithm for looped correspondence estimation yielding consistent correspondences that can be used for reliable propagation of editing operations along a stereoscopic video sequence, avoiding drift and error accumulation common to standard approaches. Taking an additional spatially adjacent image into account, we extend standard two-image optical flow algorithms and exploit data redundancy within stereoscopic video sequences. Our proposed algorithm not only yields robust and consistent correspondences for stereo footage, but also improves the accuracy of standard two-image motion estimation as demonstrated for several test scenes with known ground truth.

The theory of compressed sensing has a natural application in interferometric aperture synthesis. As in many real-world applications, however, the assumption of random sampling, which is elementary to many propositions of this theory, is not met. Instead, the induced sampling patterns exhibit a large degree of regularity. In this paper, we statistically quantify the effects of this kind of regularity for the problem of radio interferometry where astronomical images are sparsely sampled in the frequency domain. Based on the favorable results of our statistical evaluation, we present a practical method for interferometric image reconstruction that is evaluated on observational data from the Very Large Array (VLA) telescope.

Abstract This state of the art report covers reconstruction methods for transparent and specular objects or phenomena. While the 3D acquisition of opaque surfaces with Lambertian reflectance is a well‐studied problem, transparent, refractive, specular and potentially dynamic scenes pose challenging problems for acquisition systems. This report reviews and categorizes the literature in this field. Despite tremendous interest in object digitization, the acquisition of digital models of transparent or specular objects is far from being a solved problem. On the other hand, real‐world data is in high demand for applications such as object modelling, preservation of historic artefacts and as input to data‐driven modelling techniques. With this report we aim at providing a reference for and an introduction to the field of transparent and specular object reconstruction. We describe acquisition approaches for different classes of objects. Transparent objects/phenomena that do not change the straight ray geometry can be found foremost in natural phenomena. Refraction effects are usually small and can be considered negligible for these objects. Phenomena as diverse as fire, smoke, and interstellar nebulae can be modelled using a straight ray model of image formation. Refractive and specular surfaces on the other hand change the straight rays into usually piecewise linear ray paths, adding additional complexity to the reconstruction problem. Translucent objects exhibit significant sub‐surface scattering effects rendering traditional acquisition approaches unstable. Different classes of techniques have been developed to deal with these problems and good reconstruction results can be achieved with current state‐of‐the‐art techniques. However, the approaches are still specialized and targeted at very specific object classes. We classify the existing literature and hope to provide an entry point to this exiting field.

Abstract We present an image‐based rendering system to viewpoint‐navigate through space and time of complex real‐world, dynamic scenes. Our approach accepts unsynchronized, uncalibrated multivideo footage as input. Inexpensive, consumer‐grade camcorders suffice to acquire arbitrary scenes, for example in the outdoors, without elaborate recording setup procedures, allowing also for hand‐held recordings. Instead of scene depth estimation, layer segmentation or 3D reconstruction, our approach is based on dense image correspondences, treating view interpolation uniformly in space and time: spatial viewpoint navigation, slow motion or freeze‐and‐rotate effects can all be created in the same way. Acquisition simplification, integration of moving cameras, generalization to difficult scenes and space–time symmetric interpolation amount to a widely applicable virtual video camera system.

Abstract We present a new method for estimating the radiance function of complex area light sources. The method is based on Jensen's photon mapping algorithm. In order to capture high angular frequencies in the radiance function, we incorporate the angular domain into the density estimation. However, density estimation in position‐direction space makes it necessary to find a tradeoff between the spatial and angular accuracy of the estimation. We identify the parameters which are important for this tradeoff and investigate the typical estimation errors. We show how the large data size, which is inherent to the underlying problem, can be handled. The method is applied to different automotive tail lights. It can be applied to a wide range of other real‐world light sources.

Abstract While there are various commercial‐strength editing tools available today for still images, object‐based manipulation of real‐world video footage is still a challenging problem. In this system paper, we present a framework for interactive video editing. Our focus is on footage from a single, conventional video camera. By relying on spatio‐temporal editing techniques operating on the video cube, we do not need to recover 3D scene geometry. Our framework is capable of removing and inserting objects, object motion editing, non‐rigid object deformations, keyframe interpolation, as well as emulating camera motion. We demonstrate how movie shots with moderate complexity can be persuasively modified during post‐processing.

This paper studies the problem of estimating the sub-frame temporal off-set between unsychronized, non-stationary cameras. Based on motion trajec-tory correspondences, the estimation is done in two steps. First, we propose an algorithm to robustly estimate the frame accurate offset by analyzing the trajectories and matching their characteristic time patterns. Using this result, we then show how the estimation of the fundamental matrix between two cameras can be reformulated to yield the sub-frame accurate offset from nine correspondences. We verify the robustness and performance of our approach on synthetic data as well as on real video sequences. 1

Abstract We present a novel multi‐view, projective texture mapping technique. While previous multi‐view texturing approaches lead to blurring and ghosting artefacts if 3D geometry and/or camera calibration are imprecise, we propose a texturing algorithm that warps (“floats”) projected textures during run‐time to preserve crisp, detailed texture appearance. Our GPU implementation achieves interactive to real‐time frame rates. The method is very generally applicable and can be used in combination with many image‐based rendering methods or projective texturing applications. By using Floating Textures in conjunction with, e.g., visual hull rendering, light field rendering, or free‐viewpoint video, improved rendering results are obtained from fewer input images, less accurately calibrated cameras, and coarser 3D geometry proxies.

Abstract The ability to interpolate between images taken at different time and viewpoints directly in image space opens up new possiblities. The goal of our work is to create plausible in‐between images in real time without the need for an intermediate 3D reconstruction. This enables us to also interpolate between images recorded with uncalibrated and unsynchronized cameras. In our approach we use a novel discontiniuity preserving image deformation model to robustly estimate dense correspondences based on local homographies. Once correspondences have been computed we are able to render plausible in‐between images in real time while properly handling occlusions. We discuss the relation of our approach to human motion perception and other image interpolation techniques.

Fluid simulation is one of the most active research areas in computer graphics. However, it remains difficult to obtain measurements of real fluid flows for validation of the simulated data. In this paper, we take a step in the direction of capturing flow data for such purposes. Specifically, we present the first time-resolved Schlieren tomography system for capturing full 3D, non-stationary gas flows on a dense volumetric grid. Schlieren tomography uses 2D ray deflection measurements to reconstruct a time-varying grid of 3D refractive index values, which directly correspond to physical properties of the flow. We derive a new solution for this reconstruction problem that lends itself to efficient algorithms that robustly work with relatively small numbers of cameras. Our physical system is easy to set up, and consists of an array of relatively low cost rolling-shutter camcorders that are synchronized with a new approach. We demonstrate our method with real measurements, and analyze precision with synthetic data for which ground truth information is available.

By jointly applying a model-based marker-less motion capture approach and multi-view texture generation 3D videos of human actors can be reconstructed from multi-view video streams. If the input data were recorded under calibrated lighting, the texture information can also be used to measure time-varying surface reflectance. This way, 3D videos can be realistically displayed under novel lighting conditions. Reflectance estimation is only feasible if the multi-view texture-to-surface registration is consistent over time. In this paper, we propose two image-based warping methods that compensate registration errors due to inaccurate model geometry and shifting of apparel over the body.

Abstract In this paper, we propose a novel framework to represent visual information. Extending the notion of conventional image‐based rendering, our framework makes joint use of both light fields and holograms as complementary representations. We demonstrate how light fields can be transformed into holograms, and vice versa. By exploiting the advantages of either representation, our proposed dual representation and processing pipeline is able to overcome the limitations inherent to light fields and holograms alone. We show various examples from synthetic and real light fields to digital holograms demonstrating advantages of either representation, such as speckle‐free images, ghosting‐free images, aliasing‐free recording, natural light recording, aperture‐dependent effects and real‐time rendering which can all be achieved using the same framework. Capturing holograms under white light illumination is one promising application for future work.

Many problems in computer vision can be formulated as a minimization problem for an energy functional. If this functional is given as an integral of a scalar-valued weight function over an unknown hypersurface, then the sought-after minimal surface can be determined as a solution of the functional's Euler-Lagrange equation. This paper deals with a general class of weight functions that may depend on surface point coordinates as well as surface orientation. We derive the Euler-Lagrange equation in arbitrary dimensional space without the need for any surface parameterization, generalizing existing proofs. Our work opens up the possibility of solving problems involving minimal hypersurfaces in a dimension higher than three, which were previously impossible to solve in practice. We also introduce two applications of our new framework: We show how to reconstruct temporally coherent geometry from multiple video streams, and we use the same framework for the volumetric reconstruction of refractive and transparent natural phenomena, here bodies of flowing water.

We present a new method for real-time rendering of sophisticated lighting effects in and around refractive objects. It enables us to realistically display refractive objects with complex material properties, such as arbitrarily varying refractive index, inhomogeneous attenuation, as well as spatially-varying anisotropic scattering and reflectance properties. User-controlled changes of lighting positions only require a few seconds of update time. Our method is based on a set of ordinary differential equations derived from the eikonal equation, the main postulate of geometric optics. This set of equations allows for fast casting of bent light rays with the complexity of a particle tracer. Based on this concept, we also propose an efficient light propagation technique using adaptive wavefront tracing. Efficient GPU implementations for our algorithmic concepts enable us to render a combination of visual effects that were previously not reproducible in real-time.

By means of passive optical motion capture, real people can be authentically animated and photo-realistically textured. To import real-world characters into virtual environments, however, surface reflectance properties must also be known. We describe a video-based modeling approach that captures human shape and motion as well as reflectance characteristics from a handful of synchronized video recordings. The presented method is able to recover spatially varying surface reflectance properties of clothes from multiview video footage. The resulting model description enables us to realistically reproduce the appearance of animated virtual actors under different lighting conditions, as well as to interchange surface attributes among different people, e.g., for virtual dressing. Our contribution can be used to create 3D renditions of real-world people under arbitrary novel lighting conditions on standard graphics hardware.

This paper proposes a method for analyzing and synthesizing video sequences, specifically suited for image sequences of natural phenomena. We combine a low-dimensional representation of arbitrary image sequences and an image morphing technique to create realistic in-between images. The visualization based on the Isomap algorithm allows users to easily select parts of the video that have periodic character. From these segments, new sequences can be synthesized in realtime. To smooth transition artifacts between the reordered subsequences a Monge-Kantorovich based image morphing method is applied to interpolate in-between images. Our approach is useful for e.g. video key-frame animation, automatic looping or creating slow-motion sequences.

In this paper, we present a video processing algorithm for texture replacement of moving garments in monocular video recordings. We use a color-coded pattern which encodes texture coordinates within a local neighborhood in order to determine the geometric deformation of the texture. A time-coherent texture interpolation is obtained by the use of 3D radial basis functions. Shading maps are determined with a surface reconstruction technique and applied to new textures which replace the color pattern in the video sequence. Our method enables exchanging fabric pattern designs of garments worn by actors as a video post-processing step.

Anyone who has gazed through the eyepiece of an astronomical telescope knows that, with the exception of the Moon and the planets, extra-solar astronomical objects are disappointing to observe visually. This is mainly due to their low surface brightness, but also depends on the visibility, sky brightness and telescope aperture. We propose a system which projects images of astronomical objects (with focus on nebulae and galaxies), animations and additional information directly into the eyepiece view of an astronomical telescope. As the telescope orientation is queried continuously, the projected image is adapted in real-time to the currently visible field of view. For projection, a custom-built video projection module with high contrast and low maximum luminance value was developed. With this technology visitors to public observatories have the option to experience the richness of faint astronomical objects while directly looking at them through a telescope.

We present a recording scheme, image formation model and reconstruction method that enables image-based modeling of flowing bodies of water from multivideo input data. The recorded water is dyed with a fluorescent chemical to measure the thickness of a column of water which leads to an image formation model based on integrated emissivities along a viewing ray. This model allows for a photo-consistency based error measure for a weighted minimal surface, which is recovered using a PDE obtained from the Euler-Lagrangian formulation of the problem. The resulting equation is solved using the level set method.

We reconstruct geometry for a time-varying scene given by a number of video sequences. The dynamic geometry is represented by a 3D hypersurface embedded in space-time. The intersection of the hypersurface with planes of constant time then yields the geometry at a single time instant. In this paper, we model the hypersurface with a collection of triangle meshes, one for each time frame. The photo-consistency error is measured by an error functional defined as an integral over the hypersurface. It can be minimized using a PDE driven surface evolution, which simultaneously optimizes space-time continuity as well. Compared to our previous implementation based on level sets, triangle meshes yield more accurate results, while requiring less memory and computation time. Meshes are also directly compatible with triangle-based rendering algorithms, so no additional post-processing is required.

Stars form in dense clouds of interstellar gas and dust. The residual dust surrounding a young star scatters and diffuses its light, making the star's "cocoon" of dust observable from Earth. The resulting structures, called reflection nebulae, are commonly very colorful in appearance due to wavelength-dependent effects in the scattering and extinction of light. The intricate interplay of scattering and extinction cause the color hues, brightness distributions, and the apparent shapes of such nebulae to vary greatly with viewpoint. We describe an interactive visualization tool for realistically rendering the appearance of arbitrary 3D dust distributions surrounding one or more illuminating stars. Our rendering algorithm is based on the physical models used in astrophysics research. The tool can be used to create virtual fly-throughs of reflection nebulae for interactive desktop visualizations, or to produce scientifically accurate animations for educational purposes, e.g., in planetarium shows. The algorithm is also applicable to investigate on-the-fly the visual effects of physical parameter variations, exploiting visualization technology to help gain a deeper and more intuitive understanding of the complex interaction of light and dust in real astrophysical settings.

In multi-user virtual environments real-world people interact via digital avatars. In order to make the step from the real world onto the virtual stage convincing the digital equivalent of the user has to be personalized. It should reflect the shape and proportions, the kinematic properties, as well as the textural appearance of its real-world equivalent. In this paper, we present a novel spatio-temporal approach to create a personalized avatar from multi-view video data of a moving person. The avatar's geometry is generated by shape-adapting a template human body model. Its surface texture is assembled from multi-view video frames showing arbitrary different body poses. consistent surface texture for the model is generated using multi-view video frames from different camera views and different body poses. With our proposed method photo-realistic human avatars can be robustly generated.

We present a new image-based algorithm for surface reconstruction of moving garment from multiple calibrated video cameras. Using a color-coded cloth texture, we reliably match circular features between different camera views. As surface model we use an a priori known triangle mesh. By identifying the mesh vertices with texture elements we obtain a coherent parameterization of the surface over time without further processing. Missing data points resulting from self-shadowing are plausibly interpolated by minimizing a thin-plate functional. The deforming geometry can be used for different graphics applications, e.g. for realistic retexturing. We show results for real garments demonstrating the accuracy of the recovered flexible shape.

We present a physically-based approach to compute the colors of the sky during the twilight period before sunrise and after sunset. The simulation is based on the theory of light scattering by small particles. A realistic atmosphere model is assumed, consisting of air molecules, aerosols, and water. Air density, aerosols, and relative humidity vary with altitude. In addition, the aerosol component varies in composition and particle-size distribution. This allows us to realistically simulate twilight phenomena for a wide range of different climate conditions. Besides considering multiple Rayleigh and Mie scattering, we take into account wavelength-dependent refraction of direct sunlight as well as the shadow of the Earth. Incorporating several optimizations into the radiative transfer simulation, a photo-realistic hemispherical twilight sky is computed in less than two hours on a conventional PC. The resulting radiometric data is useful, for instance, for high-dynamic range environment mapping, outdoor global illumination calculations, mesopic vision research and optical aerosol load probing.

From our terrestrially confined viewpoint, the actual three-dimensional shape of distant astronomical objects is, in general, very challenging to determine. For one class of astronomical objects, however, spatial structure can be recovered from conventional 2D images alone. So-called planetary nebulae (PNe) exhibit pronounced symmetry characteristics that come about due to fundamental physical processes. Making use of this symmetry constraint, we present a technique to automatically recover the axisymmetric structure of many planetary nebulae from photographs. With GPU-based volume rendering driving a nonlinear optimization, we estimate the nebula's local emission density as a function of its radial and axial coordinates and we recover the orientation of the nebula relative to Earth. The optimization refines the nebula model and its orientation by minimizing the differences between the rendered image and the original astronomical image. The resulting model allows creating realistic 3D visualizations of these nebulae, for example, for planetarium shows and other educational purposes. In addition, the recovered spatial distribution of the emissive gas can help astrophysicists gain deeper insight into the formation processes of planetary nebulae.

We present a model-based approach to encode multiple synchronized video streams depicting a dynamic scene from different viewpoints. With approximate 3D scene geometry available, we compensate for motion as well as disparity by transforming all video images to object textures prior to compression. A two-level hierarchical coding strategy is employed to efficiently exploit inter-texture coherence as well as to ensure quick random access during decoding. Experimental validation shows that attainable compression ratios range up to 50:1 without subsampling. The proposed coding scheme is intended for use in conjunction with free-viewpoint video and 3D-TV applications.

We present a model-based approach to encode multiple synchronized video streams which show a dynamic scene from different viewpoints. By utilizing 3D scene geometry, we compensate for motion and disparity by transforming all video images to object textures prior to compression. A 4D SPIHT wavelet compression algorithm exploits interframe coherence in both temporal and spatial dimension. Unused texels increase the compression, and a shape mask can be omitted at the cost of higher decoder complexity. The proposed coding scheme is intended for use in conjunction with free-viewpoint video and 3D-TV applications.

In computer animation, human motion capture from video is a widely used technique to acquire motion parameters. The acquisition process typically requires an intrusion into the scene in the form of optical markers which are used to estimate the parameters of motion as well as the kinematic structure of the performer. Marker-free optical motion capture approaches exist, but due to their dependence on a specific type of a priori model they can hardly be used to track other subjects, e.g. animals. To bridge the gap between the generality of marker-based methods and the applicability of marker-free methods, we present a flexible non-intrusive approach that estimates both, a kinematic model and its parameters of motion from a sequence of voxel-volumes. The volume sequences are reconstructed from multi-view video data by means of a shape-from-silhouette technique. The described method is well-suited for but not limited to motion capture of human subjects.

Determining the three-dimensional structure of distant astronomical objects is a challenging task, given that terrestrial observations provide only one viewpoint. For this task, bipolar planetary nebulae are interesting objects of study because of their pronounced axial symmetry due to fundamental physical processes. Making use of this symmetry constraint, we present a technique to automatically recover the axisymmetric structure of bipolar planetary nebulae from two-dimensional images. With GPU-based volume rendering driving a nonlinear optimization, we estimate the nebula's local emission density as a function of its radial and axial coordinates, and we recover the orientation of the nebula relative to Earth. The optimization refines the nebula model and its orientation by minimizing the differences between the rendered image and the original astronomical image. The resulting model enables realistic 3D visualizations of planetary nebulae, e.g. for educational purposes in planetarium shows. In addition, the recovered spatial distribution of the emissive gas allows validating computer simulation results of the astrophysical formation processes of planetary nebulae.

Virtual Reality thrives on interactivity, realism, and increasingly also animation. The virtual world is not a static place anymore: dynamic entities mimicing natural phenomena are finding their way into computer games and special effects. Typically, physics-based models or ad-hoc behavioral descriptions are drawn on to emulate water waves, flames, smoke, cloth motion, ... For interactive VR applications, unfortunately, simulating complex physical processes is often too time-consuming, while, on the other hand, simplified model descriptions yield un-natural, artifical animation results.Alternatively, natural events may be acquired from the real thing. Given a handful of synchronized video recordings, this talk presents examples how complex, time-varying natural phenomena may be modeled from reality to be incorporated into time-critical 3D graphics applications. The reward are photo-realistic rendering results and truly authentic animations.

For realistic animation of an artificial character a body model that represents the character's kinematic structure is required. Hierarchical skeleton models are widely used which represent bodies as chains of bones with interconnecting joints. In video motion capture, animation parameters are derived from the performance of a subject in the real world. For this acquisition procedure too, a kinematic body model is required. Typically, the generation of such a model for tracking and animation is, at best, a semi-automatic process. We present a novel approach that estimates a hierarchical skeleton model of an arbitrary moving subject from sequences of voxel data that were reconstructed from multi-view video footage. Our method does not require a-priori information about the body structure. We demonstrate its performance using synthetic and real data.

Abstract This paper presents an efficient hardware‐accelerated method for novel view synthesis from a set of images or videos. Our method is based on the photo hull representation, which is the maximal photo‐consistent shape. We avoid the explicit reconstruction of photo hulls by adopting a view‐dependent plane‐sweeping strategy. From the target viewpoint slicing planes are rendered with reference views projected onto them. Graphics hardware is exploited to verify the photo‐consistency of each rasterized fragment. Visibilities with respect to reference views are properly modeled, and only photo‐consistent fragments are kept and colored in the target view. We present experiments with real images and animation sequences. Thanks to the more accurate shape of the photo hull representation, our method generates more realistic rendering results than methods based on visual hulls. Currently, we achieve rendering frame rates of 2–3 fps. Compared to a pure software implementation, the performance of our hardware‐accelerated method is approximately 7 times faster. Categories and Subject Descriptors (according to ACM CCS): CR Categories: I.3.3 [Computer Graphics]: Picture/Image Generation; I.3.7 [Computer Graphics]: Three‐Dimensional Graphics and Realism.

Athletes and coaches in most professional sports make use of high-tech equipment to analyze and, subsequently, improve the athlete's performance. High-speed video cameras are employed, for instance, to record the swing of a golf club or a tennis racket, the movement of the feet while running, and the body motion in apparatus gymnastics. High-tech and high-speed equipment, however, usually implies high-cost as well. In this paper, we present a passive optical approach to capture high-speed motion using multi-exposure images obtained with low-cost commodity still cameras and a stroboscope. The recorded motion remains completely undisturbed by the motion capture process. We apply our approach to capture the motion of hand and ball for a variety of baseball pitches and present algorithms to automatically track the position, velocity, rotation axis, and spin of the ball along its trajectory. To demonstrate the validity of our setup and algorithms, we analyze the consistency of our measurements with a physically based model that predicts the trajectory of a spinning baseball. Our approach can be applied to capture a wide variety of other high-speed objects and activities such as golfing, bowling, or tennis for visualization as well as analysis purposes.

This paper deals with simultaneous depth map estimation and background separation in a multi-view setting with several fixed calibrated cameras, two problems which have previously been addressed separately. We demonstrate that their strong interdependency can be exploited elegantly by minimizing a discrete energy functional, which evaluates both properties at the same time. Our algorithm is derived from the powerful "multi-camera scene reconstruction via graph cuts" algorithm presented by Kolmogorov and Zabih (2002). Experiments with both real-world as well as synthetic scenes demonstrate that the presented combined approach yields even more correct depth estimates. In particular, the additional information gained by taking background into account increases considerably the algorithm's robustness against noise.

We present a hardware-accelerated method for video-based rendering relying on an approximate model of scene geometry. Our goal is to render high-quality views of the scene from arbitrary viewpoints in real-time, using as input synchronized video-streams from only a small number of calibrated cameras distributed around the scene. Despite the fact that only a very coarse geometry reconstruction is possible, our rendering approach based on textured billboards preserves the details present in the source images. By exploiting multi-texturing and blending facilities of modern graphics cards, we achieve real-time frame-rates on current off-the-shelf hardware. One of the possible applications of our algorithm is to use it in an inexpensive system to display 3D-videos, which the user can watch from an interactively chosen viewpoint, or ultimately even live 3D-television.

Given the illumination and the surface normal at a point on a convex surface, the effect of small changes in surface normal direction on observed reflected radiance is examined. A signal-processing framework suitable for the hemispherical geometry of the problem is applied to discuss surface reflection in terms of convolving incident illumination with the reflectance function. Depending on the surface material's reflection properties as well as on the incident illumination distribution, normal direction deviation may or may not be noticeable. Expressions are derived to quantitatively evaluate the amount of change in reflected radiance. Different scenarios exemplify the results. The presented theoretical results have implications for shape-from-shading techniques as well as for scene-based compression methods.

High-quality nonintrusive human motion capture is necessary for acquisition of model-based free-viewpoint video of human actors. Silhouette-based approaches have demonstrated that they are able to accurately recover a large range of human motion from multiview video. However, they fail to make use of all available information, specifically that of texture information. This paper presents an algorithm that uses motion fields constructed from optical flow in multiview video sequences. The use of motion fields augments the silhouette-based method by incorporating texture-information into the tracking process. The algorithm is a key-component in a larger free-viewpoint video system of human actors. Our results demonstrate that our method accurately estimates pose parameters and allows for realistic texture generation in 3D video sequences.

Acquiring online the evolving shape of a dynamic scene from a handful of video streams may be considered one of the most challenging, but at the same time also most auspicious tasks in contemporary computer graphics and computer vision research. The anticipation of revolutionary new applications such as interactive 3D television broadcasts motivates the ongoing work on free-viewpoint video rendering. The paper aims at giving a state-of-progress report on this lively research endeavor. Different acquisition setups and online reconstruction approaches are exemplified. Yielding interactive frame rates, depth map-based techniques, polyhedral as well as volumetric visual hull reconstruction approaches, and combined methods employing visual hull-guided depth map estimation are presented. The experience gained with these approaches allows us to identify future research directions towards real-time analysis and high-quality synthesis of dynamic, real-world scenes.

Abstract This paper presents a rendering method for translucent objects, in which viewpoint and illumination can be modified at interactive rates. In a preprocessing step, the impulse response to incoming light impinging at each surface point is computed and stored in two different ways: The local effect on close‐by surface points is modeled as a per‐texel filter kernel that is applied to a texture map representing the incident illumination. The global response (i.e. light shining through the object) is stored as vertex‐to‐vertex throughput factors for the triangle mesh of the object. During rendering, the illumination map for the object is computed according to the current lighting situation and then filtered by the precomputed kernels. The illumination map is also used to derive the incident illumination on the vertices which is distributed via the vertex‐to‐vertex throughput factors to the other vertices. The final image is obtained by combining the local and global response. We demonstrate the performance of our method for several models. ACM CSS:I.3.7 Computer Graphics—Three‐Dimensional Graphics and Realism Color Radiosity

To store and transmit the large amount of image data necessary for Image-based Rendering (IBR), efficient coding schemes are required. This paper presents two different approaches which exploit three-dimensional scene geometry for multi-view compression. In texture-based coding, images are converted to view-dependent texture maps for compression. In model-aided predictive coding, scene geometry is used for disparity compensation and occlusion detection between images. While both coding strategies are able to attain compression ratios exceeding 2000:1, individual coding performance is found to depend on the accuracy of the available geometry model. Experiments with real-world as well as synthetic image sets show that texture-based coding is more sensitive to geometry inaccuracies than predictive coding. A rate-distortion theoretical analysis of both schemes supports these findings. For reconstructed approximate geometry models, model-aided predictive coding performs best, while texture-based coding yields superior coding results if scene geometry is exactly known.

In free-viewpoint video, the viewer can interactively choose his viewpoint in 3-D space to observe the action of a dynamic real-world scene from arbitrary perspectives. The human body and its motion plays a central role in most visual media and its structure can be exploited for robust motion estimation and efficient visualization. This paper describes a system that uses multi-view synchronized video footage of an actor's performance to estimate motion parameters and to interactively re-render the actor's appearance from any viewpoint.The actor's silhouettes are extracted from synchronized video frames via background segmentation and then used to determine a sequence of poses for a 3D human body model. By employing multi-view texturing during rendering, time-dependent changes in the body surface are reproduced in high detail. The motion capture subsystem runs offline, is non-intrusive, yields robust motion parameter estimates, and can cope with a broad range of motion. The rendering subsystem runs at real-time frame rates using ubiquous graphics hardware, yielding a highly naturalistic impression of the actor. The actor can be placed in virtual environments to create composite dynamic scenes. Free-viewpoint video allows the creation of camera fly-throughs or viewing the action interactively from arbitrary perspectives.

This paper presents a rendering method for translucent objects, in which view point and illumination can be modified at interactive rates. In a preprocessing step the impulse response to incoming light impinging at each surface point is computed and stored in two different ways: The local effect on close-by surface points is modeled as a per-texel filter kernel that is applied to a texture map representing the incident illumination. The global response (i.e. light shining through the object) is stored as vertex-to-vertex throughput factors for the triangle mesh of the object. During rendering, the illumination map for the object is computed according to the current lighting situation and then filtered by the precomputed kernels. The illumination map is also used to derive the incident illumination on the vertices which is distributed via the vertex-to-vertex throughput factors to the other vertices. The final image is obtained by combining the local and global response. We demonstrate the performance of our method for several models.

The acquisition of human motion data is of major importance for creating interactive virtual environments, intelligent user interfaces, and realistic computer animations. Today's performance of off-the-shelf computer hardware enables marker-free non-intrusive optical tracking of the human body. In addition, recent research shows that it is possible to efficiently acquire and render volumetric scene representations in real-time. This paper describes a system to capture human motion at interactive frame rates without the use of markers or scene-intruding devices. Instead, 2D computer vision and 3D volumetric scene reconstruction algorithms are applied directly to the image data. A person is recorded by multiple synchronized cameras, and a multilayer hierarchical kinematic skeleton is fitted to each frame in a two-stage process. We present results with a prototype system running on two PCs.

Multi-view image coding benefits from knowledge of the depicted scene's 3D geometry. To exploit geometry information for compression, two different approaches can be distinguished. In texture-based coding, images are converted to texture maps prior to compression. In image-based predictive coding, geometry is used for disparity compensation and occlusion detection between images. Coding performance of both approaches depends on the accuracy of the available geometry model. Texture-based and image-based coding are compared with regard to the influence of geometry accuracy on coding efficiency. The results are theoretically explained. Experiments with natural as well as synthetic image sets show that texture-based coding is more sensitive to small geometry inaccuracies than image-based coding. For approximate geometry models, image-based coding performs best, while texture-based coding yields superior coding results if scene geometry is exactly known.

Two compression schemes for coding multiple images of a static scene are compared. Both codecs apply approximate 3-D scene geometry: in model-aided coding, images are predicted by model-driven disparity compensation, while the model-based coder uses the geometry to convert images into texture maps for compression. Images of real-world objects are used to evaluate both codecs. The coding performance is examined subject to varying geometry accuracy. Compression factors of up to 1000:1 at an acceptable reconstruction quality verify the usefulness of the 3-D geometry for coding multi-viewpoint imagery.

S.919-922

Two light-field compression schemes are presented. The codecs are compared with regard to compression efficiency and rendering performance. The first proposed coder is based on video-compression techniques that have been modified to code the four-dimensional light-field data structure efficiently. The second coder relies entirely on disparity-compensated image prediction, establishing a hierarchical structure among the light-field images. The coding performance of both schemes is evaluated using publicly available light fields of synthetic, as well as real-world, scenes. Compression ratios vary between 100:1 and 2000:1, depending on the reconstruction quality and light-field scene characteristics.

This paper highlights recent advances in image compression aided by 3-D geometry information. As two examples, we present a model-aided video coder for efficient compression of head-and-shoulder scenes and a geometry-aided coder for 4-D light fields for image-based rendering. Both examples illustrate that an explicit representation of 3-D geometry is advantageous if many views of the same 3-D object or scene have to be encoded. Waveform-coding and 3-D model-based coding can be combined in a rate-distortion framework, such that the generality of waveform coding and the efficiency of 3-D models are available where needed.

A coder for light fields is presented. Due to the large amount of data needed to represent a complete light field, a hierarchical decomposition is employed. The full light field is built up by recursively predicting intermediate images from a small subset of light-field images. Intermediate images are predicted by disparity-compensating multiple surrounding images. The predicted images are refined using DCT coding. Rate-distortion measurements for two standard light fields verify the efficiency of the proposed scheme. Compression ratios of 1000:1 are achieved at acceptable quality of the rendered views.