Interacting with real-time rendered 3D content from powerful machines on smaller devices is becoming ubiquitous through commercial products that enable in-home streaming within the same local network.However, support for high resolution, low latency in-home streaming at high image quality is still a challenging problem.To enable this, we enhance an existing open source framework for in-home streaming.We add highly optimized DXT1 (DirectX Texture Compression) support for thin desktop and notebook clients.For rendered light fields, we improve the encoding algorithms for higher image quality.Within a 10 Gigabit Ethernet (10 GbE) network, we achieve streaming up to 5K resolution at 55 frames per second.Through new low-level algorithmic improvements, we increase the compression speed of ETC1 (Ericsson Texture Compression) by a factor of 5. We are the first to bring ETC2 compression to real-time speed, which increases the streamed image quality.Last, we reduce the required data rate by more than a factor of 2 through foveated compression with real-time eye tracking.

With the adaption of virtual reality in the consumer space, spherical panorama photos are gaining popularity. Through wide-angle head-mounted displays, they can be experienced in a natural way and offer the user an immersive view of the captured scene. While being used in virtual reality, the alignment of the saved image does not matter much. However, when displaying the panorama on a 2D screen, the alignment can make a difference on how pleasant the image looks. We propose an automatic method to do lossless shifting of the image to make it look better on 2D screens.

With increasing spatial and temporal resolution in head-mounted displays (HMDs), using eye trackers to adapt rendering to the user is getting important to handle the rendering workload. Besides using methods like foveated rendering, we propose to use the current visual field for rendering, depending on the eye gaze. We use two effects for performance optimizations. First, we noticed a lens defect in HMDs, where depending on the distance of the eye gaze to the center, certain parts of the screen towards the edges are not visible anymore. Second, if the user looks up, he cannot see the lower parts of the screen anymore. For the invisible areas, we propose to skip rendering and to reuse the pixels colors from the previous frame. We provide a calibration routine to measure these two effects. We apply the current visual field to a renderer and get up to 2x speed-ups.

Applications like rendering of images using computer graphics methods are usually requiring more sophisticated light models to give better control. Complex scenes in computer generated images are requiring very differentiated light models to give a realistic rendering of the scene. That usually includes a high number of (virtual) light sources to model a scene to reproduce accurate shadows and shadings. In particular in the production of visual effects for movies and TV the real scene lighting needs to be captured very accurately to give a realistic rendering of virtual objects into that scene. In this context the light modeling is usually done manually by skilled artists in a time consuming process. This contribution describes a new technique for estimation of discrete spot light sources. The method uses a consumer grade DSLR camera equipped with a fisheye lens to capture light probe images registered to the scene. From these probe images the geometric and radiometric properties of the dominant light sources in the scene are estimated. The first step is a robust approach to identify light sources in the light probes and to find exact positions by triangulation. Then the light direction and radiometric fall-off properties are formulated and estimated in a least square minimization approach. There are a number of advantages in our approach. First, the probing camera is registered using a multi-camera setup which requires the minimum amendments to the studio. Second, we are not limited to any specific probing object since the properties of each light are estimated based on processing the probe images. In addition, since the probing camera can move freely in the area of interest, there are no limits in terms of the covered space. Large field of view of the fisheye lens is also beneficial in this matter. Calibration and Registration of Cameras. We propose a two-step calibration and registration approach. In the first step, a planar asymmetric calibration pattern is used for simultaneous calibration of the intrinsics and the pose of all the witness cameras and the principal camera using a bundle adjustment module. In the next step, parameters of witness cameras are kept fixed and the probing camera is registered in the same coordinate system by using color features of an attached calibration rig. Position Estimation. To estimate the 3D position vectors of the light sources, one needs to shoot rays from every detected light blob in all probe images and triangulate the corresponding rays from at least two probe positions for each source. Figure 1 summarizes the required steps.

Lenses in modern consumer HMDs introduce distortions like astigmatism: only the center area of the displayed content can be perceived sharp while with increasing distance from the center the image gets out of focus. We show with three new approaches that this undesired side effect can be used in a positive way to save calculations in blurry areas. For example, using sampling maps to lower the detail in areas where the image is blurred through astigmatism, increases performance by a factor of 2 to 3. Further, we introduce a new calibration of user-specific viewing parameters that increase the performance by about 20-75%.

Today, mobile devices like smartphones and tablets are becoming more powerful and exhibit enhanced 3D graphics performance.However, the overall computing power of these devices is still limited regarding usage scenarios like photorealistic gaming, enabling an immersive virtual reality experience or real-time processing and visualization of big data.To overcome these limitations application streaming solutions are constantly gaining focus.The idea is to transfer the graphics output of an application running on a server or even a cluster to a mobile device, conveying the impression that the application is running locally.User inputs on the mobile client side are processed and sent back to the server.The main criteria for successful application streaming are low latency, since users want to interact with the scene in near real-time, as well as high image quality.Here, we present a novel application framework suitable for streaming applications from high-end machines to mobile devices.Using real-time ETC1 compression in combination with a distributed rendering architecture we fully leverage recent progress in wireless computer networking standards (IEEE 802.11ac) for mobile devices, achieving much higher image quality at half the latency compared to other inhome streaming solutions.

Abstract Removing dynamic objects from videos is an extremely challenging problem that even visual effects professionals often solve with time‐consuming manual frame‐by‐frame editing. We propose a new approach to video completion that can deal with complex scenes containing dynamic background and non‐periodical moving objects. We build upon the idea that the spatio‐temporal hole left by a removed object can be filled with data available on other regions of the video where the occluded objects were visible. Video completion is performed by solving a large combinatorial problem that searches for an optimal pattern of pixel offsets from occluded to unoccluded regions. Our contribution includes an energy functional that generalizes well over different scenes with stable parameters, and that has the desirable convergence properties for a graph‐cut‐based optimization. We provide an interface to guide the completion process that both reduces computation time and allows for efficient correction of small errors in the result. We demonstrate that our approach can effectively complete complex, high‐resolution occlusions that are greater in difficulty than what existing methods have shown.

3D Television and Immersive Cinema, Virtual and Augmented Reality, do we enter soon a total space of polysensual illusion? The aim of this contribution is to create an understanding that the present image revolution using indeed new technologies has also developed a large number of so far unknown visual expressions that cannot be conceived without our image history. Art History and Image Science help in understanding the leading and forming functions of today´s image worlds in our society. With the history of illusion and immersion, the history of artificial life or the tradition of telepresence, Image Science offers sub-histories of the present image revolutions. We know that a central problem of current cultural policy stems from serious lack of knowledge about the origins of the audiovisual media and this stands in complete contradistinction to current demands for more media and image competence. Social media competence, which goes beyond mere technical skills, is difficult to acquire if the area of historic media experience is excluded. Although many people view the concept of presence, virtual or mixed realities as a totally new phenomenon, it has its foundations in an unrecognized history of immersive images. Immersion is undoubtedly a key to any understanding of the development of media in general. Overseeing 2000 years of immersive images and by bringing them in a relativity with the image competence of their users, this talk aims to gain distance and with that a reflective thinking space towards the desire to create ever new immersive image experiences.

An approach for the generation of 3D surface model sequences from a set of synchronised, calibrated cameras is presented. The method is based on the computation of the visual-hull. An analysis classifies typical error sources for artefacts in the reconstructed 3D models into occlusion, approximation and sampling errors. For the synthesis of new images for TV- and film-applications the roughness of the reconstructed 3D models has shown to be very disturbing. In order to reduce this roughness two methods are used sequentially: a new super-sampling approach that reduces the sampling error, and a Gaussian surface smoothing. The application of the modelling results from a 12 camera studio system is shown for the generation of virtual views of moving actors in standard post-production packages.

A new studio system for the production of three-dimensional (3-D) content is introduced. The system combines the ability to capture dynamic scenes, based on a multicamera system in a chroma-key environment, with a view-dependent projection for actor feedback. The system allows the generation and rendering of 3-D models in preview quality for on-set visualization in real time and in high quality for postproduction applications in an offline phase. The shape reconstruction is based on the computation of the visual hull. The view-dependent projection component allows actors to be immersed into a virtual scene and to interact with virtual components. The functional modules of the system were implemented in a highly distributed system using standard, inexpensive IT components. Therefore, the system is quite scalable and can be fitted into most studio environments. For the integration of the system components, a middleware architecture was developed.

No abstract available.

A scene analysis system for automated 3-D modeling of buildings is presented. It combines surface reconstruction techniques with object recognition to generate 3-D models for computer graphics applications. The system permits the insertion of high level constraints, like a specific angle between two house walls, in an explicit knowledge base as a semantic net. The applicability of those constraints is proved by asserting and testing hypotheses in an interpretation phase. In the case of rejection, a more general constraint or model is selected. The capabilities of the system were shown for the modeling of buildings using depth from stereo and contour information. The system reconstructs the surface of scene objects using constraints selected in the prior interpretation.