Volumetric video is regarded worldwide as the next important development step in the field of media production. Especially in the context of the extremely rapid development of the Virtual Reality (VR) and Augmented Reality (AR) markets, volumetric video is becoming a key technology. In this paper, a new capture and processing system for volumetric video is presented, called 3D Human Body Reconstruction (3DHBR). The system is based on 16 stereo pairs of high-resolution cameras capturing a moving person in 360 degree. A novel stereo approach provides depth information from all perspectives, which is then fused to a single consistent 3D point cloud. A meshing and mesh reduction algorithm finally produces a sequence of meshes that can be integrated into common render engines. Given that, an integration of realistic dynamic 3D reconstructions of moving persons in VR and AR applications is possible.

In this work, a novel and fast algorithm for real-time 3D body reconstruction from stereo sequences is proposed. The main contributions of this work consist of a novel approach for a statistically guided stereo processing and a data parallel iteration scheme for 3D estimation that includes temporal predecessors from a local spatial neighborhood. A purely GPU based implementation is provided that exhibits a nearly linear scaling of the runtime with respect to the number of GPUs. This leads to an inherent sub-pixel processing due to the availability of hardware supported texture lookups. Our implementation is able to process 4K (UHD) stereo streams on a 4×4 grid with 30 fps on a single state-of-the-art consumer graphics card. The algorithmic performance of our approach is demonstrated in the context of an immersive TV application.

It is widely accepted that the emerging concept of immersive media is one of the key technologies for future entertainment. One feature of immersive media is to use large panoramic screens with ultrahigh resolution. This technology requires a completely new production grammar that is not necessarily compatible with conventional production formats. Against this background the paper describes a new approach called format-agnostic production that bridges the gap between different target formats and applications. It does not use fixed numbers of frames, lines and pixels, or even geometry. The philosophy is to merge video signals from different camera sensors with different spatio-temporal resolutions and dynamic ranges to a wide-angle panoramic view of the scene. Hence, to support a wide range of applications from mobile devices to immersive media, any desired shot framing and resolution can be obtained by scanning, panning, or zooming. Apart from a general description of the concept and the system architecture, the paper will mainly focus on technical details of ultrahigh-resolution panoramic imaging which plays a central role in the concept of format-agnostic production.

This paper gives an overview of the state-of-the-art in 3-D video postproduction and processing as well as an outlook to remaining challenges and opportunities. First, fundamentals of stereography are outlined that set the rules for proper 3-D content creation. Manipulation of the depth composition of a given stereo pair via view synthesis is identified as the key functionality in this context. Basic algorithms are described to adapt and correct fundamental stereo properties such as geometric distortions, color alignment, and stereo geometry. Then, depth image-based rendering is explained as the widely applied solution for view synthesis in 3-D content creation today. Recent improvements of depth estimation already provide very good results. However, in most cases, still interactive workflows dominate. Warping-based methods may become an alternative for some applications in the future, which do not rely on dense and accurate depth estimation. Finally, 2-D to 3-D conversion is covered, which is an important special area for reuse of existing legacy 2-D content in 3-D. Here various advanced algorithms are combined in interactive workflows.

This paper gives an overview on the framework of production rules that are required for the acquisition of adequate stereo content. It is well known that if stereo is not produced properly, consequences for the human 3-D perception might be eye strain and visual fatigue. Therefore, to avoid discomfort, stereographers use a production grammar that reflects extensive knowledge and experience in the fields of 3-D perception, stereo geometry, video technology, and content creation. Some of these rules are of heuristic nature, and others are motivated by perceptual, geometric, or technical aspects. The application of these rules at the set usually results in time-consuming manual work for rigging and adjusting stereo cameras before shooting. Against this background the paper reviews relevant aspects of 3-D perception, the fundamentals of stereo geometry, and the main rules of stereo production. Finally, it describes the functionality of camera assistance systems that have recently been proposed for stereo acquisition and production in order to ease the work of stereographers at the set and, with it, to save time and to make stereo productions more efficient.

The paper discusses an assistance system for stereo shooting and 3D production, called Stereoscopic Analyzer (STAN). A feature-based scene analysis estimates in real-time the relative pose of the two cameras in order to allow optimal camera alignment and lens settings directly at the set. It automatically eliminates undesired vertical disparities and geometrical distortions through image rectification. In addition, it detects the position of near- and far objects in the scene to derive the optimal inter-axial distance (stereo baseline), and gives a framing alert in case of stereoscopic window violation. Against this background the paper describes the system architecture, explains the theoretical background and discusses future developments.

In this paper we present an efficient high resolution image based visual hull (IBVH) algorithm that entirely runs in real-time on a single consumer graphics card. The target application is a real-time 3D video conferencing system. One major contribution of this paper is a novel caching strategy for the reduction of line segment intersection tests. In contrast to existing approaches, it additionally allows us to pre-select a close estimation of the set of relevant pixels in the desired view. Based on this, we obtain a significant computational speedup, especially for high resolutions. Further, we propose an efficient way to use IBVH for the generation of voxel equivalent 3D models. We compare our techniques in terms of resolution and runtime with state of the art real-time multi GPU voxel based approaches. Our experiments show that we achieve a speed-up by a factor of five and more for high resolutions.

Recently, popularity of 3D video has been growing significantly and it may turn into a home user mass market in the near future. However, diversity of 3D video content formats is still hampering wide success. An overview of available and emerging 3D video formats and standards is given, which are mostly related to specific types of applications and 3D displays. This includes conventional stereo video, multiview video, video plus depth, multiview video plus depth and layered depth video. Features and limitations are explained. Finally, depth enhanced stereo (DES) is introduced as a flexible, generic, and efficient 3D video format that can unify all others and serve as universal 3D video format in the future.

A system for video on multiscopic 3D displays is considered where the data representation consists of multiview video plus scene depth. At most, 3 multiview video signals are being transmitted and used together with the depth data to generate intermediate views at the receiver. The paper presents an approach to such an intermediate view interpolation that separates unreliable image regions along depth discontinuities from reliable image regions. These image regions are processed with different algorithms and then fused to obtain the final interpolated view. In contrast to previous layered approaches, two boundary layers and one reliable layer is used. Moreover, the presented technique does not rely on 3D graphics support but uses image-based 3D warping instead. For enhanced quality intermediate view generation, hole-filling and filtering methods are described. As a result, high quality intermediate views for an existing 9-view auto-stereoscopic display are presented, which prove the suitability of the approach for advanced 3D video (3DV) systems.

In this paper, a system for video rendering on multiscopic 3D displays is considered where the data is represented as layered depth video (LDV). This representation consists of one full or central video with associated per-pixel depth and additional residual layers. Thus, only one full view with additional residual data needs to be transmitted. The LDV data is used at the receiver to generate all intermediate views for the display. The paper presents the LDV layer extraction as well as the view synthesis, using a scene reliability-driven approach. Here, unreliable image regions are detected and in contrast to previous approaches the residual data is enlarged to reduce artifacts in unreliable areas during rendering. To provide maximum data coverage, the residual data remains at its original positions and will not be projected towards the central view. The view synthesis process also uses this reliability analysis to provide higher quality intermediate views than previous approaches. As a final result, high quality intermediate views for an existing 9-view auto-stereoscopic display are presented, which prove the suitability of the LDV approach for advanced 3D video (3DV) systems.

Image cube trajectory (ICT) analysis is a new and robust method to estimate the 3D structure of a scene from a set of 2D images. For a moving camera each 3D point is represented by a trajectory in a so called image cube. In our previous work we have shown that it is possible to reconstruct the 3D scene from the parameters of these trajectories. A key component for this process is the trajectory detection within the cube. It is based on the image cube parameterization as well as the robust estimation of the trajectory color and trajectory color variation. In this paper we will focus on the second problem in more detail. We propose an algorithm which estimates the trajectory parameters in sub-pixel resolution with high accuracy. The corresponding 3D scene structure can be reconstructed with high level of detail even for complex scenes, multiple occlusions and very fine structures.

The paper discusses an advanced approach for 3DTV services that is based on the concept of an N× video-plus-depth data representation. It particularly considers aspects of interoperability, scalability, and adaptability for the case that different multi-baseline geometries are used for multi-view capturing and 3D reproduction. In addition, it presents a method for the creation of depth maps and an algorithm for depth-image-based rendering related to the system approach.

Block-based video-coding for 4:4:4 color sampling is extended by an adaptive color space transform. The presented technique enables an encoder to switch between several given color space representations in order to optimize rate-distortion performance. Simulations based on the current draft of the H.264/MPEG4-AVC 4:4:4 extensions demonstrate that our technique provides a rate-distortion performance equal or better than that obtained when using any of the individual color spaces only.

In future video communication services, the user's communication device, such as PC, laptop, PDA or mobile phone is equipped with new interaction modalities. These can be cameras and microphones on the capturing side and speech synthesis and video/3D graphics on the rendering side. Haptic and tactile interfaces become also available. These modalities help the user to interact more intuitive with complex devices and tools and provide new services. Hence, a key challenge of new modalities is robustness and stability under general conditions in arbitrary environments. Furthermore, inexperienced users should be able to use these new capabilities without dedicated knowledge of device settings or algorithms. In this paper, we will present some key components for a robust vision-based user interface, which are integrated in an advanced future video communication service

Interoperability, scalability and adaptability are important features for a successful introduction of future 3D TV services. Hence, new concepts must be able to adapt the multi-view geometry of the capturing system to the geometry of the 3D reproduction systems. An approach is discussed, which considers these adaptation issues based on the concept of an Ntimesvideo-plus-depth data representation. The core algorithms for depth map creation on the analysis side and depth image based rendering on the reproduction side are presented

An overview of 3D and free viewpoint video is given in this paper with special focus on related standardization activities in MPEG. Free viewpoint video allows the user to freely navigate within real world visual scenes, as known from virtual worlds in computer graphics. Examples are shown, highlighting standards conform realization using MPEG-4. Then the principles of 3D video are introduced providing the user with a 3D depth impression of the observed scene. Example systems are described again focusing on their realization based on MPEG-4. Finally multi-view video coding is described as a key component for 3D and free viewpoint video systems. The conclusion is that the necessary technology including standard media formats for 3D and free viewpoint is available or will be available in the near future, and that there is a clear demand from industry and user side for such applications. 3D TV at home and free viewpoint video on DVD will be available soon, and will create huge new markets

This paper describes an approach for creating high-resolution video panoramas of large-scale sport events. In an exemplary football scenario, we used two ARRIFLEX D-20 "film-style" digital cameras to capture both sides of the playing field as well as large parts of the stands. From the recorded left and right view images, we created a joint panoramic view with a resolution of 501 6x1400 pel (5k) using sophisticated image processing algorithms. These highly immersive football panoramas were screened with our modular, high-resolution multiprojection system during the FIFA World Cup 2006 in a 600 seat CinemaxX movie theater in Berlin providing the viewers with the feeling of actually watching the game from a good seat on the stands.

Image cube trajectory (ICT) Analysis is a new and robust method to estimate the 3D structure of a scene from a set of 2D images. For a moving camera each 3D point is represented by a trajectory in an image cube. In previous publications we have shown that it is possible to reconstruct the 3D scene from the parameters of these trajectories. Nevertheless, the algorithm was restricted to simple parameterized camera movements. In this paper we discuss the problem of arbitrary camera motion. We benefit from the fact that in many cases, in particular for hand-held cameras, the image deviations caused by rotational variation of the camera parameters is much higher than for translational variation. We show that it is possible to compensate such rotational and translational deviations by transformation and resampling of the image cube. We obtain more uniform and smooth trajectory structures which can be analyzed by standard ICT analysis algorithms.

Interactivity in the sense of being able to explore and navigate audio-visual scenes by freely choosing viewpoint and viewing direction, is an important key feature of new and emerging audio-visual media. This paper gives an overview of suitable technology for such applications, with a focus on international standards, which are beneficial for consumers, service providers, and manufacturers. We first give a general classification and overview of interactive scene representation formats as commonly used in computer graphics literature. Then, we describe popular standard formats for interactive three-dimensional (3-D) scene representation and creation of virtual environments, the virtual reality modeling language (VRML), and the MPEG-4 BInary Format for Scenes (BIFS) with some examples. Recent extensions to MPEG-4 BIFS, the Animation Framework eXtension (AFX), providing advanced computer graphics tools, are explained and illustrated. New technologies mainly targeted at reconstruction, modeling, and representation of dynamic real world scenes are further studied. The user shall be able to navigate photorealistic scenes within certain restrictions, which can be roughly defined as 3-D video. Omnidirectional video is an extension of the planar two-dimensional (2-D) image plane to a spherical or cylindrical image plane. Any 2-D view in any direction can be rendered from this overall recording to give the user the impression of looking around. In interactive stereo two views, one for each eye, are synthesized to provide the user with an adequate depth cue of the observed scene. Head motion parallax viewing can be supported in a certain operating range if sufficient depth or disparity data are delivered with the video data. In free viewpoint video, a dynamic scene is captured by a number of cameras. The input data are transformed into a special data representation that enables interactive navigation through the dynamic scene environment.

We present a new approach to video coding that applies video analysis based on global motion features. A super-resolution mosaic is build for each frame to be encoded from a number of previously transmitted frames. This super-resolution mosaic is used to detect macroblocks that are only affected by global motion. For such macroblocks no prediction error is transmitted. They are purely reconstructed by prediction from the super-resolution mosaic which results in significant bitrate savings. Our results indicate total bitrate savings of 20% and more compared to a state of the art H.264/AVC codec at the same visual quality.

Real-time stereo analysis is an important research area in computer vision. In this context, we propose a stereo algorithm for an immersive video-conferencing system by which conferees at different geographical places can meet under similar conditions as in the real world. For this purpose, virtual views of the remote conferees are generated and adapted to the current viewpoint of the local participant. Dense vector fields of high accuracy are required in order to guarantee an adequate quality of the virtual views. Due to the usage of a wide baseline system with strongly convergent camera configurations, the dynamic disparity range is about 150 pixels. Considering computational costs, a full search or even a local search restricted to a small window of a few pixels, as it is implemented in many real-time algorithms, is not suitable for our application because processing on full-resolution video according to CCIR 601 TV standard with 25 frames per second is addressed-the most desirable as a pure software solution running on available processors without any support from dedicated hardware. Therefore, we propose in this paper a new fast algorithm for stereo analysis, which circumvents the window search by using a hybrid recursive matching strategy based on the effective selection of a small number of candidates. However, stereo analysis requires more than a straightforward application of stereo matching. The crucial problem is to produce accurate stereo correspondences in all parts of the image. Especially, errors in occluded regions and homogenous or less structured regions lead to disturbing artifacts in the synthesized virtual views. To cope with this problem, mismatches have to be detected and substituted by a sophisticated interpolation and extrapolation scheme.

This paper details a new modular approach to virtual view creation that is designed to work in conjunction with a proposed scalable teleconferencing configuration. This scalable system is configured using building blocks defined as stereo camera analysis blocks (SCAB). SCABs consist of axe-parallel narrow baseline stereo cameras. These provide design flexibility and improve the image analysis process. Virtual view creation is modular in such that we can add or remove SCABs based on our particular requirements without having to modify the view synthesis algorithm. A new approach to virtual view creation from multiple images is defined. This contains two separate processes: surface segmentation, which identifies surfaces, and surface selection and merging, which selects and integrates the best view of a required surface into the virtual view. The surfaces are identified via a sampling density function while the surface selection procedure is based on a weighting scheme.

Epipolar image analysis is a robust method for 3D scene depth reconstruction that uses all available views of an image sequence simultaneously. It is restricted to horizontal, linear, and equidistant camera movements. In this paper, we present a concept for an extension of epipolar image analysis to other camera configurations like, e.g., circular movements. Instead of searching for straight lines in the epipolar image, we explicitly compute the trajectories of particular points through the image cube. Variation of the unknown depth leads to different curves. From those, the one corresponding to the true depth is selected by evaluating color constancy along the curve. In order to handle occlusions correctly an explicit occlusion compatible ordering scheme is derived for the case of circular movements. To compensate the influence of perspective projection we introduce a depth corrected epipolar image analysis algorithm which we call image cube trajectory analysis (ICT).

In this paper we present an advanced 3D signal processing for an extended architecture of 'Virtual Team User Environments (VIRTUE)' - a proposed concept on immersive 3D videoconferencing. Due to certain reasons the current VIRTUE architecture is restricted to a connection between three single user terminals. To overcome these shortcomings, a flexible multi-user concept is currently under investigation. One key challenge of such a multi-user concept is a robust 3D video processing including tracking, segmentation and depth analysis, which can be applied to a couple of participants sitting in front of the extended VIRTUE terminal. Against this background this paper presents a joint segmentation and disparity estimation approach which allows an accurate and robust 3D analysis providing the required input for an arbitrary synthesis of virtual views at the remote stations.

We introduce a new concept of an immersive 3D video-conference system and immersive tele-collaboration. Based on the principle of a shared virtual table environment (SVTE), the key features of the new system concept are explained and compared to the other SVTE approaches like tele-cubicles. Furthermore we present a complete system architecture based on MPEG-4 technology and discuss first implementations.

This paper introduces newly developed hardware components and configuration aspects for a complex video-processing system based on PCI architectures. Originally designed for an immersive videoconference system, the modular, scalable and cost efficient multi-processor structure can also be applied to other demanding signal processing applications. Announcing the videoconference system, we derive and explain the structure of a PCI subsystem based on four TriMedia processors and an open interface architecture. First application results of the introduced components are presented regarding PCI compatibility, processing power as well as the system architecture for an immersive videoconference system.

In this paper, we present results of a comparative study on disparity analysis of convergent stereo systems. If the epipolar geometry is known, disparity analysis can be performed by computing the disparity along the epipolar line in the original views. Using a parallel stereo algorithm, the convergent views have to be rectified in order to have horizontal and parallel epipolar lines. In this study, both approaches are investigated with respect to the computational effort and the quality of the disparity analysis results. 1

This paper refers to a shape-adaptive DCT algorithm (SA-DCT) originally proposed by Sikora and Makai (see ibid., vol.5, p.59-62, 1995). The SA-DCT has been developed in the framework of the ongoing MPEG-4 standardization phase of ISO/IEC, and has been included in the video verification model of MPEG-4. In this context, the focus of the paper is to emphasize a systematic performance limitation of conventional SA-DCT, and to propose an extended version (/spl Delta/DC-SA-DCT) which avoids this restriction in general. This modification considerably improves the efficiency of SA-DCT, and can easily be implemented in existing SA-DCT tools.

This paper presents an object-based layered video coding scheme which achieves very high compression efficiency along with the provision for advanced content-based functionalities, e.g., content-based scalability or content-based access and manipulation of video data. In a first step, a video sequence is segmented into several arbitrarily shaped "object layers." To achieve the desired content-based functionalities, a baseline shape-adaptive discrete cosine transform (DCT) coding algorithm is introduced which can be seen as an extension of conventional block-based DCT coding schemes (e.g., H.261, H.263, MPEG-1, or MPEG-2) toward coding of arbitrarily shaped image content. In order to increase compression efficiency, the baseline object-based layered approach can be extended with an object-based motion prediction toolbox. Using this toolbox, the coding scheme can potentially select specific prediction techniques for every object layer to be coded. To illustrate the concept, an extension of the baseline shape-adaptive DCT algorithm with a technique for global background motion estimation and compensation is described which significantly improves the compression efficiency of suitable video sequences compared to standard MPEG coding schemes.