In the paper we investigated the methods of obtaining high-resolution, high frame-rate virtual views based on low frame-rate cameras for applications in high-performance multiview systems. We demonstrated how to set up synchronization for multiview acquisition systems to record required data and then how to process the data to create virtual views at a higher frame rate, while preserving high resolution of the views. We analyzed various ways to combine time frame interpolation with an alternative side-view synthesis technique which allows us to create a required high frame-rate video of a virtual viewpoint. The results prove that the proposed methods are capable of delivering the expected high-quality, high-resolution and high frame-rate virtual views.

In this paper we present a novel method for depth estimation from stereoscopic 360-degree video. As for now, no such methods have been presented in the literature. First we show mathematical foundations for correspondence matching polar coordinates. Then, the attained formulas are simplified to show relationship to depth estimation in rectified pairs of rectangular images. The proposed method has been implemented in Depth Estimation Reference Software (DERS) developed by Motion Picture Experts Group (MPEG) of International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC). Finally, we present experimental results in form of depth maps estimated for “Dancing 360” stereoscopic omnidirectional sequence.

The demonstration is aimed at presenting a simple free-viewpoint television system that is under development at Poznań University of Technology, Poznań, Poland. Video acquisition using sparsely distributed pairs of video cameras, the depth estimation from video, the virtual video rendering on a server feature the perspective FTV system. The original depth estimation and view synthesis algorithms ensure good quality of the virtual views generated during virtual walks around a scene. The system is developed for broadcasting sport (e.g. judo, karate, volleyball) and cultural (e.g. amateur or professional theater performances) events, as well as for interactive courses and manuals (medical, cosmetics, dancing, technical etc.).

The paper presents the concept of a practical free-viewpoint television system with purely optical depth estimation. The system consists of camera modules that contain pairs or triples of cameras together with the respective microphones. The camera modules can be sparsely located in arbitrary positions around a scene. Each camera module is equivalent to a video camera with a depth sensor and microphones. The hardware requirements, the video and audio processing algorithms and the preliminary experimental results are reported. In particular, for such systems, a compression technique is discussed that is more efficient than the new 3D-HEVC technology. A set of new test sequences obtained with the use of camera pairs are presented.

In the paper, we propose a new method for the virtual view synthesis called as Multiview Synthesis. In our approach, virtual views are synthesized using two neighboring real views, but the disoccluded areas are not inpainted, but filled by the information from the further real views - additional views, traditionally not included in the view synthesis. The whole synthesis is performed with triangles rather than with individual pixels. In the proposed Multiview Synthesis, additional steps of adaptive color correction, blurred edge removal and spatial edge blurring are also included. The experimental results show that both the objective and subjective quality of the synthesized views is significantly higher than the quality of views obtained using the state-of-the-art MPEG reference view synthesis software.

We deal with the processing of multiview video acquired by the use of practical thus relatively simple acquisition systems that have a limited number of cameras located around a scene on independent tripods. The real-camera locations are nearly arbitrary as it would be required in the real-world Free-Viewpoint Television systems. The appropriate test video sequences are also reported. We describe a family of original extensions and adaptations of the multiview video processing algorithms adapted to arbitrary camera positions around a scene. The techniques constitute the video processing chain for Free-Viewpoint Television as they are aimed at estimating the parameters of such a multi-camera system, video correction, depth estimation and virtual view synthesis. Moreover, we demonstrate the need for new compression technology capable of efficient compression of sparse convergent views. The experimental results for processing the proposed test sequences are reported.

A complete encoding solution for efficient intra-based depth map compression is proposed in this paper. The algorithm, denominated predictive depth coding (PDC), was specifically developed to efficiently represent the characteristics of depth maps, mostly composed by smooth areas delimited by sharp edges. At its core, PDC involves a directional intra prediction framework and a straightforward residue coding method, combined with an optimized flexible block partitioning scheme. In order to improve the algorithm in the presence of depth edges that cannot be efficiently predicted by the directional modes, a constrained depth modeling mode, based on explicit edge representation, was developed. For residue coding, a simple and low complexity approach was investigated, using constant and linear residue modeling, depending on the prediction mode. The performance of the proposed intra depth map coding approach was evaluated based on the quality of the synthesized views using the encoded depth maps and original texture views. The experimental tests based on all intra configuration demonstrated the superior rate-distortion performance of PDC, with average bitrate savings of 6%, when compared with the current state-of-the-art intra depth map coding solution present in the 3D extension of a high-efficiency video coding (3D-HEVC) standard. By using view synthesis optimization in both PDC and 3D-HEVC encoders, the average bitrate savings increase to 14.3%. This suggests that the proposed method, without using transform-based residue coding, is an efficient alternative to the current 3D-HEVC algorithm for intra depth map coding.

In the paper, novel methods for quality control in multiview with depth video compression are described. First, a simple steepest-descent-based algorithm for adjusting quantization parameters for texture and corresponding depth to obtain the maximal quality for a given bitrate is proposed. Then, experimental results are shown, basing on which a mathematical model for calculation of quantization parameters for the depth and texture is proposed. The paper also covers subject of coding depth with different resolution than the texture. The work is presented in context of the most contemporary compression methods related to AVC, but could also be easily adapted to upcoming 3D-video coding extensions of HEVC.

Recently, stereoscopic video systems are studied in the context of many new applications. This paper addresses the question how the road traffic surveillance may benefit from stereoscopic video analysis. In particular, we propose a novel method for estimating vehicle dimensions using stereoscopic video analysis. This method is based on active appearance model (AAM), a technique already used, e.g. for face detection. Experimental results are provided in the paper in order to demonstrate satisfactory precision of the measurements.

The paper deals with nonlinear transformations of depth-sample values and their applications to multiview-plus-depth video coding. It is proposed to apply such a transformation prior to depth coding. For this transformation, the experiments demonstrate improved quality of the virtual views that are synthesized with use of the decoded depth. Such experiments were made for two versions of the prospective extensions of the Advanced Video Coding (MPEG-4 part 10, ITU-T Rec. H.264). standard. The experimental results are given for the scenario where multiview video is encoded independently from depth and also for the scenario where multiview video is coded jointly with depth.

We propose a new coding technology for 3D video represented by multiple views and the respective depth maps. The proposed technology is demonstrated as an extension of the recently developed high efficiency video coding (HEVC). One base views are compressed into a standard bitstream (like in HEVC). The remaining views and the depth maps are compressed using new coding tools that mostly rely on view synthesis. In the decoder, those views and the depth maps are derived via synthesis in the 3D space from the decoded baseview and from data corresponding to small disoccluded regions. The shapes and locations of those disoccluded regions can be derived by the decoder without any side information transmitted. To achieve high compression efficiency, we propose several new tools such as depth-based motion prediction, joint high frequency layer coding, consistent depth representation, and nonlinear depth representation. The experiments show high compression efficiency of the proposed technology. The bitrate needed for transmission of two side views with depth maps is mostly less than 50% than that of the bitrate for a single-view video.

In this paper, we present the most efficient coding tools that are used in a new video compression technology for multiple views with the depth maps. This very well performing technology was designed and developed in response to MPEG Call for Proposals on 3D Video Coding Technology. The proposed technology exploits a tool that reduces the side views and the side depth maps to small disoccluded regions. This way only one central view and one depth map are coded in the HEVC syntax while the remaining views and the depth maps are synthesized in the decoder from the small disoccluded regions and from the central-view data. Therefore, the bitrate needed for a side view is mostly below 20% of the bitrate for single-view video.

The paper deals with multiview video coding using the new technology of High Efficiency Video Coding (HEVC). Implementation of multiview video coding in the framework of HEVC is described together with new specific tools proposed by the authors. Extensive experimental results are reported for compression performance comparison of MVC (ISO 14496-10), HEVC simulcast and two versions of proposed “multiview HEVC”. For “multiview HEVC” the results indicate significant bitrate reduction of about 50%, as compared to the state-of-the-art MVC technology standardized as a part of AVC (MPEG-4, H.264).

During the last two decades, a new technology generation of video compression was introduced about each 9 years. Each new compression-technology generation provides halving of necessary bitrates as compared to the last previous generation. This increasing single-view compression performance is related to increasing compression performance of multiview video coding. For multiview video with associated depth maps, additional significant bitrate reduction may be achieved. The paper reports the original compression technology that was designed and developed at Poznań University of Technology in response to MPEG Call for Proposals on 3D Video Coding Technology. The main idea of this technique is to predict very efficiently the side views and the depth maps from the base view.

In this paper we propose a new technique for refinement of depth maps. The technique exploits a stereoscopic pair of images and full-pixel precision disparity map in order to produce the output disparity map with sub-pixel precision. The technique employs view synthesis for verification of hypotheses on disparity values, which are formed to iteratively improve the disparity map. Results of experiments, presented for still images and video sequences, show that proposed technique increases quality of depth and has low computational cost. Applications of the proposed technique can be found in future real-time 3D video broadcasting systems which require fast and robust disparity estimation algorithm with high precision.

In this paper we propose a novel approach to error concealment that can be applied to MVC and other 3D video coding technologies. The image content, that is lost due to errors, is recovered with use of multiple error-concealment techniques. In our work we have used three techniques: well-known temporal- and intra-based techniques and a novel inter-view technique. Proposed inter-view recovery employs Depth Image Based Rendering (DIBR), which requires neighboring views and corresponding depth maps. Those depth maps can be delivered in the bit-stream or estimated in the receiver. In order to obtain the final reconstruction, the best technique is selected locally. For that, an original recovery quality measurement method, based on cross-checking, has been proposed. The idea has been implemented and assessed experimentally, with use of 3D video test sequences. The objective and subjective results show that the proposed approach provide good quality of reconstructed video.

Stereo matching techniques usually match segments or blocks of pixels. This paper proposes to match segments defined as fuzzy sets of pixels. The proposed matching method is applicable to various techniques of stereo matching as well as to different measures of differences between pixels. In the paper, embedment of this approach into the state-of-the-art depth estimation software is described. Obtained experimental results show that the proposed way of stereo matching increases reliability of various depth estimation techniques.