Recently, we have introduced a framework for the compression and interactive streaming of image-based scene representations based on hybrid video coding techniques. In that scheme, the storage rate (R), distortion (D), transmission rate (T), and decoding complexity (C) are traded off against each other to minimize the user perceived delay in a remote navigation scenario. In this work, we extend that scheme by introducing a technique for progressive transmission and rendering from such RDTC compressed image-based scene representations. Real-time experiments show that a trade off between the user perceived delay and the rendering quality can be achieved and that the delay can be significantly reduced compared to independent or rate-distortion optimized compression.

This paper studies the calibration of a multi-sensor capture device for the acquisition of image-based scene representations. It is composed of cameras and a laser rangefinder. We use a plane based approach to calibrate the compound system, determining both intrinsic and extrinsic parameters. The novelty of our approach is that the cameras and the scanner are treated jointly. We have tested our method both in simulations and on real data and observe very good calibration results.

Rendering of virtual views in interactive streaming of compressed image-based scene representations requires random access to arbitrary parts of the reference image data. The degree of interframe dependencies exploited during encoding has an impact on the transmission and decoding time and, at the same time, delimits the (storage) rate-distortion (RD) tradeoff that can be achieved. In this work, we extend the classical RD optimization approach using hybrid video coding concepts to a tradeoff between the storage rate (R), distortion (D), transmission data rate (T), and decoding complexity (C). We present a theoretical model for this RDTC space with a focus on the decoding complexity and, in addition, the impact of client side caching on the RDTC measures is considered and evaluated. Experimental results qualitatively match those predicted by our theoretical models and show that an adaptation of the encoding process to scenario specific parameters like computational power of the receiver and channel throughput can significantly reduce the user-perceived delay or required storage for RDTC optimized streams compared to RD optimized or independently encoded scene representations.

Interactive streaming of compressed image-based scene representations requires random access to the reference image data. The degree of interframe dependencies exploited during encoding has an impact on the transmission and decoding time and, at the same time, delimits the (storage) rate-distortion (RD) tradeoff that can be achieved. The transmission data rate and the decoding complexity at the client have received attention in the literature, but their incorporation into the optimization procedure for compression and streaming is missing. If scenario-specific measures are considered, the traditional RD optimization can be extended to a tradeoff between the (storage) rate (R), distortion (D), transmission data rate (T), and decoding complexity (C). In the first part of this sequel of papers, we have theoretically analyzed the RDTC space for the compression of densely sampled image-based scene representations. In this second part, we consider practical RDTC optimization. We propose a modeling and encoding parameter selection procedure that allows us to adapt the compression to scenario-specific properties. The impact of client side caching is considered and evaluated using an experimental testbed. Our results show a significant reduction of the user perceived delay, memory consumption or required minimum channel and storage bitrate for RDTC optimized streams compared to classical RD optimized or independently encoded scene representations.

Interactive navigation in image-based scenes requires random access to the compressed reference image data. When using state of the art block-based hybrid video coding techniques, the degree of inter and intra block dependencies introduced during compression has an impact on the effort required to access reference image data and therefore delimits the response time for interactive applications. In this work a theoretical model for the decoding-complexity of compressed image-based scene representations is presented and evaluated. Results show the validity of the model. Additionally, results for decoding-complexity constrained rate distortion optimization (RDC) using our model show the benefit of incorporating the computational power of client devices into the compression process.

Remote navigation in image-based scene representations requires random access to parts of the compressed reference image data to compose virtual views. The degree of dependencies introduced during compression has an impact on the effort that is required to access reference image data and at the same time delimits the rate-distortion (RD) tradeoff that can be achieved. If a limited channel bitrate and computational power of client devices are taken into account, encoding can be performed in a RD optimal manner with respect to the expected maximum transmission data rate (T) and decoding complexity (C). In this work we present a practical framework for parameter estimation for RDTC optimal encoding of image-based scene representations.