We present MosaicMVS, a novel learning-based depth estimation framework for a mosaic-based omnidirectional multi-view stereo (MVS) camera setup. It uses a regular field of view (FOV) MVS network for an omnidirectional imaging setup with explicit consideration of hypothetical voxel-wise FOV overlaps. The resulting depth predictions are accurate and agree on the omnidirectional multi-view geometry. Unlike existing MVS setups, MosaicMVS camera setup can be easily applied to omnidirectional indoor scenes without having to account for constraints such as intricate epipolar constraints and the distortion of omnidirectional cameras. We validate the effectiveness of our framework on a new challenging indoor dataset in terms of depth estimation, reconstruction, and view synthesis. We also present new evaluation metric to check reconstruction performance using post-processed masks for accurate evaluation without any ground truth depth map or laser-scanned reconstructions. Experimental results show that our framework outperforms the state-of-the-art MVS methods in a large margin in all test scenes.

In the field of video depth estimation, significant strides have been made with deep learning-based multi-view stereo approaches. However, existing studies struggle to produce consistently accurate depth maps that account for both multi-view geometry and temporal consistency from monocular video contents. To overcome this limitation, we introduce CMVDE, an innovative video depth estimation framework that leverages a multi-view geometric-temporal coupling approach in an end-to-end manner. Our proposed geometric consistency module efficiently generates multi-view geometric features by employing mutual cross-view epipolar attention between adjacent video frames. Additionally, it compresses these features using the novel multi-scale feature compressor, producing an effective input tensor for the subsequent module. Moreover, our framework enhances temporal consistency across consecutive video frames with the temporal consistency module based on convolutional LSTM 1 leveraging previous depth information as geometric guidance. Compared to state-of-the-art models, our approach achieves superior performance in depth quality and consecutive consistency on the ScanNet 2 and 7-Scenes 3 datasets, surpassing previous multi-view video depth estimation methods.

This paper presents a novel structured camera pose estimation framework for mosaic-based omnidirectional imaging, i.e., producing a wide field of view (FoV) image that covers an entire sphere of the surroundings from a set of regular FoV images. With the effective utilization of geometric priors, the proposed framework exploits an individual image's connected structure while sequentially extracting correspondence between them. In the proposed framework, 2DSfM, a structure from motion method for multi-view images in structured 2D grids, is also proposed. Additionally, we propose a constraint term for rotation vectors in the bundle adjustment process that efficiently incorporates structural priors. We demonstrate our framework on structured omnidirectional image scenes and compare to existing frameworks. The experimental results show that our framework outperforms well-known conventional frameworks regarding both average reprojection error and reconstruction results.

In this paper, we introduce our experience on the development of a three-dimensional audio-visual (3D AV) service system based on the terrestrial digital multimedia broadcasting (T-DMB) system. 3D AV service is now much more feasible than before with the fast advancement of hardware technologies, especially 3D flat panel display, processors and memory. 3D AV service over DMB system is very attractive due to the facts that (1) glassless 3D viewing with small display is relatively easy to implement and more suitable to the single user environment like DMB, (2) DMB is a new media and thus has more flexibility in adding new services on the existing ones, (3) 3D AV handling capability of 3D DMB terminal has lots of potential to generate new types of services if it is added with other components like built-in stereo camera. In order to provide successful 3D DMB services over existing DMB system, we need to solve several issues like (1) guaranteeing backward compatibility with the T-DMB system, (2) minimizing the overhead on the transmitted bit-rate and the required processing power of the terminal, (3) providing good 3D depth perception without a noticeable eye strain. We propose a very efficient and backward compatible system architecture for the 3D DMB, and show how we can get better depth perception with the limited bit budget of the DMB system