The ever-increasing demand for immersive applications has made point cloud an important data type for 3D processing. Tree-based data structures are commonly used for representing point clouds where memory pointers are used to realize the connection among points. The significant cost of data storage and irregular access patterns for processing points make such data structures largely inefficient. In this paper, we examine a point cloud representation using compressed geometric arrays (CGA) that reduces the size of point cloud and limits the amount of memory indirection. Our experimental results on a set of critical point cloud operations indicate 998× speed-up, 410× better bandwidth utilization, and 58% storage reduction for CGA over the state-of-the-art point cloud library (PCL).

With the increasing demand for 3D modeling by the emerging immersive applications, the 3D point cloud has become an essential representation format for processing 3D images and video. Because of the inherent sparsity in 3D data and the significant memory requirements for representing points, point cloud processing is a challenging task. In this paper, we propose a novel data structure for representing point clouds with a reduced memory requirement and a faster lookup than the state-of-the-art formats. The proposed format is examined for temporal encoding in geometric point cloud compression. Our simulation results show that the proposed temporal prediction enhances the compression rate and quality by 13-33% as compared to MPEG G-PCC. Moreover, the proposed data structure provides 16-54 × faster point lookup operations and more than 1.4 × reduction in memory consumption compared to the octree structure used in the MPEG G-PCC.

Immersive video places the user inside the video scene, allowing the user to control the direction of the view. To achieve this, the view of every direction must be recorded using either a panoramic camera or multiple cameras placed at different positions with different angels. The size of the captured video can be quite large due to multiple video streams, one from each camera. Even with compression standards such as Multiview Video Coding (MVC), the transmission of the whole MVC video is still bandwidth-costly, especially for heterogeneous users whose bandwidths vary. In this paper, we present a new approach for immersive video streaming by using Scalable Multiview Video Coding (SMVC) to create multiple layers of the immersive video, supporting heterogeneous receivers more efficiently. Our method limits the number of views in its base layer, while it uses view scalability and free view-point scalability in the additional layers to synthesize more views at the receiver and provide high quality free view-point viewing to the user. Performance evaluations demonstrate that our method: 1-synthesizes missing views more accurately, as evident subjectively, and 2-achieves an average and maximum gain of 0.75 and 1.4 in Bjontegaard BD-Bitrate scale, respectively, compared to existing work which simply group adjacent views in the same layer.

Multi-view/3D video is currently available in games, entertainment, education, security, and surveillance applications . Since the amount of data in multi-view/3D increases proportionally with the number of cameras, and due to different bandwidth and playback capabilities of receivers, appropriate compression of multi-view/3D video to produce the correct bitrate while maintaining smooth video quality is crucial, a task that is mostly performed by the rate control module of the encoder. There are many existing rate control algorithms for single-view and multi-view video coding considering the specific features or aspects of these videos. In this paper, we introduce a novel view-level rate distortion (RD) model. We use a systematic methodology to derive this RD model by investigating the impact of multi-view/3D video characteristics on the bitrate of a compressed video. Our proposed RD model considers the concepts of intra-view and inter-view disparity as an effective feature of multi-view/3D video to estimate the overall bitrate of each view more accurately. Evaluation results indicate that our proposed view-level RD model outperforms existing linear models by a factor of 3 and can predict the rate of each view with relatively high precision and a low estimation error of 12% on average.

In recent years, multi-view/3D video applications, such as three-dimensional television (3DTV) and free-viewpoint television (FTV), have drawn increasing attention. Since the amount of data that has to be stored or transmitted increases proportionally with the number of cameras, efficient compression of multi-view/3D video is crucial. Scalable multi-view video coding is one of the methods to address this challenge. But, in streaming multi-view/3D video over a network to heterogeneous receivers, efficient video compression while maintaining a high quality of received video is very challenging. This paper presents a novel method for rate-distortion optimization in scalable multiview video. We apply the Karush-Kuhn-Tucker (KKT) conditions in minimizing the perceptual distortion of decoded video under the conditions that the sum of bits generated from different views is constrained within a given bit budget. Since the constraint-based optimization problem is usually computational intensive, our proposed approach considers the concept of disparity between layers and disparity between views to reduce this computational complexity. Simulation results indicate that the proposed approach is able to meet network bandwidth limitations with acceptable overall video quality.

Dynamically changing the spatial resolution in a video conferencing session is useful for seamlessly adapting the bitrate to changing network conditions and for improving the user experience. Similar to earlier standards, the emerging High Efficiency Video Coding (H.265/HEVC) standard does not allow prediction across different resolutions, so an Instantaneous Decoding Refresh (IDR) picture must be sent to reinitialize the stream when a resolution change happens. IDR pictures take significantly more bits compared to predictively coded pictures. Thus, using them for resolution switching significantly reduces coding efficiency and increases the delay. In this paper we propose a method to support efficient adaptive resolution change using the emerging scalable H.265/HEVC standard. The proposed approach utilizes the inter-layer predicted random access pictures at the enhancement layer for resolution switching, instead of IDR pictures. The experimental results show that when the proposed method was used, the bitrate was reduced at the switching point by 34% on average for the tested video sequences. In addition, visual examples are shown demonstrating the improved visual quality with the proposed method.

Multi-view 3D video is currently attracting growing attention in several applications such as the 3DTV, free-view point video and entertainment industry where it can be used to provide multi-perspective viewing and 3D scene experiences. In multi-view 3D video, several 3D video sequences should be captured simultaneously from the same scene but through different viewing angles. One of the major challenges in this field is how to transmit the large amount of data of a multi-view 3D video sequence over error prone channels to heterogeneous devices with different bandwidth, resolution, and processing power, while maintaining a high visual quality. Scalable Multi-view 3D Video Coding (SMVC) is one of the methods to address this challenge. But there are many difficulties in SMVC that makes it impractical in most 3D video applications. In this work, we propose an adaptive framework to use SMVC in various 3D video applications effectively. The current prototype shows enhanced capability in handling the existing 3D video applications.

With the growing demand for 3D video, efforts are underway to incorporate it in the next generation of broadcast and streaming applications and standards. 3D video is currently available in games, entertainment, education, security, and surveillance applications. A typical scenario for multiview 3D consists of several 3D video sequences captured simultaneously from the same scene with the help of multiple cameras from different positions and through different angles. Multiview video coding provides a compact representation of these multiple views by exploiting the large amount of inter-view statistical dependencies. One of the major challenges in this field is how to transmit the large amount of data of a multiview sequence over error prone channels to heterogeneous mobile devices with different bandwidth, resolution, and processing/battery power, while maintaining a high visual quality. Scalable Multiview 3D Video Coding (SMVC) is one of the methods to address this challenge; however, the evaluation of the overall visual quality of the resulting scaled-down video requires a new objective perceptual quality measure specifically designed for scalable multiview 3D video. Although several subjective and objective quality assessment methods have been proposed for multiview 3D sequences, no comparable attempt has been made for quality assessment of scalable multiview 3D video. In this article, we propose a new methodology to build suitable objective quality assessment metrics for different scalable modalities in multiview 3D video. Our proposed methodology considers the importance of each layer and its content as a quality of experience factor in the overall quality. Furthermore, in addition to the quality of each layer, the concept of disparity between layers (inter-layer disparity) and disparity between the units of each layer (intra-layer disparity) is considered as an effective feature to evaluate overall perceived quality more accurately. Simulation results indicate that by using this methodology, more efficient objective quality assessment metrics can be introduced for each multiview 3D video scalable modalities.

Transmission of multi-view video content is not practical in most mobile environments due to the limited bandwidth and processing power of mobile devices. To support such environments, one can limit the number of views that are being transmitted, known as Scalable Multi-view Video Coding (SMVC). In this paper, we propose a new view selection method for view scalability in multi-view video coding in mobile environments, which uses inter and intra view dissimilarities to determine the most suitable views for the base layer corresponding to the prediction structure and user selected limited number of views. By selecting more correlated views for the base layer, the proposed method provides an improved performance, as confirmed by simulation results, even when all the enhancement layers are dropped due to network limitations.

Transmission of video over error prone and still bandwidth limited wireless channels demand high compression efficiency and resilience to packet losses and errors. Scalable or layered video coding applied to highly compression efficient codecs is an ideal solution to the problem. However, scalability reduces compression efficiency of the coders. In this paper we show how compression efficiency of two-layer SNR scalable video coders can be retained via joint base-enhancement layer optimization. Simulation results show that joint base-enhancement layer optimization significantly outperforms separate optimization of the layers, and it closely follows the compression performance of the single-layer optimized codec.