The emerging technologies of Virtual Reality (VR) and 360$^\circ $video introduce new challenges for state-of-the-art video communication systems. Enormous data volume and spatial user navigation are unique characteristics of 360$^\circ$videos that necessitate a space-time effective allocation of the available network streaming bandwidth over the 360$^\circ$video content to maximize the Quality of Experience (QoE) delivered to the user. Towards this objective, we investigate a framework for viewport-driven rate-distortion optimized 360$^\circ$video streaming that integrates the user view navigation patterns and the spatiotemporal rate-distortion characteristics of the 360$^\circ$video content to maximize the delivered user viewport video quality, for the given network/system resources. The framework comprises a methodology for assigning dynamic navigation likelihoods over the 360$^\circ$video spatiotemporal panorama, induced by the user navigation patterns, an analysis and characterization of the 360$^\circ$video panorama's spatiotemporal rate-distortion characteristics that leverage preprocessed spatial tilling of the content, and an optimization problem formulation and solution that capture and aim to maximize the delivered expected viewport video quality, given a user's navigation patterns, the 360$^\circ$video encoding/streaming decisions, and the available system/network resources. We formulate a Markov model to capture the navigation patterns of a user over the 360$^\circ$video panorama and simultaneously extend our actual navigation datasets by synthesizing additional realistic navigation data. Moreover, we investigate the impact of using two different tile sizes for equirectangular tiling of the 360$^\circ$video panorama. Our experimental results demonstrate the advantages of our framework over the conventional approach of streaming a monolithic uniformly-encoded 360$^\circ$video and a state-of-the-art navigation-speed based reference method. Considerable average and instantaneous viewport video quality gains of up to 5 dB are demonstrated in the case of five popular 4 K 360$^\circ$videos. In addition, we explore the impact of two different popular 360$^\circ$video quality metrics applied to evaluate the streaming performance of our system framework and the two reference methods. Finally, we demonstrate that by exploiting the unequal rate-distortion characteristics of the different spatial sectors of the 360$^\circ$video panorama, we can enable spatially more uniform and temporally higher 360$^\circ$video viewport quality delivered to the user, relative to monolithic streaming.

Heterogeneous spatial quality in 360-degree videos is characterized by regions of high and low quality appearing in the same video chunk. Encoding quality-varying omnidirectional videos enables bandwidth waste optimization without compromising the quality inside the client's viewport. Heterogeneous spatial quality has been implemented using either the concept of tiling or Facebook's offset projection. In this paper, we study the preparation of heterogeneous 360-degree videos with three main contributions. We propose two novel approaches that prepare heterogeneous quality versions of a 360-degree video based on two video decomposition techniques: the Gaussian pyramid and the Laplace pyramid. We also introduce a spherical filter for omnidirectional videos which allows to further reduce the video bit-rate by exploring the spherical properties of the 360-degree videos. We compare the performance of the proposed three methods to tiling and the offset projection in representative scenarios of heterogeneous spatial quality in 360-degree videos and highlight the key trade-off between bit-rate and video quality to consider when implementing these approaches.

In this article, we propose video delivery schemes insuring around 1s delivery latency with Dynamic Adaptive Streaming over HTTP (DASH), which is a standard version of HTTP Live Streaming (HLS), so as to benefit from the video representation switching between successive video segments. We also propose HTTP/2-based algorithms to apply video frame discarding policies inside a video segment when a selected DASH representation does not match with the available network resources. The current solutions with small buffer suffer from rebuffering events. Rebuffering not only impacts the Quality of Experience (QoE) but also increases the delivery delay between the displayed and the original video streams. In this work, we completely eliminate rebuffering events by developing optimal and practical video frame discarding algorithms to meet the 1s latency constraint. In all our algorithms, we request the video frames individually through HTTP/2 multiple streams, and we selectively drop the least meaningful video frames thanks to HTTP/2 stream resetting feature. Our simulations show that the proposed algorithms eliminate rebuffering while insuring an acceptable video quality with at least a Peak Signal to Noise Ratio (PSNR) of 35dB compared to 25dB of the basic First In First Out (FIFO) algorithm. We also quantify and qualify the resulting temporal distortion of the video segments per algorithm. An important number of missing video frames results in a temporal fluidity break known as video jitter . The displayed video looks like a series of snapshots. We show that both the optimal Integer Linear Program (ILP) and practical algorithms decrease the frequency and duration of the jitters. For example, practical algorithms reduce the number of crashed displayed videos (presenting one jitter longer than 1,350ms) with 22% compared to the basic FIFO algorithm. We also show that requesting video frames separately with HTTP/2 slightly increases the overhead from 4.34% to 5.76%.

The growing popularity of virtual and augmented reality communications and 360° video streaming is moving video communication systems into much more dynamic and resource-limited operating settings. The enormous data volume of 360° videos requires an efficient use of network bandwidth to maintain the desired quality of experience for the end user. To this end, we propose a framework for viewport-driven rate-distortion optimized 360° video streaming that integrates the user view navigation pattern and the spatiotemporal rate-distortion characteristics of the 360° video content to maximize the delivered user quality of experience for the given network/system resources. The framework comprises a methodology for constructing dynamic heat maps that capture the likelihood of navigating different spatial segments of a 360° video over time by the user, an analysis and characterization of its spatiotemporal rate-distortion characteristics that leverage preprocessed spatial tilling of the 360° view sphere, and an optimization problem formulation that characterizes the delivered user quality of experience given the user navigation patterns, 360° video encoding decisions, and the available system/network resources. Our experimental results demonstrate the advantages of our framework over the conventional approach of streaming a monolithic uniformly-encoded 360° video and a state-of-the-art reference method. Considerable video quality gains of 4 - 5 dB are demonstrated in the case of two popular 4K 360° videos.

A video with heterogeneous spatial quality is a video where some regions of the frame have a different quality than other regions (for instance, a better quality could mean more pixels and less encoding distortion). Such a quality-variable encoding is a key enabler of Virtual Reality application, with 360-degree videos. So far, the main technique that has been proposed to prepare spatially heterogeneous quality is based on the concept of tiling. More recently, Facebook has implemented another approach: the offset projection where more emphasis is put on a specific direction of the frame. In this paper, we study quality-variable 360-degree videos with two main contributions. First, we provide the theoretical analysis of the offset projection and show the impact of the parameter settings on the video quality. Second, we propose another approach which consists in preparing the 360-degree video from a Gaussian pyramid of downscaled and blurred versions of the video. We perform an evaluation of tiling, offset and Gaussian-based approaches in representative scenarios of heterogeneous spatial quality in 360-degree videos and highlight the main trade-off to consider when implementing these approaches.

Full immersion inside a Virtual Reality (VR) scene requires six Degrees of Freedom (6DoF) applications where the user is allowed to perform translational and rotational movements within the virtual space. The implementation of 6DoF applications is however still an open question. In this paper we study a multi-viewpoint (MVP) 360-degree video streaming system, where a scene is simultaneously captured by multiple omnidirectional video cameras. The user can only switch positions to predefined viewpoints (VPs). We focus on the new challenges that are introduced by adaptive MVP 360-degree video streaming. We introduce several options for video encoding with existing technologies, such as High Efficiency Video Coding (HEVC) and for the implementation of VP switching. We model three video-segment download strategies for an adaptive streaming client into Mixed Integer Linear Programming (MILP) problems: an omniscient download scheduler; one where the client proactively downloads all VPs to guarantee fast VP switch; one where the client reacts to the user's navigation pattern. We recorded a one MVP 360-degree video with three VPs, implemented a mobile MVP 360-degree video player, and recorded the viewing patterns of multiple users navigating the content. We solved the adaptive streaming optimization problems on this video considering the collected navigation traces. The results emphasize the gains obtained by using tiles in terms of objective quality of the delivered content. They also emphasize the importance of performing further study on VP switching prediction to reduce the bandwidth consumption and to measure the impact of VP switching delay on the subjective Quality of Experience (QoE).

The delivery and display of 360-degree videos on Head-Mounted Displays (HMDs) presents many technical challenges. 360-degree videos are ultra high resolution spherical videos, which contain an omnidirectional view of the scene. However only a portion of this scene is displayed on the HMD. Moreover, HMD need to respond in 10 ms to head movements, which prevents the server to send only the displayed video part based on client feedback. To reduce the bandwidth waste, while still providing an immersive experience, a viewport-adaptive 360-degree video streaming system is proposed. The server prepares multiple video representations, which differ not only by their bit-rate, but also by the qualities of different scene regions. The client chooses a representation for the next segment such that its bit-rate fits the available throughput and a full quality region matches its viewing. We investigate the impact of various spherical-to-plane projections and quality arrangements on the video quality displayed to the user, showing that the cube map layout offers the best quality for the given bit-rate budget. An evaluation with a dataset of users navigating 360-degree videos demonstrates that segments need to be short enough to enable frequent view switches.

With the decreasing price of Head-Mounted Displays (HMDs), 360-degree videos are becoming popular. The streaming of such videos through the Internet with state of the art streaming architectures requires, to provide high immersion feeling, much more bandwidth than the median user's access bandwidth. To decrease the need for bandwidth consumption while providing high immersion to users, scientists and specialists proposed to prepare and encode 360-degree videos into quality-variable video versions and to implement viewport-adaptive streaming. Quality-variable versions are different versions of the same video with non-uniformly spread quality: there exists some so-called Quality Emphasized Regions (QERs). With viewport-adaptive streaming the client, based on head movement prediction, downloads the video version with the high quality region closer to where the user will watch. In this paper we propose a generic theoretical model to find out the optimal set of quality-variable video versions based on traces of head positions of users watching a 360-degree video. We propose extensions to adapt the model to popular quality-variable version implementations such as tiling and offset projection. We then solve a simplified version of the model with two quality levels and restricted shapes for the QER. With this simplified model, we show that an optimal set of four quality-variable video versions prepared by a streaming server, together with a perfect head movement prediction, allow for 45% bandwidth savings to display video with the same average quality as state of the art solutions or allows an increase of 102% of the displayed quality for the same bandwidth budget.

While Virtual Reality applications are increasingly attracting the attention of developers and business analysts, the behaviour of users watching 360-degree (i.e. omnidirectional) videos has not been thoroughly studied yet. This paper introduces a dataset of head movements of users watching 360-degree videos on a Head-Mounted Display (HMD). The dataset includes data collected from 59 users watching five 70 s-long 360-degree videos on the Razer OSVR HDK2 HMD. The selected videos span a wide range of 360-degree content for which different viewer's involvement, thus navigation patterns, could be expected. We describe the open-source software developed to produce the dataset and present the test material and viewing conditions considered during the data acquisition. Finally, we show some examples of statistics that can be extracted from the collected data, for a content-dependent analysis of users' navigation patterns. The source code of the software used to collect the data has been made publicly available, together with the entire dataset, to enable the community to extend the dataset.

Transport protocols that can exploit multiple paths, especially MPTCP, do not match the requirements of video streaming: high average transmission delay, too strict reliability, and frequent head-of-line phenomenons resulting in abrupt throughput drops. In this paper, we address this mismatch by introducing a cross-layer scheduler, which leverages information from both application and transport layers to re-order the transmission of data and prioritize the most significant parts of the video. Our objective is to maximize the amount of video data that is received in time at the client. We show that current technologies enable the implementation of this cross-layer scheduler without much overhead. We then demonstrate the validity of our approach by studying the performance of an optimal cross-layer scheduler. The gap between the performance of the traditional scheduler versus the optimal scheduler justifies our motivation to implement a cross-layer scheduler in practice. We propose one implementation with basic cross-layer awareness. To evaluate the performance of our proposal, we aggregate a dataset of real MPTCP sessions and we use video stream encoded with HEVC. Our results show that our cross-layer proposal outperforms the traditional scheduler. Viewers not only benefit from the inherent advantages of using MPTCP (such as a better resilience to path failure) but also get a better QoE compared to the traditional scheduler.