The increased compression ratios achieved by the High Efficiency Video Coding (HEVC) standard lead to reduced robustness of coded streams, with increased susceptibility to network errors and consequent video quality degradation. This paper proposes a method based on a two-stage approach to improve the error robustness of HEVC streaming, by reducing temporal error propagation in the case of frame loss. The prediction mismatch that occurs at the decoder after frame loss is reduced through the following two stages. First, at the encoding stage, the reference pictures are dynamically selected based on constraining conditions and Lagrangian optimization, which distributes the use of reference pictures, by reducing the number of prediction units that depend on a single reference. Second, at the streaming stage, a motion vector (MV) prioritization algorithm, based on spatial dependencies, selects an optimal subset of MVs to be transmitted, redundantly, as side information to reduce mismatched MV predictions at the decoder. The simulation results show that the proposed method significantly reduces the effect of temporal error propagation. Compared with the reference HEVC, the proposed reference picture selection method is able to improve the video quality at low-packet-loss rates (e.g., 1%) using the same bitrate, achieving quality gains up to 2.3 dB for 10% of packet loss ratio. It is shown, for instance, that the redundant MVs are able to boost the performance achieving quality gains of 3 dB when compared with the reference HEVC, at the cost using 4% increase in total bitrate.

In this paper a robust encoding scheme is proposed to improve the visual quality of HEVC decoded video when intra frames are lost along the streaming path. For this purpose, the encoding process includes frame loss simulation and subsequent error concealment, to find the most efficient method that should be used by a decoder to recover lost intra frames. In this novel scheme, each image is divided into partitions, which are associated with the error concealment method that achieves the lowest distortion. Then this information is signalled to the decoder through SEI messages in the coded stream. In order to efficiently use the signalling overhead, rate-distortion optimisation is used to achieve the best trade-off between the number of transmitted symbols and distortion of reconstructed frames. Experimental results show the effectiveness of the proposed method to enhance the quality of reconstructed intra frames under different packet loss ratios (PLR). For PLR=10%, the robust coding scheme is able to improve the average PSNR of all frames affected by errors, up to 1.50 dB and 3.44 dB in Low-Delay and Random-Access configurations respectively, at a maximum overhead cost of 0.24%.

Multiview entertainment is the next step in 3D immersive media networking owing to its improved depth perception and free-viewpoint viewing capability whereby users can observe the scene from the desired viewpoint. This paper outlines a delivery system for multiview plus depth video, combining the broadcast and broadband networks. The digital video broadcast (DVB) network is used along with adaptive peer-to-peer (P2P) distribution over the Internet to deliver high-volume multimedia to users. The DVB network has been used to deliver part of the 3D service, owing to its robustness and wide availability, as a mechanism to guarantee the minimum 3D quality of experience. The developed system brings key contributions in the P2P transport for real-time multimedia delivery, including a user preference-aware adaptation mechanism, adaptive redundant chunk scheduling for robustness, incentives to decrease the load on the content server for improved system scalability, and resynchronization capability with the DVB transmission. The introduced features are compared with those of some other well-known P2P solutions to highlight the quantitative gains. A subjective testing campaign has also been organized on the developed hybrid platform, which proves the effectiveness of user-aware adaptation over network-based adaptation on a mean opinion score scale.

Advances in video coding and networking technologies have paved the way for the Multi-View Video (MVV) streaming. However, large amounts of data and dynamic network conditions result in frequent network congestion, which may prevent video packets from being delivered on time. As a consequence, the 3D viewing experience may be degraded significantly, unless quality-aware adaptation methods are deployed. There is no research work to discuss the MVV adaptation of decision strategy or provide a detailed analysis of a dynamic network environment. This work addresses the mentioned issues for MVV streaming over HTTP for emerging multi-view displays. In this research work, the effect of various adaptations of decision strategies are evaluated and, as a result, a new quality-aware adaptation method is designed. The proposed method is benefiting from layer based video coding in such a way that high Quality of Experience (QoE) is maintained in a cost-effective manner. The conducted experimental results on MVV streaming using the proposed strategy are showing that the perceptual 3D video quality, under adverse network conditions, is enhanced significantly as a result of the proposed quality-aware adaptation.

The improved compression efficiency achieved by the High Efficiency Video Coding (HEVC) standard has the counter-effect of decreasing error resilience in transmission over error-prone channels. To increase the error resilience of HEVC streams, this paper proposes a checkerboard reference picture selection method in order to reduce the prediction mismatch at the decoder in case of frame losses. The proposed approach not only allows to reduce the error propagation at the decoder, but also enhances the quality of reconstructed frames by selectively constraining the choice of reference pictures used for temporal prediction. The underlying approach is to increase the amount of accurate temporal information at the decoder when transmission errors occur, to improve the video quality by using an efficient combination of diverse motion fields. The proposed method compensates for the small loss of coding efficiency at frame loss rates as low as 3%. For a single frame-loss event the proposed method can achieve up to 2 dB of gain in the affected frames and an average quality gain of 0.84 dB for different error prone conditions.

This paper addresses the problem caused by motion vector coding dependencies on the error resilience performance of the emergent High Efficiency Video Coding (HEVC) standard. We propose a method based on the prediction dependency of motion vectors (MV) to select the most relevant ones for redundant coding with reduced overhead. The spatial dependencies are analysed in the encoder to prioritise the MVs that should be selected for redundancy, based on the number of subsequent dependent coding units. Then, a subset of prioritised MVs is transmitted as redundancy (referred to as side information in the paper), to reduce the use and propagation of mismatched MV predictions in case of transmission errors or data loss. The simulation results show that the proposed MV selection method can effectively identify the most relevant motion field, achieving improved error robustness with a reduced redundancy overhead. Exploiting only 30% of the generated MVs for redundancy, average quality gains of up to 1 dB are achieved compared to a uniform MV selection scheme, and up to 2 dB compared to the original HEVC standard with no redundant encoded information.

This paper proposes an immersive audio rendering scheme for networked 3D multimedia systems. The spatial audio rendering method based on wave field synthesis is particularly useful for applications where multiple listeners experience a true spatial soundscape while being free to move without losing spatial sound properties. The proposed approach can be considered as a general solution to the static listening restriction imposed by conventional methods, which rely on an accurate sound reproduction within a sweet spot only. The paper reports on the results of numerical analysis and experimental validation using various sound sources. It is demonstrated and confirmed that while covering the majority of the listening area, the developed approach can create a variety of virtual audio objects at target positions with very high accuracy. Subjective evaluation results show that an accurate spatial impression can be achieved with multiple simultaneous audible depth cues improving localization accuracy over single object rendering.

Streaming 3D multi-view video to multiple clients simultaneously remains a highly challenging problem due to the high-volume of data involved and the inherent limitations imposed by the delivery networks. Delivery of multimedia streams over Peer-to-Peer (P2P) networks has gained great interest due to its ability to maximise link utilisation, preventing the transport of multiple copies of the same packet for many users. On the other hand, the quality of experience can still be significantly degraded by dynamic variations caused by congestions, unless content-aware precautionary mechanisms and adaptation methods are deployed. In this paper, a novel, adaptive multi-view video streaming over a P2P system is introduced which addresses the next generation high resolution multi-view users' experiences with autostereoscopic displays. The solution comprises the extraction of low-overhead supplementary metadata at the media encoding server that is distributed through the network and used by clients performing network adaptation. In the proposed concept, pre-selected views are discarded at a times of network congestion and reconstructed with high quality using the metadata and the neighbouring views. The experimental results show that the robustness of P2P multi-view streaming using the proposed adaptation scheme is significantly increased under congestion.

The quality assessment of impaired stereoscopic video is a key element in designing and deploying advanced immersive media distribution platforms. A widely accepted quality metric to measure impairments of stereoscopic video is, however, still to be developed. As a step toward finding a solution to this problem, this paper proposes a full reference stereoscopic video quality metric to measure the perceptual quality of compressed stereoscopic video. A comprehensive set of subjective experiments is performed with 14 different stereoscopic video sequences, which are encoded using both the H.264 and high efficiency video coding compliant video codecs, to develop a subjective test results database of 116 test stimuli. The subjective results are analyzed using statistical techniques to uncover different patterns of subjective scoring for symmetrically and asymmetrically encoded stereoscopic video. The subjective result database is subsequently used for training and validating a simple but effective stereoscopic video quality metric considering heuristics of binocular vision. The proposed metric performs significantly better than state-of-the-art stereoscopic image and video quality metrics in predicting the subjective scores. The proposed metric and the subjective result database will be made publicly available, and it is expected that the proposed metric and the subjective assessments will have important uses in advanced 3D media delivery systems.

Delivery of 3D immersive entertainment to the home remains a highly challenging problem due to the large amount of data involved, and the need to support a wide variety of different displays. Support of such displays may require different numbers of views, delivered over time varying networks. This requires a delivery scheme featuring scalable compression to adapt to varying network conditions, and error resiliency to overcome disturbing losses in the 3D perception. Audio and video attention models can be used in designing an optimal content-aware compression and transmission scheme, by prioritizing the most visually important areas of the video. This paper gives an overview of a content-aware, scalable multi-view audiovisual entertainment delivery framework. Results are shown to evaluate the kinds of error robustness improvements that could be seen using such a system.

The ability to provide a realistic perception of depth is the core added functionality of modern 3D video display systems. At present, there is no standard method to assess the perception of depth in 3D video. Existence of such methods would immensely enhance the progression of 3D video research. This paper focuses on the depth perception assessment in color plus depth representation of 3D video. In this paper, we subjectively evaluate the depth perceived by the users on an auto stereoscopic display, and analyze its variation with the impairments introduced during the compression of the depth images. The variation of the subjective perception of depth is explained based on another evaluation that is carried out to identify the Just Noticeable Difference in Depth (JNDD) perceived by the subjects. The JNDD corresponds to the sensitivity of the observers to the changes in depth in a 3D video scene. Even though only the effects of compression artifacts are considered in this paper, the proposed assessment technique, based on the JNDD values can be used in any future depth perception assessment work.

Free viewpoint video enables the visualisation of a scene from arbitrary viewpoints and directions. However, this flexibility in video rendering provides a challenge in 3D media for achieving spatial synchronicity between the audio and video objects. When the viewpoint is changed, its effect on the perceived audio scene should be considered to avoid mismatches in the perceived positions of audiovisual objects. Spatial audio coding with such flexibility requires decomposing the sound scene into audio objects initially, and then synthesizing the new scene according to the geometric relations between the A/V capturing setup, selected viewpoint and the rendering system. This paper proposes a free viewpoint audio coding framework for 3D media systems utilising multiview cameras and a microphone array. A real-time source separation technique is used for object decomposition followed by spatial audio coding. Binaural, multichannel sound systems and wave field synthesis systems are addressed. Subjective test results shows that the method achieves spatial synchronicity for various viewpoints consistently, which is not possible by conventional recording techniques.

In this paper we propose a novel video object edge adaptive upsampling scheme for application in video-plus-depth and Multi-View plus Depth (MVD) video coding chains with reduced resolution. Proposed scheme is for improving the rate-distortion performance of reduced-resolution depth map coders taking into account the rendering distortion induced in free-viewpoint videos. The inherent loss in fine details due to downsampling, particularly at video object boundaries causes significant visual artefacts in rendered free-viewpoint images. The proposed edge adaptive upsampling filter allows the conservation and better reconstruction of such critical object boundaries. Furthermore, the proposed scheme does not require the edge information to be communicated to the decoder, as the edge information used in the adaptive upsampling is derived from the reconstructed colour video. Test results show that as much as 1.2 dB gain in free-viewpoint video quality can be achieved with the utilization of the proposed method compared to the scheme that uses the linear MPEG re-sampling filter. The proposed approach is suitable for video-plus-depth as well as MVD applications, in which it is critical to satisfy bandwidth constraints while maintaining high free-viewpoint image quality.

We present a novel and efficient multi-view depth map enhancement method proposed as a post-processing of initially estimated depth maps. The proposed method is based on edge, motion and scene depth-range adaptive median filtering and allows for an improved quality of virtual view synthesis. To enforce the spatial, temporal and inter-view coherence in the multi-view depth maps, the median filtering is applied to 4-dimensional windows that consist of the spatially neighboring depth map values taken at different viewpoints and time instants. A fast iterative block segmentation approach is adopted to adaptively shrink these windows in the presence of edges and motion for preservation of sharpness and realistic rendering and for improvement of the compression efficiency. We show that our enhancement method leads to a reduction of the coding bit-rate required for representation of the depth maps and also leads to a gain in the quality of synthesized views at arbitrary virtual viewpoints.

We present a novel multi-view depth map enhancement method deployed as a post-processing of initially estimated depth maps, which are incoherent in the temporal and inter-view dimensions. The proposed method is based on edge and motion-adaptive median filtering and allows for an improved quality of virtual view synthesis. To enforce the spatial, temporal and inter-view coherence in the multiview depth maps, the median filtering is applied to 4-dimensional windows that consist of the spatially neighbor depth map values taken at different viewpoints and time instants. These windows have locally adaptive shapes in a presence of edges or motion to preserve sharpness and realistic rendering. We show that our enhancement method leads to a reduction of a coding bit-rate required for representation of the depth maps and also to a gain in the quality of synthesized views at an arbitrary virtual viewpoint. At the same time, the method carries a low additional computational complexity.

In this letter, a new method is proposed for multiview depth map compression. It is intended to skip some parts of certain depth map viewpoints without encoding and to just predict those skipped parts by exploiting the multiview correspondences and some flags transmitted. It is targeted to save the bit rate allocated for depth map sequences to a great extent. Multiview correspondences are exploited for each skipped depth map frame by making use of the depth map frames belonging to neighboring views and captured at the same time instant. A prediction depth map frame is constructed block by block on a free viewpoint qualitywise selective basis from a couple of candidate predictors generated through the implicit and explicit usage of the 3-D scene geometry. Especially at lower bit rates, dropping higher temporal layers of certain depth map viewpoints and replacing them with corresponding predictors generated using the proposed multiview aided approach save a great amount of bit rate for those depth map viewpoints. At the same time, the perceived quality of the reconstructed stereoscopic videos is maintained, which is proved through a set of subjective tests.

In this paper, a new multi-view coding (MVC) scheme is proposed and evaluated. The scheme offers improved low-delay view random access capability and at the same time comparable compression performance with respect to the reference multi-view coding scheme currently used. The proposed scheme uses the concept of multiple-resolution view coding, exploiting the trade-off between quantization distortion and downsampling distortion at changing bit-rates, which in turn provides improved coding efficiency. Bi-predictive (B) coded views, used in the conventional MVC method, are replaced with predictive coded downscaled views, reducing the view dependency in a multi-view set and hence reducing the random view access delay, but preserving the compression performance at the same time. Results show that the proposed method reduces the view random access delay in a MVC system significantly, but has a similar objective and subjective performance with the conventional MVC method.