Three-dimensional (3-D) displays provide a more realistic experience of entertainment by providing its viewers an added sensation of depth by artificially exploiting light rays to stimulate certain depth cues in the human visual system, especially binocular stereopsis. Due to its close relationship with human visual perception, mass market deployment of 3-D displays will be significantly dependant upon addressing the related perceptual factors such as visual comfort. In order to address the perceptual factors, it is very important to understand how humans experience depth on 3-D displays and how sensitive they are for different depth cues. In this paper, the sensitivity of humans for different depth cues is analyzed as applicable to 3-D viewing on stereoscopic displays. Mathematical models are derived to explain the just noticeable difference in depth (JNDD) for three different depth cues, namely binocular disparity, retinal blur, and relative size. Extensive subjective assessments are performed on a stereoscopic display with passive polarized glasses and on an auto-stereoscopic display to validate the mathematical models for JNDD. It is expected that the proposed models will have important use cases in 3-D display designing as well as 3-D content production.

Delivery of 3D immersive entertainment to the home remains a highly challenging problem due to the large amount of data involved, and the need to support a wide variety of different displays. Support of such displays may require different numbers of views, delivered over time varying networks. This requires a delivery scheme featuring scalable compression to adapt to varying network conditions, and error resiliency to overcome disturbing losses in the 3D perception. Audio and video attention models can be used in designing an optimal content-aware compression and transmission scheme, by prioritizing the most visually important areas of the video. This paper gives an overview of a content-aware, scalable multi-view audiovisual entertainment delivery framework. Results are shown to evaluate the kinds of error robustness improvements that could be seen using such a system.

The ability to provide a realistic perception of depth is the core added functionality of modern 3D video display systems. At present, there is no standard method to assess the perception of depth in 3D video. Existence of such methods would immensely enhance the progression of 3D video research. This paper focuses on the depth perception assessment in color plus depth representation of 3D video. In this paper, we subjectively evaluate the depth perceived by the users on an auto stereoscopic display, and analyze its variation with the impairments introduced during the compression of the depth images. The variation of the subjective perception of depth is explained based on another evaluation that is carried out to identify the Just Noticeable Difference in Depth (JNDD) perceived by the subjects. The JNDD corresponds to the sensitivity of the observers to the changes in depth in a 3D video scene. Even though only the effects of compression artifacts are considered in this paper, the proposed assessment technique, based on the JNDD values can be used in any future depth perception assessment work.

Research on error resilience in multi-view coding is currently receiving considerable interest. While there is a multitude of literature concerning error recovery in 2D video, due to the statistical difference in motion compensation among temporal frames and disparity compensation among view points, such methods are inadequate to cater to the requirements of multiview video transmission. This paper addresses the above issue by transmission of redundant disparity vectors for error recovery purposes. The proposed system, which is implemented using the Joint Scalable Video Model (JSVM) codec and tested using a simulated Internet Protocol (IP) packet network environment, can be used along with a suitable error concealment scheme to provide robust multi-view video transmission. The experimental results suggest that the proposed algorithm experiences a slight degradation of quality in error free environments due to the inclusion of redundant data. However, it improves the reconstructed picture quality significantly in error prone environments, specifically for Packet Loss Rates (PLRs) greater than 7%.

The most distinguishing feature of 3D display systems, compared to their traditional 2D counterparts, is their ability to provide an additional perception of depth to its viewers. Thus, the mechanisms behind human depth perception play a significant role in 3D video systems. While there has been significant amounts of research carried out to understand human depth perception, in the areas of physiology and psychology, its applicability to 3D display systems is seldom spoken. Understanding the mechanisms of depth perception is of utmost importance to the development of 3D video technologies that are heavily based on exploitation of human perception. In this paper, it is explained with the aid of existing physiological and psychological models how humans perceive depth in 3D video displays. Based on these explanations a mathematical model is derived to explain the just noticeable difference in depth (JNDD) as perceived by a viewer, watching 3D video. The derived model is experimentally validated on an auto-stereoscopic display. This model is expected to be useful in both 3D content productions as well as in 3D content processing and compression.

In this paper we propose a novel video object edge adaptive upsampling scheme for application in video-plus-depth and Multi-View plus Depth (MVD) video coding chains with reduced resolution. Proposed scheme is for improving the rate-distortion performance of reduced-resolution depth map coders taking into account the rendering distortion induced in free-viewpoint videos. The inherent loss in fine details due to downsampling, particularly at video object boundaries causes significant visual artefacts in rendered free-viewpoint images. The proposed edge adaptive upsampling filter allows the conservation and better reconstruction of such critical object boundaries. Furthermore, the proposed scheme does not require the edge information to be communicated to the decoder, as the edge information used in the adaptive upsampling is derived from the reconstructed colour video. Test results show that as much as 1.2 dB gain in free-viewpoint video quality can be achieved with the utilization of the proposed method compared to the scheme that uses the linear MPEG re-sampling filter. The proposed approach is suitable for video-plus-depth as well as MVD applications, in which it is critical to satisfy bandwidth constraints while maintaining high free-viewpoint image quality.

We present a novel and efficient multi-view depth map enhancement method proposed as a post-processing of initially estimated depth maps. The proposed method is based on edge, motion and scene depth-range adaptive median filtering and allows for an improved quality of virtual view synthesis. To enforce the spatial, temporal and inter-view coherence in the multi-view depth maps, the median filtering is applied to 4-dimensional windows that consist of the spatially neighboring depth map values taken at different viewpoints and time instants. A fast iterative block segmentation approach is adopted to adaptively shrink these windows in the presence of edges and motion for preservation of sharpness and realistic rendering and for improvement of the compression efficiency. We show that our enhancement method leads to a reduction of the coding bit-rate required for representation of the depth maps and also leads to a gain in the quality of synthesized views at arbitrary virtual viewpoints.

This paper presents a content prioritization technique for perceptually enhanced communication of 3D video data over wireless local area network. The prioritisation is performed on the basis of a novel segmentation technique, which utilizes the depth information of 3D content for separating individual objects of the sequence. These objects are then encoded and transmitted separately under different protection levels depending on their perceived importance for perceptual visual quality. The algorithm performance is demonstrated for object based MPEG-4 video transmission over home network and is compared with a basic prioritisation scheme based on splitting a sequence into even and odd frames and encoding them separately with equivalent prioritization as in the proposed scheme. The results demonstrate significant improvement in subjective quality with the proposed scheme.

In this paper, an Unequal Error Protection (UEP) scheme for the transmission of 3-D (Three - Dimensional) video over WiMAX communication channel is proposed. The colour plus depth map stereoscopic video is coded with backward compatibility using a Scalable Video Coding (SVC) architecture, where users with conventional video decoders/receivers can receive the conventional 2-D (Three - Dimensional) video stream whereas users with SVC decoders/receivers and necessary 3-D video displays may render 3-D video. The proposed error protection scheme is based on the perceptual importance of the coded 3-D video components. The UEP method allocates more protection to the colour video packets than the depth map packets in order to receive good quality 2-D/3-D video. The protection levels are assigned through allocating differentiated transmission power for colour and depth map video packets during transmission. On the fly power allocation is based on the estimated distortion of the colour image sequence. The objective and perceptual quality evaluations show that the proposed UEP scheme improves the quality of 2-D video while achieving pleasing quality for 3-D viewers.

We present a novel multi-view depth map enhancement method deployed as a post-processing of initially estimated depth maps, which are incoherent in the temporal and inter-view dimensions. The proposed method is based on edge and motion-adaptive median filtering and allows for an improved quality of virtual view synthesis. To enforce the spatial, temporal and inter-view coherence in the multiview depth maps, the median filtering is applied to 4-dimensional windows that consist of the spatially neighbor depth map values taken at different viewpoints and time instants. These windows have locally adaptive shapes in a presence of edges or motion to preserve sharpness and realistic rendering. We show that our enhancement method leads to a reduction of a coding bit-rate required for representation of the depth maps and also to a gain in the quality of synthesized views at an arbitrary virtual viewpoint. At the same time, the method carries a low additional computational complexity.

In this letter, a new method is proposed for multiview depth map compression. It is intended to skip some parts of certain depth map viewpoints without encoding and to just predict those skipped parts by exploiting the multiview correspondences and some flags transmitted. It is targeted to save the bit rate allocated for depth map sequences to a great extent. Multiview correspondences are exploited for each skipped depth map frame by making use of the depth map frames belonging to neighboring views and captured at the same time instant. A prediction depth map frame is constructed block by block on a free viewpoint qualitywise selective basis from a couple of candidate predictors generated through the implicit and explicit usage of the 3-D scene geometry. Especially at lower bit rates, dropping higher temporal layers of certain depth map viewpoints and replacing them with corresponding predictors generated using the proposed multiview aided approach save a great amount of bit rate for those depth map viewpoints. At the same time, the perceived quality of the reconstructed stereoscopic videos is maintained, which is proved through a set of subjective tests.

This paper presents a novel channel optimisation scheme that enhances the quality of object based video, transmitted over a fixed bandwidth channel. The optimisation methodology is based on an accurate modelling of video packet distortion at the encoder. Video packets are ranked according to their expected distortion and are then mapped to one of a number of different priority radio bearers. In the proposed scheme the video compression technique uses motion compensated prediction and video frames are split into a number of video packets. The algorithm performance is demonstrated for object based MPEG-4 video transmission over a UMTS/FDD system. The results demonstrate that the performance gain achieved with the proposed scheme can reach 2 dB, compared with the equal error protection scheme for video transmission over a fixed bandwidth channel.

In this paper, an unequal power allocation (UPA) scheme for the transmission of SVC (scalable video coded) packets, over WiMAX communications channels is proposed. Transmission power is an important resource in wireless links, and its dynamic use enhances the successful transmission of video packets. Specifically, the proposed UPA scheme improves the decoded video quality by jointly optimizing the source rate allocated to encode the video and the power allocated to transmit the video packets. Base layer packets are allocated more power compared to enhancement layer packets, ensuring the efficient use of the available power budget. For fast, on the fly allocation of the power, lookup tables have been generated. The results show improvements in objective and subjective video quality.

This proposal presents a new error robust strategy for encoding redundant pictures for the H.264/AVC standard. The method is based on providing motion vectors as redundant data, i.e. providing extra protection to the motion information of the encoded stream. The proposed system is implemented based on the existing redundant coding algorithm of the scalable extension of H.264/AVC. The performance of the algorithm is evaluated using various objective quality measurements under both error free and error prone Internet protocol (IP) packet network environments. The proposed algorithm increases the bandwidth utilization with slight degradation in the primary picture quality for error free conditions, compared to the existing redundant coding method of JSVM (joint scalable video model). Furthermore, the simulation results under packet loss environments show that the proposed algorithm outperforms the existing redundant picture coding of JSVM.

In this paper, a new multi-view coding (MVC) scheme is proposed and evaluated. The scheme offers improved low-delay view random access capability and at the same time comparable compression performance with respect to the reference multi-view coding scheme currently used. The proposed scheme uses the concept of multiple-resolution view coding, exploiting the trade-off between quantization distortion and downsampling distortion at changing bit-rates, which in turn provides improved coding efficiency. Bi-predictive (B) coded views, used in the conventional MVC method, are replaced with predictive coded downscaled views, reducing the view dependency in a multi-view set and hence reducing the random view access delay, but preserving the compression performance at the same time. Results show that the proposed method reduces the view random access delay in a MVC system significantly, but has a similar objective and subjective performance with the conventional MVC method.

Stereoscopic video is one of the simplest forms of multi view video, which can be easily adapted for communication applications. Much current research is based on colour and depth map stereoscopic video, due to its reduced bandwidth requirements and backward compatibility. Existing immersive media research is more focused on application processing than aspects related to transfer of immersive content over communication channels. As video over packet networks is affected by missing frames, caused by packet loss, this paper proposes a frame concealment method for colour and depth map based stereoscopic video. The proposed method exploits the motion correlation of colour and depth map image sequences. The colour motion information is reused for prediction during depth map coding. The redundant motion information is then used to conceal transmission errors at the decoder. The experimental results show that the proposed frame concealment scheme performs better than applying error concealment for colour and depth map video separately in a range of packet error conditions.

Automatic video object segmentation based on spatial-temporal information has been a research topic for many years. Existing approaches can achieve good results in some cases, such as where there is a simple background. However, in the case of cluttered backgrounds or low quality video input, automatic video object segmentation is still a problem without a general solution. A novel approach is introduced in this work, to deal with this problem by using depth information in the algorithm. The proposed approach obtains the initial object masks based on depth map and on motion detection. The object boundaries are obtained by updating object masks using a simultaneous combination of multiple cues, including spatial location, intensity, and edge, within an active contour model. The experimental result shows that this method is effective and has good output, even with cluttered backgrounds. It is also robust when the quality of input depth and video is low.

Scalable shape encoding is an important requirement of highly scalable object-based video coding. In this paper, a new scalable vertex-based shape coding scheme is proposed that uses temporal prediction. During shape coding of a video object, the object shape in the first frame is scalable intra-coded using the method described in M. Hu et al., (2004). For scalable shape coding of subsequent frames, temporal prediction is conducted during the coding of coarser layers. Contour matching in the curvature scale space domain is conducted in order to get higher matching accuracy. Experimental results show that the proposed scalable shape coding scheme can achieve better R-D performance than existing predictive shape coding methods and CAE in MPEG-4.

Scalable shape encoding is one of the important steps to achieving highly scalable object-based video coding. In this paper, a new scalable vertex-based shape intra-coding scheme has been described. To improve the encoding performance, we propose a new vertex selection scheme, which can reduce the number of approximation vertices. We also propose a new vertex encoding method, in which the information on the coarser layers and statistical entropy coding are exploited for high encoding efficiency. Experimental results show that the proposed scheme can provide 25-60% gain over the scalable encoding method in Buhan Jordan et al. (1998). For some sequences, it can achieve 5-10% gain over the conventional non-scalable vertex-based coding method (O'Connell (1997)) in bit rate, at the price of additional complexity.

We present an efficient scalable object contours tracking algorithm and its application for video segmentation. It can track the contour of both rigid and nonrigid objects, even with partial occlusion. For each object, motion information is first used to predict the object contour. Next, texture information along the object contour is used to refine the predicted contour. Finally, active snake is used to snap the detected contour to the real object with high accuracy. The proposed energy terms in active snake model are normalized colour gradient and foreground/ background region properties along the contour. Experimental results illustrate the performance of the proposed tracking algorithm, which is able to track the object boundaries with large motion and partial occlusion.

Optimising delivery of video codecs such as MPEG-4 is vital to ensure that acceptable quality can be offered to 3G network customers. The error prone nature of mobile channels means that the video codec that is employed must be fairly robust. In the past, such research has required alteration to existing standards. Given the imminent implementation of 3G technologies, changing the standards is not a convenient option. This paper presents a technique for altering MPEG-4 encoding parameters to increase the error robustness of the output bitstream. It exploits the fact that frames with a high degree motion are often more sensitive to error than those with low amounts of motion. Using a simple model, the video packet length and number of adaptive intra refresh (AIR) blocks are varied according to the amount of motion in a frame. Simulations using a GPRS channel are presented to confirm the benefits of the proposed scheme.