Recent autostereoscopic 3-D (A3D) displays suffer from many limitations such as narrow viewing angle, low resolution, and shallow depth effects. As these limitations mainly originate from the insufficiency of pixel resources, it is not easy to obtain a feasible solution that can solve all the limitations simultaneously. In many cases, it will be better to find a good compromising design. Generally, the multiview display and the integral imaging display are the representative designs of A3D. However, as they are too canonical and lack flexibility in design, they tend to be a tradeoff. To address these design issues, we have analyzed the multiview display and the integral image display in a light-field coordinate and developed a 3-D display design framework in a light-field space. The developed framework does not use the “view” concept anymore. Instead, it considers the spatial distribution of rays of the 3-D display and provides more flexible and sophisticated design methods. In this paper, the developed design method is explained using a new pixel value assigning algorithm, called the light-field rendering, and vision-based parameter calibration methods for 3-D displays. We have also analyzed the blur effects caused by the depth and display characteristics. By implementing the proposed method, we have designed a 65-in 96-view display with a 4K panel. The developed prototype has showed almost seamless parallax with a high-resolution comparable to the conventional four to five views displays. This paper will be useful to readers interested in A3D displays, especially in the multiview and the integral imaging displays.

Face recognition under viewpoint and illumination changes is a difficult problem, so many researchers have tried to solve this problem by producing the pose- and illumination- invariant feature. Zhu et al. [26] changed all arbitrary pose and illumination images to the frontal view image to use for the invariant feature. In this scheme, preserving identity while rotating pose image is a crucial issue. This paper proposes a new deep architecture based on a novel type of multitask learning, which can achieve superior performance in rotating to a target-pose face image from an arbitrary pose and illumination image while preserving identity. The target pose can be controlled by the user's intention. This novel type of multi-task model significantly improves identity preservation over the single task model. By using all the synthesized controlled pose images, called Controlled Pose Image (CPI), for the pose-illumination-invariant feature and voting among the multiple face recognition results, we clearly outperform the state-of-the-art algorithms by more than 4~6% on the MultiPIE dataset.

This paper presents a novel method which combines global shape parameters and scalable local descriptors for accurate body parts recognition from a single depth image in real-time. Human poses are of extremely large variation in aspects of visual shapes, because human can take poses from daily activities to gymnastic actions. In order to cover wide-range of the human poses, the proposed algorithm employs a unified structure which combines pose clustering and body parts classification. We name the proposed method Randomized Decision Bush (RDB). Specifically, global shape parameters which can discriminate coarse level shapes are utilized for pose clustering while scalable local shape descriptors are employed for accurate classification. RDB splits the various human poses into multiple clusters which contain similar shapes of the poses. As a result, it provides robust clustering which enables fine level classification within the cluster. The experimental results show improvements on recognizing body parts due to the pose clustering and classification with scalable local descriptors. Additionally, we significantly reduce the complexity of training a large number of human shapes.

The depth information in multiview 3D video sequences are mainly utilized to synthesize new video sequences at virtual view points. Decoded depth maps, however, have typical coding artifacts such as blocking noise according to the compression ratio. Furthermore, depth maps are even resized to reduce the size of compressed bitstreams. Defects in depth maps degrade the quality of synthesized view. This paper proposes a depth resampling method as post processing for encoding/decoding of mixed resolution multiview 3D videos. Different filtering approaches are applied according to the type of depth map. Gray scale dilation filter is applied to graphics depth data to directly remove artifact in synthesized view, and segmentation based filter is applied to estimated or measured depth data to enhance edge and to reduce the compression noise. Experimental result using MPEG reference software shows that the BD bitrate gain in terms of synthesized PSNR is -5.69% for 7 test sequences.

In H.264/AVC, motion vector prediction is only required to reduce temporal redundancy. In order to support inter-view motion vector prediction in multi-view video coding, the prediction process in H.264/AVC is extended in inter-view direction. In general, the temporal and inter-view motion vector prediction uses motion vectors of neighboring blocks around a current block. However, current 3D video coding deals with a multi-view video plus depth (MVD) format, so motion vectors in texture images can be predicted from the corresponding depth images. In this paper, we introduce an improved depth-based motion vector prediction method for the efficient 3D video coding. Simulation results show that the proposed method obtains higher coding efficiency than the conventional method.

This paper introduces a new wide-gamut RGB primary set and a new color-encoding method that are developed for ultrahigh-definition television (UHDTV). The extended gamut RGB-primary set compared with the conventional CRT-centric gamut is proposed in favor of inclusion of real-world surface colors and in consideration of the current and prospective colorimetric characteristics of flat panel displays using which UHDTV programs will likely be reproduced. The new color encoding method is derived to enable luminance and chrominance information to be accurately separated, resulting in orthogonal video signals. The new luma and color-difference signals generated by the newly derived encoding scheme are demonstrated to have insignificant crosstalk. It is also shown that this fact can contribute to improving image quality after subsampling color-difference components and to increasing compression efficiency.

This paper presents a bidirectional motion estimation (ME) method based on tracking feature trajectories and compensating for occlusion to enhance the temporal resolution of an input video sequence. First, we extract features and estimate their trajectories in the forward direction. We continue to track features only if the reliability of their trajectory is sufficiently high. Accordingly, if a feature can be continuously tracked through multiple frames, the proposed method assumes that its trajectory is a true motion vector (MV). Then, these forward feature trajectories are used as reference motion directions for backward block-based ME. If a block does not include any feature trajectories, we use the MVs of neighboring blocks as candidate MVs to propagate the predetermined true MV and to preserve the spatial correlation of MVs. Furthermore, the proposed method can detect occluded regions, where a continuously tracked feature does not have a corresponding point in the current frame. If an occluded region is detected, the intermediate frame is generated from either the previous frame or the current frame. In the non-occluded region, we generate the intermediate frames by taking into consideration the neighboring MVs to reduce blocking artifacts. Experimental results showed that the proposed temporal frame interpolation (TFI) method can improve the visual quality compared with conventional TFI methods, both objectively and subjectively.

3D video services can be supported by a multi-view video plus depth (MVD) format. The MVD format is a promising approach for realizing a natural 3D scene efficiently with a small amount of data, because it can generate virtual views for arbitrary view-points at receiver sides. Many researchers have investigated relation between a texture image and its corresponding depth map in the MVD format. In this paper, we analyze their mode correlation and then propose a fast mode decision algorithm for depth image coding based on the analysis. Experimental results indicate that the proposed method drastically reduces its encoding time with negligible quality losses in both the reconstructed depth maps and the rendered synthetic texture images.

In this paper, we present a novel depth map enhancement scheme in depth map coding. For this purpose, we develop a coupled shock filter that enhances the coded depth map by the edge sharpening and the noise smoothing. Unlike the previous works that control the edge enhancement by the Gaussian blur or anisotropic diffusion, our approach is different in that the edge sharpening is performed. More specifically, in the first phase, we develop the inflection point in such a way that a pixel belongs to the influence zone of a maximum or a minimum made on the basis of the Laplacian. In the second phase, we suppress the amplified noise while preserving the edge. Two-phase structures allows for the strong slope restoration. Finally, the depth enhancement that performs the new approach is implemented. This results in 6.67% (rendered view) BD-bitrate reduction on average when compared with 3DV AVC-based test model and offers the superior subjective quality in the rendered view.

Depth video coding is one of the key techniques to realize the three-dimensional (3D) video system. The depth video is spatially high correlated compared to its corresponding color video in general. In this paper, we propose a high priority intra coding method for depth video coding to efficiently reduce the spatial redundancy. The proposed intra mode is designed based on the intra 16×16 mode. It exploits the derived prediction direction from the adjacent neighboring pixels and does not encode any residual data. The proposed intra mode is allowed I slice as well as P and B slices. It is newly defined for I slice as skip mode and replaced with the skip or direct modes for P and B slices. By the experiments on JMVC 8.2, we confirm that the proposed high priority intra mode reduces about 14.63% bit rate compared to JMVC 8.2 at the same rendering quality.

This paper presents an efficient intra prediction scheme for depth map coding in multi-view plus depth (MVD) system. As compared to the conventional intra prediction algorithm, the proposed method segments the current block into k regions, and applies different prediction scheme for each segmented region, which results in higher prediction accuracy by avoiding wrong estimation across different regions. Moreover, we also provide the efficient lossless coding scheme for region segmentation information based on unique depth characteristics, by which the proposed scheme is competitive w.r.t. R-D cost. The simulation results show the superior performance of the proposed scheme comparing to the H.264/AVC intra prediction and other edge-based intra scheme for depth map.

In general, a depth image in a multi-view video plus depth (MVD) format is employed as additional data to synthesize a virtual texture view. In this paper, the proposed method generates a virtual depth view through depth view synthesis and then adaptively uses the synthesized virtual depth view during encoding, according to an evaluation on effect of the synthesized depth view on the multi-view rendering quality. Experimental results demonstrate that the proposed method significantly reduces a depth image coding bit rate without degrading the rendering quality.

Multi-view video plus depth format, which includes texture and depth images, has been recently introduced as a video representation to support depth perception of scenes and efficient view generation at arbitrary positions. Especially, a depth image has been one of the significantly important issues for successful services of 3-D video applications. In this paper, we introduce a fast and efficient multi-view depth image coding method using the texture images. The proposed method determines to skip some blocks of the depth image at the early stage without a normal encoding process including rate-distortion optimization, based on temporal and inter-view correlations between the previously encoded texture images. The skipped blocks are predicted from the neighboring depth images. Experimental results demonstrate that the proposed method not only achieves the drastically high coding performance but also reduces the complexity of the encoder.

The proposed method creates a high-resolution (HR) image on the basis of the frame registration of multiple low-resolution (LR) images. Not only does the super-resolution (SR) method based on using multiple LR images generally enhance the restored HR image quality compared to that based on using a single LR image, but it also increases the complexity and frame memory for hardware implementation. In order to generate an HR image, the multi-frame SR method has to estimate all motion vectors (MVs) between the target LR image and all the reference LR images. Additionally, the total frame memories used for storing LR images have to be preset according to the number of all the reference LR images. Therefore, the proposed multi-frame SR method focuses on a real-time and low frame memory system, thereby reducing the number of motion estimation (ME) operations and the total frame memory required, and preserving the image quality in an HR image restoration. First, we classify the input LR image into a feature and a uniform region in order to reduce the frame memory because the performance of SR algorithms is predominantly affected by restoring a feature region rather than a uniform region. Accordingly, we only save and use the feature region of the multiple LR images and not the uniform region for restoring an HR image. Next, the MV of each feature is estimated frame-wise to reduce the complexity of ME, and these MVs are accumulated as the feature trajectories through multiple LR frames. In the proposed method, the ME operation is conducted once between the reference LR image and the target LR image, and the estimated MVs are linked to the feature trajectories. These accumulated feature trajectories are used for generating an HR image. Experimental results show that the proposed multi-frame SR method can reduce the complexity and frame memory to one-third, while the quality of the restored HR image is equal to that obtained by using the conventional SR methods.

In this paper, we described geometric prediction structure for multi-view video coding. By exploiting the geometric relations between each camera pose, we can make prediction pair which maximizes the spatial correlation of each view. To analyze the relationship of each camera pose, we defined the mathematical mean view and view distance in 3D space. We proposed an algorithm for establishing the geometric prediction structure based on view center and view distance. Using this prediction structure, inter-view prediction is performed to camera pair of maximum spatial correlation. In our prediction structure, we also considered the view scalability. Experiments are done using JMVC software on MPEG-FTV test sequences. Overall performance is measured in the PSNR and subjective image quality measure such as PSPNR.

In this paper, we present a new video coding scheme to improve coding efficiency. For this purpose, we develop a loop filter that restores the compressed pictures in accordance with the image local characteristic. More specifically, a picture is segmented in such a way that the rate distortion cost is minimized and each pixel of the picture is restored using Wiener filter specific to the region. The restored picture is used for the motion-compensated prediction of future pictures and this can improve coding efficiency. In particular, we propose segmentation mode to reflect the local attributes of a picture. Finally, video coding that performs the new approach is implemented and results in 12.7% BD-bitrate reduction on average in Baseline Profile when compared to H.264/AVC.

A novel intra coding scheme is proposed to improve coding performance in intra pictures of H.264/AVC. The proposed method generates two sub-images, which are defined as a sampled image and a prediction error image in this paper, from an original image, and then encodes them separately. Especially, in an intra prediction process of encoding, the sampled image employs the original intra prediction modes, while the prediction error image uses newly defined four intra prediction modes. Experimental results demonstrate that the proposed method achieves significantly higher intra coding performance and reduces encoding complexity with the smaller number of a rate-distortion (RD) cost calculation process, as compared with the original intra coding method of H.264/AVC.

Multi-view video plus depth (MVD) format, which consists of texture and depth images, has been recently presented as video representation to support depth perception of scenes and efficient view generation at the arbitrary positions. In particular, a depth image has been one of the significantly important issues for successful services of highly advanced multi-media video applications, such as three-dimensional television (3DTV) and free-viewpoint television (FTV). In this paper, we present a novel approach for efficient multiview depth map coding. We assume that a texture image in the MVD format is first encoded and then the corresponding depth image is encoded. According to an analysis of interview correlation between the previously encoded texture images, the proposed method skips some blocks of the depth image without encoding. The skipped blocks in the depth map are predicted from the neighboring depth images at the same time instant. Experimental results demonstrate that the proposed method reduces the coding bitrate of up to 74.8% and improves PSNR of up to 3.51dB in P and B views.

This paper presents a bidirectional motion estimation (ME) method based on tracking feature trajectories and compensating for occlusion to enhance the temporal resolution of an input video sequence. First, we extract features and estimate their trajectories in the forward direction. We continue to track features only if the reliability of their trajectory is sufficiently high. Accordingly, if a feature can be continuously tracked through multiple frames, the proposed method assumes that its trajectory is a true motion vector (MV). Then, these forward feature trajectories are used as reference motion directions for backward block-based ME. If a block does not include any feature trajectories, we use the MVs of neighboring blocks as candidate MVs to propagate the predetermined true MV and preserve the spatial correlation of MVs. Further, the proposed method can detect occlusion regions, where a continuously tracked feature does not have a corresponding point in the current frame. In the detected occlusion region, the intermediate frame is generated from either the previous frame or the current frame. Outside the occlusion region, we generate the intermediate frames by taking into consideration the neighboring MVs to reduce blocking artifacts. Experimental results show that the proposed temporal frame interpolation (TFI) method can improve visual quality over conventional TFI methods, both objectively and subjectively.

Today's display devices are incapable of reproducing the full color range that humans can perceive; hence, color gamut expansion technology has become an active area of research. Among other things, the multi-primary display (MPD) device has received much attention lately because it provides simple and effective mechanisms for widening the color gamut. However, MPD has problems related to over-determined systems and metamerism. This paper seeks to provide solutions to these problems by proposing a cusp mapping method to address gamut mapping issues and LUT implementation of dimension reduction of over-determined systems to address primary decomposition issues. The proposed methods were tested on a 5-primary DLP/sup TM/ projection television. The experimental results address metamerism issues and show increasing as much as 20% at the color-reproducing region that the most common 3-primary DLP/sup TM/ projection television can not reproduce.