In this paper, we propose a novel motion vector processing (MVP) approach for temporal error concealment (TEC) in mobile video applications. Most existing TEC techniques estimate the lost motion vectors by minimizing a given distortion such as boundary variation or neighbor matching in pixel domain, which requires high power consumption compared to approaches that do not use pixel information. In wireless video applications, power consumption is a crucial factor and adoption of video postprocessing algorithms by mobile devices depends greatly on the complexity of the algorithms. The proposed algorithm uses only the received motion vectors without pixel data in the concealment process which achieves concealed video quality comparable to that of pixel domain approach but at much lower computational complexity. The performance gain is achieved by using new motion vector processing approaches that include (i) frame-to- frame motion detection, (ii) local motion classification and (iii) motion trajectory tracking. Furthermore, the proposed MVP-TEC approach provides better video quality compared to other TEC methods that do not perform motion analysis.

This paper presents a new multi-mode video object segmentation scheme to automatically segment head-and-shoulder objects from a video sequence. The proposed system supports two modes of segmentation: intra-mode and inter-mode. The intra-mode segmentation integrates skin detection, facial feature localization and verification, object shape approximation, split-and-merge region growing, and object region selection and guarantees good performance of the proposed segmentation scheme. The inter-mode object segmentation makes use of background modeling and subtraction to take advantage of temporal correlation of video frame while increasing the robustness of the segmentation and speeding up the performance. The main contributions of the proposed method includes: (i) a robust and efficient background modeling in inter-mode object segmentation, (ii) a facial feature verification and a multi-face separation algorithm in intra-mode segmentation. The performance of the algorithm is illustrated by simulation results carried out on various head-and-shoulder video sequences.

This paper proposes a metric to quantify the effect of the video frame loss according to their impact toward perceived temporal quality. This metric utilizes information obtained from pixel domain and particularly aims at measuring the temporal video quality degradation caused by both regular and irregular frame loss. As one application, the proposed temporal quality metric is used to evaluate the benefit of adaptive thresholding in frame skipping algorithms at the encoder. Temporal quality metric shows high prediction accuracy compared to subjective quality evaluation. Furthermore, it is shown by the experimental results that proposed temporal quality metric precisely differentiates between different frame skipping approaches and can be effectively used to evaluate them. With the help of the proposed quality metric, encoders can be designed to drop frames effectively with minimal perceptual video quality degradation.

Frame rate up conversion (FRUC) methods that employ motion have been proven to provide better image quality compared to nonmotion-based methods. While motion-based methods improve the quality of interpolation, artifacts are introduced in the presence of incorrect motion vectors. In this paper, we study the design problem of optimal temporal interpolation filter for motion-compensated FRUC (MC-FRUC). The optimal filter is obtained by minimizing the prediction error variance between the original frame and the interpolated frame. In FRUC applications, the original frame that is skipped is not available at the decoder, so models for the power spectral density of the original signal and prediction error are used to formulate the problem. The closed-form solution for the filter is obtained by Lagrange multipliers and statistical motion vector error modeling. The effect of motion vector errors on resulting optimal filters and prediction error is analyzed. The performance of the optimal filter is compared to nonadaptive temporal averaging filters by using two different motion vector reliability measures. The results confirm that to improve the quality of temporal interpolation in MC, the interpolation filter should be designed based on the reliability of motion vectors and the statistics of the MC prediction error.

In low bandwidth video coding applications, frame rate is reduced to increase the spatial quality of the frames. However, video sequences that are encoded at low frame rates demonstrate motion jerkiness artifacts when displayed. Therefore, a mechanism is required at the decoder to increase the frame rate while keeping an acceptable level of spatial quality. In this paper, we present a new method to perform video frame interpolation by sending effective side information for frame rate up conversion applications. The proposed scheme encodes the skipped frames lightly by sending motion vectors and an important information map which indicates to the decoder the type of interpolation method to perform. We also propose a novel overhead reduction method to keep the side information cost low. Experimental results show that the proposed algorithm outperforms decoder-only frame rate up conversion methods and gives better performance in terms of PSNR and visual quality over encoding at full frame rate without frame skipping.

In this paper, the effect of motion vector accuracy on the efficiency of motion compensated frame rate up conversion (MC-FRUC) is studied. The motion vector processing problem is formulated and analyzed via motion vector modelling. A processing scheme is proposed to improve the interpolated frame quality at the decoder. The practical application of integrating the motion vector processing algorithm in a standard H.263 decoder for MC-FRUC is also discussed. Experimental results show that a 0.4-0.6 dB gain in the H.263 codec, and a 0.5-2 dB gain in off-line frame rate conversion can be obtained by the proposed algorithm.

In this paper, we present theoretical analysis on how the global motion parameter accuracies affect the efficiency of motion compensated video coding. The inaccurate global motion compensation is modelled by introducing probabilistic rotation, scale and translation parameter errors. Approximate expressions that relate the power spectrum of the prediction error to motion parameter errors is derived. By doubling the accuracy of the motion parameters, up to 6 dB theoretical gains can be obtained in prediction error variance.