This paper addresses an approach to video coding utilizing an iterative nonorthogonal expansion technique called "matching pursuits" (MP) in combination with a new algorithm for selecting an appropriate coding technique at each frame in a sequence. This decision algorithm is called "selective update" and is based on an estimate of the amount and type of motion occurring between coded frames in a video sequence. This paper demonstrates that the matching pursuits approach is most efficient for video coding when motion compensation results in a prediction error which is well localized to the edges of moving objects. In the presence of global motion, such as panning and zooming, or in the presence of objects entering or leaving a scene, matching pursuits becomes less effective than orthogonal transform-based coding techniques like the block-based discrete cosine transform (DCT). The rate-distortion characteristics of matching pursuits and block-wise DCT coding are used to demonstrate how MP coding can be more efficient than block-wise DCT-based coding. When an appropriate combination of these nonorthogonal and orthogonal transforms are used for encoding a complete low bit-rate video sequence, improved overall compression efficiency can be achieved. Results are shown which demonstrate the effectiveness of a hybrid video codec based on this concept.

We briefly describe some of our work on the use of stochastic models to describe the displacement vector field (DVF) in an image sequence. Specifically, autoregressive models are used which describe the abrupt transitions in the DVF with the use of a line process, but also result in spatio-temporally recursive structures. The use of such models in developing maximum a posteriori estimators for the DVF and the line process is subsequently described. Finally, the extension and application of the resulting estimator to the problems of object tracking, video compression and restoration of video sequences is reviewed.

An important component of any spatial temporal gradient motion estimation algorithm is the accuracy by which spatial gradients are calculated. When an image sequence is corrupted by noise, the problem of determining these spatial gradients becomes extremely difficult. This is immediately apparent, since the magnitude response of the derivative operator is |/spl omega/|. In other words, the components of an image are amplified upon differentiation in proportion to their frequency value. Thus, high-frequency noise terms will dominate any low-frequency features in the differentiated image. If this corrupted differentiated image is then used within a spatio-temporal gradient motion estimator, the noise will erroneously influence the estimated motion vector. The problem of estimating the spatial gradient is treated as an inverse problem with noise. Formulating the problem in this manner results in a recursive gradient estimator that suppresses the effects of noise.

In this paper, a thorough review is presented of noise reduction filters for digital image sequences. Detailed descriptions of several spatiotemporal and temporal noise reduction algorithms are provided. To aid in comparing between these different algorithms, we classify them based on their support (i.e., 3-D or 1-D filter) and whether or not motion compensation is employed. Several algorithms from each of the four categories are implemented and tested on real sequences degraded to various signal-to-noise ratios. These experimental results are discussed and analyzed to determine the overall advantages and disadvantages of the four general classifications, as well as, the individual filters.>

Image and video coding algorithms have found a number of applications ranging from video telephony on the public switched telephone networks (PSTN) to HDTV. However, as the bit rate is lowered, most of the existing techniques, as well as current standards, such as JPEG, H. 261, and MPEG-1 produce highly visible degradations in the reconstructed images primarily due to the information loss caused by the quantization process. In this paper, we propose an iterative technique to reduce the unwanted degradations, such as blocking and mosquito artifacts while keeping the necessary detail present in the original image. The proposed technique makes use of a priori information about the original image through a nonstationary Gauss-Markov model. Utilizing this model, a maximum a posteriori (MAP) estimate is obtained iteratively using mean field annealing. The fidelity to the data is preserved by projecting the image onto a constraint set defined by the quantizer at each iteration. The proposed solution represents an implementation of a paradigm we advocate, according to which the decoder is not simply undoing the operations performed by the encoder, but instead it solves an estimation problem based on the available bitstream and any prior knowledge about the source image. The performance of the proposed algorithm was tested on a JPEG, as well as on an H.261-type video codec. It is shown to be effective in removing the coding artifacts present in low bit rate compression.>

A recursive model-based algorithm for obtaining the maximum a posteriori (MAP) estimate of the displacement vector field (DVF) from successive image frames of an image sequence is presented. To model the DVF, we develop a nonstationary vector field model called the vector coupled Gauss-Markov (VCGM) model. The VCGM model consists of two levels: an upper level, which is made up of several submodels with various characteristics, and a lower level or line process, which governs the transitions between the submodels. A detailed line process is proposed. The VCGM model is well suited for estimating the DVF since the resulting estimates preserve the boundaries between the differently moving areas in an image sequence. A Kalman type estimator results, followed by a decision criterion for choosing the appropriate line process. Several experiments demonstrate the superior performance of the proposed algorithm with respect to prediction error, interpolation error, and robustness to noise.

We develop a recursive model-based maximum a posteriori (MAP) estimator that simultaneously estimates the displacement vector field (DVF) and the intensity field from a noisy-blurred image sequence. Current motion-compensated spatio-temporal noise filters treat the estimation of the DVF as a preprocessing step. Generally, no attempt is made to verify the accuracy of these estimates prior to their use in the filter. By simultaneously estimating these two fields, we establish a link between the two estimators. It is through this link that the DVF estimate and its corresponding accuracy information are shared with the other intensity estimator, and vice versa. To model the DVF and the intensity field, we use coupled Gauss-Markov (CGM) models. A CGM model consists of two levels: an upper level, which is made up of several submodels with various characteristics, and a lower level or line field, which governs the transitions between the submodels. The CGM models are well suited for estimating the displacement and intensity fields since the resulting estimates preserve the boundaries between the stationary areas present in both fields. Detailed line fields are proposed for the modeling of these boundaries, which also take into account the correlations that exist between these two fields. A Kalman-type estimator results, followed by a decision criterion for choosing the appropriate set of line fields. Several experiments using noisy and noisy-blurred image sequences demonstrate the superior performance of the proposed algorithm with respect to prediction error and mean-square error.

Most of the transform-based image compression techniques produce visible artifacts in the reconstructed image, particularly at low bit rates. In this paper, we propose an iterative technique to reduce the unwanted degradations such as blocking artifacts while keeping the necessary detail present in the original image. The proposed technique makes use of a priori information about the original image by using a nonstationary Gauss-Markov model. A MAP estimate is obtained iteratively using mean field annealing. A additional a priori information about the transform of the original image is incorporated into the estimation process by projecting the image onto a set defined by the quantizer at each iteration. The performance of the proposed algorithm was tested on JPEG compressed images. It is shown to be effective in removing the coding artifacts present in the low bit rate compressed images.>

In this paper, we present a novel coding technique that makes use of the nonstationary characteristics of an image sequence displacement field to estimate and encode motion information. We utilize an MPEG style codec in which the anchor frames in a sequence are encoded with a hybrid approach using quadtree, DCT, and wavelet-based coding techniques. A quadtree structured approach is also utilized for the interframe information. The main objective of the overall design is to demonstrate the coding potential of a newly developed motion estimator called the coupled linearized MAP (CLMAP) estimator. This estimator can be used as a means for producing motion vectors that may be regenerated at the decoder with a coarsely quantized error term created in the encoder. The motion estimator generates highly accurate motion estimates from this coarsely quantized data. This permits the elimination of a separately coded displaced frame difference (DFD) and coded motion vectors. For low bit rate applications, this is especially important because the overhead associated with the transmission of motion vectors may become prohibitive. We exploit both the advantages of the nonstationary motion estimator and the effective compression of the anchor frame coder to improve the visual quality of reconstructed QCIF format color image sequences at low bit rates. Comparisons are made with other video coding methods, including the H.261 and MPEG standards and a pel-recursive-based codec.

In this paper, we develop a recursive model-based maximum a posteriori (MAP) estimator that simultaneously estimates the displacement vector field (DVF) and intensity field from a noisy- blurred image sequence. Current motion-compensated spatio-temporal filters treat the estimation of the DVF as a preprocessing step. Thus, no attempt is made to verify the accuracy of these estimates prior to their use in the filter. By simultaneously estimating these two fields, information is made available to each filter regarding the reliability of the estimates provided by the other filter. Nonstationary models are used for both the DVF and the intensity field in the proposed estimator, thus avoiding the smoothing of boundaries present in both.

In this paper, we develop an algorithm for obtaining the maximum a posteriori (MAP) estimate of the displacement vector field (DVF) from two consecutive image frames of an image sequence acquired under quantum-limited conditions. The estimation of the DVF has applications in temporal filtering, object tracking, and frame registration in low-light level image sequences as well as low-dose clinical x-ray image sequences. The quantum-limited effect is modeled as an undesirable, Poisson-distributed, signal-dependent noise artifact. The specification of priors for the DVF allows a smoothness constraint for the vector field. In addition, discontinuities of the field are taken into account through the introduction of a line process for neighboring vectors. A Bayesian formulation is used in this paper to estimate the DVF and a block component algorithm is employed in obtaining a solution. Several experiments involving a phantom sequence show the effectiveness of this estimator in obtaining the DVF under severe quantum noise conditions.

A new image quality metric consistent with the properties of the human visual system is derived. Using the EM algorithm, the authors restore a blurred image and quantify the improvement in image quality with both the new metric and the mean square error (MSE). From these results, the advantages of the new metric are obvious. The EM algorithm is modified according to the underlying mathematical structure of the new metric, which results in improved performance.>