This paper studies the problem of low rank approximation of light fields for compression. A homography-based approximation method is proposed which jointly searches for homographies to align the different views of the light field together with the low rank approximation matrices. We first consider a global homography per view and show that depending on the variance of the disparity across views, the global homography is not sufficient to well-align the entire images. In a second step, we thus consider multiple homographies, one per region, the region being extracted using depth information. We first show the benefit of the joint optimization of the homographies together with the low-rank approximation. The resulting compact representation is then compressed using HEVC and the results are compared with those obtained by directly applying HEVC on the light field views re-structured as a video sequence. The experiments using different data sets show substantial PSNR-rate gain of our method, especially for real light fields.

Traditional quality estimators evaluate an image's resemblance to a reference image. However, quality estimators are not well suited to the similar but somewhat different task of utility estimation, where an image is judged instead by how useful it would be in comparison to a reference in the context of accomplishing some task. Multi-Scale Difference of Gaussian Utility (MS-DGU), a reduced-reference algorithm for image utility estimation, relies on matching image contours across scales tuned to spatial frequencies important for utility estimation. MS-DGU estimates utility with greater accuracy than previous techniques. A fast algorithm for utility-optimized image compression was developed through rate-utility optimization for MS-DGU. By simple scaling of JPEG quantization step sizes according to a “utility factor,” data rates were reduced by an average of 24% (and up to 30%) compared to standard JPEG while maintaining utility.

Reduced-reference (RR) video quality assessment methods estimate the quality of a distorted video using only partial information of the reference video. We propose a reduced reference technique based on a no-reference method presented in [1], in which the MSE of the received video with packet losses was estimated with five parameters estimated from the received video and the motion vectors, all for each macroblock. We instead compute the parameters at the encoder and transmit a subset as overhead information. The originally proposed five parameters can be reduced to four, and with appropriate parameter subsampling, the MSE can be estimated across multiple packet loss rates, video resolutions, and bit rates. Overhead is a function of video size, frame rate, and subsampling. Estimation accuracy is good at overheads as low as 95 Kbps for video coded in the range of 500 kilobits/second to 2 megabits/second. The technique does not require full decoding of the video and is therefore suitable for low-complexity in-network quality monitoring.

One- and two-way communication with digital compressed visual signals is now an integral part of the daily life of millions. Such commonplace use has been realized by decades of advances in visual signal compression. The design of effective, efficient compression and transmission strategies for visual signals may benefit from proper incorporation of human visual system (HVS) characteristics. This paper overviews psychophysics and engineering associated with the communication of visual signals. It presents a short history of advances in perceptual visual signal compression, and describes perceptual models and how they are embedded into systems for compression and transmission, both with and without current compression standards.

User experiences in 2D and 3D videoconferencing are evaluated and compared. An experimental system is designed that uses video direct-feed in 2D or 3D, providing nearly life-sized across-the-table videoconferencing to two participants without compression or transmission artifacts. 3D is achieved via polarization, selected because of its high resolution and high potential for eye contact. User experience is evaluated via a subjective test with two interactive tasks. The experiment is completed by three groups, who interact in 3D, in 2D (without polarizing glasses), and in 2D while wearing glasses, serving as a control for the use of glasses. Users of the system in 3D reported an increased ability to perceive depth, but otherwise reported similar user experiences to 2D users relating to quality of interaction. Wearing 3D glasses did not adversely impact user experience.

Welcome to the 2012 IEEE International Conference on Image Processing (ICIP), the 19th in the series of ICIPs. We hope you find ICIP inspiring and rewarding both for the knowledge you gain of advances in Image and Video processing through the conference technical presentations, and for the equally valuable “side information” that comes from personal interactions and hallway conversations with colleagues and friends, old and new, that you meet here.

Real-time, two-way transmission of American Sign Language (ASL) video over cellular networks provides natural communication among members of the Deaf community. As a communication tool, compressed ASL video must be evaluated according to the intelligibility of the conversation, not according to conventional definitions of video quality. Guided by linguistic principles and human perception of ASL, this paper proposes a full-reference computational model of intelligibility for ASL (CIM-ASL) that is suitable for evaluating compressed ASL video. The CIM-ASL measures distortions only in regions relevant for ASL communication, using spatial and temporal pooling mechanisms that vary the contribution of distortions according to their relative impact on the intelligibility of the compressed video. The model is trained and evaluating using ground truth experimental data collected in three separate studies. The CIM-ASL provides accurate estimates of subjective intelligibility and demonstrates statistically significant improvements over computational models traditionally used to estimate video quality. The CIM-ASL is incorporated into an H.264 compliant video coding framework, creating a closed-loop encoding system optimized explicitly for ASL intelligibility. The ASL-optimized encoder achieves bitrate reductions between 10% and 42%, without reducing intelligibility, when compared to a general purpose H.264 encoder.

Objective estimators for video are expected to estimate accurately subjective ratings provided by humans. This work presents a subjective experiment designed to acquire intelligibility ratings for a collection of compressed ASL videos. The distortions present in the experimental database are analyzed in terms of their impact on the performance of objective estimators. Distortions that do not significantly vary across space or time cannot adequately challenge traditional objective estimators, such as PSNR and RMS distortion contrast, and an objective intelligibility measure designed specifically for ASL video provides negligible improvements in prediction accuracy. Distortions that vary across space and time, affecting only localized regions in the video, are considered spatially and temporally diverse. When the distortions present in the experimental database are sufficiently diverse, the objective intelligibility measure estimates subjective ratings more accurately than PSNR and RMS distortion contrast.

Many applications value an assessment of distorted natural images according to their usefulness, or utility, rather than their perceptual quality. For the quality task, human observers evaluate an image based on its perceptual resemblance to a reference, whereas for the utility task, the usefulness of an image as a surrogate for a reference is under evaluation. This paper presents a novel technique for acquiring perceived utility scores derived from textual descriptions produced by observers viewing images. The technique uses an observer-centric approach, so observers dictate the relevant concepts that characterize image usefulness. This technique is used to collect perceived utility (PU) scores for 150 distorted images that simulate scenes captured by a surveillance system. The capability of both the natural image contour evaluation (NICE) utility estimator, which compares contours of the reference and test images, and popular quality estimators to estimate PU is reported. The conclusions drawn from the results augment previously reported results and establish that a multi-scale implementation of NICE (MS-NICE) is the most robust utility estimator among the estimators evaluated, since MS-NICE consistently performs as well as estimators producing the most accurate perceived utility estimates for various distortion types.

Detection of visually salient image regions is useful for applications like object segmentation, adaptive compression, and object recognition. In this paper, we introduce a method for salient region detection that outputs full resolution saliency maps with well-defined boundaries of salient objects. These boundaries are preserved by retaining substantially more frequency content from the original image than other existing techniques. Our method exploits features of color and luminance, is simple to implement, and is computationally efficient. We compare our algorithm to five state-of-the-art salient region detection methods with a frequency domain analysis, ground truth, and a salient object segmentation application. Our method outperforms the five algorithms both on the ground-truth evaluation and on the segmentation task by achieving both higher precision and better recall.

In the quality assessment task, observers evaluate a natural image based on its perceptual resemblance to a reference. For the utility assessment task, observers evaluate the usefulness of a natural image as a surrogate for a reference. Humans generally use the information captured by an imaging system and tolerate distortions as long as the underlying task is performed reliably. Conventional notions of perceived quality cannot generally predict the perceived utility of a natural image. This paper examines variations to basic components of a recently introduced utility assessment algorithm that compares the contours of a reference and test image, referred to as the natural image contour evaluation (NICE), in terms of their capability to improve the prediction of perceived utility scores. Results show that classical edge-detection algorithms incorporated into NICE provide statistically equivalent performance to other, more complex edge-detection algorithms.

Image quality metrics attempt to assess the perceived difference between an original and a distorted image. One aspect of quality assessment that is not yet well understood is the scale at which images are analyzed. This paper examines the effects that two scale-reducing operations, filtering and down-sampling, have on quality assessment. It has been shown that several popular quality metrics are equivalent to weighted measurements of mean-squared-error (MSE). Analysis is provided suggesting that the relative rankings of weighted MSE computations are not very sensitive to the scale of the input images; these predictions are verified experimentally using both weighted MSE-based and non-weighted-MSE-based assessment techniques. Featured algorithms include MSE, structural similarity index (SSIM), multi-scale SSIM (MSSIM), visual information fidelity (VIF) and visual signal-to-noise ratio (VSNR). Extensive testing on the LIVE database demonstrates that all algorithms except VSNR can be computed using images decimated by a factor of 6-8 without a substantial degradation in rank-order performance. This result illustrates that significant savings in processing are therefore possible.

In this paper, we propose a highly efficient algorithm to model the human skin color. The underlying algorithm involves generating a discrete Cosine transform (DCT) at each pixel location, using the surrounding points. These DCT coefficients are assumed to follow a generalized Gaussian distribution (GGD). Next, the model parameters are estimated using the maximum-likelihood (ML) criterion applied to a set of training skin samples. Finally, each pixel is classified as skin or the opposite if its likelihood ratio is above some threshold. The experimental results show that our model avoids excessive false detection while still retaining a high degree of correct detection.

The structural similarity (SSIM) metric and its multi-scale extension (MS-SSIM) evaluate visual quality with a modified local measure of spatial correlation consisting of three components: mean, variance, and cross-correlation. This paper investigates how the SSIM components contribute to its quality evaluation of common image artifacts. The predictive performance of the individual components and pairwise component products is assessed using the LIVE image database. After a nonlinear mapping, the product of the variance and cross- correlation components yields nearly identical linear correlation with subjective ratings as the complete SSIM and MS- SSIM computations. A computationally simple alternative to SSIM (c.f Eq. (6)) that ignores the mean component and sets the local average patch values to 128 exhibits a 1% decrease in linear correlation with subjective ratings to 0.934 from the complete SSIM evaluation with an over 20% reduction in the number of multiplications.

The H.264/ACV standard provides significant improvements in performance over earlier video coding standards at the cost of increased complexity. Our challenge is to determine H.264 parameter settings that have low complexity but still offer high video quality. In this paper, we propose two fast algorithms for finding the H.264 parameter settings that take about 1% and 8%, respectively, of the number of tests required by an exhaustive search. Both the fast algorithms result in a maximum decrease in peak-signal-to-noise ratio of less than 0.71 dB for different data sets and bitrates

This paper presents an efficient metric for quantifying the visual fidelity of natural images based on near-threshold and suprathreshold properties of human vision. The proposed metric, the visual signal-to-noise ratio (VSNR), operates via a two-stage approach. In the first stage, contrast thresholds for detection of distortions in the presence of natural images are computed via wavelet-based models of visual masking and visual summation in order to determine whether the distortions in the distorted image are visible. If the distortions are below the threshold of detection, the distorted image is deemed to be of perfect visual fidelity (VSNR = infinity) and no further analysis is required. If the distortions are suprathreshold, a second stage is applied which operates based on the low-level visual property of perceived contrast, and the mid-level visual property of global precedence. These two properties are modeled as Euclidean distances in distortion-contrast space of a multiscale wavelet decomposition, and VSNR is computed based on a simple linear sum of these distances. The proposed VSNR metric is generally competitive with current metrics of visual fidelity; it is efficient both in terms of its low computational complexity and in terms of its low memory requirements; and it operates based on physical luminances and visual angle (rather than on digital pixel values and pixel-based dimensions) to accommodate different viewing conditions.

Real-time rate-control for wavelet image coding requires characterization of the rate required to code quantized wavelet data. An ideal robust solution can be used with any wavelet coder and any quantization scheme. A large number of wavelet quantization schemes (perceptual and otherwise) are based on scalar dead-zone quantization of wavelet coefficients. A key to performing rate-control is, thus, fast, accurate characterization of the relationship between rate and quantization step size, the R-Q curve. A solution is presented using two invocations of the coder that estimates the slope of each R-Q curve via probability modeling. The method is robust to choices of probability models, quantization schemes and wavelet coders. Because of extreme robustness to probability modeling, a fast approximation to spatially adaptive probability modeling can be used in the solution, as well. With respect to achieving a target rate, the proposed approach and associated fast approximation yield average percentage errors around 0.5% and 1.0% on images in the test set. By comparison, 2-coding-pass rho-domain modeling yields errors around 2.0%, and post-compression rate-distortion optimization yields average errors of around 1.0% at rates below 0.5 bits-per-pixel (bpp) that decrease down to about 0.5% at 1.0 bpp; both methods exhibit more competitive performance on the larger images. The proposed method and fast approximation approach are also similar in speed to the other state-of-the-art methods. In addition to possessing speed and accuracy, the proposed method does not require any training and can maintain precise control over wavelet step sizes, which adds flexibility to a wavelet-based image-coding system.

Under a rate constraint, wavelet-based image coding involves strategic discarding of information such that the remaining data can be described with a given amount of rate. In a practical coding system, this task requires knowledge of the relationship between quantization step size and compressed rate for each group of wavelet coefficients, the R-Q curve. A common approach to this problem is to fit each subband with a scalar probability distribution and compute entropy estimates based on the model. This approach is not effective at rates below 1.0 bits-per-pixel because the distributions of quantized data do not reflect the dependencies in coefficient magnitudes. These dependencies can be addressed with doubly stochastic models, which have been previously proposed to characterize more localized behavior, though there are tradeoffs between storage, computation time, and accuracy. Using a doubly stochastic generalized Gaussian model, it is demonstrated that the relationship between step size and rate is accurately described by a low degree polynomial in the logarithm of the step size. Based on this observation, an entropy estimation scheme is presented which offers an excellent tradeoff between speed and accuracy; after a simple data-gathering step, estimates are computed instantaneously by evaluating a single polynomial for each group of wavelet coefficients quantized with the same step size. These estimates are on average within 3% of a desired target rate for several of state-of-the-art coders.

Many modern wavelet quantization schemes specify wavelet coefficient step sizes as continuous functions of an input step-size selection criterion; rate control is achieved by selecting an appropriate set of step sizes. In embedded wavelet coders, however, rate control is achieved simply by truncating the coded bit stream at the desired rate. The order in which wavelet data are coded implicitly controls quantization step sizes applied to create the reconstructed image. Since these step sizes are effectively discontinuous, piecewise-constant functions of rate, this paper examines the problem of designing a coding order for such a coder, guided by a quantization scheme where step sizes evolve continuously with rate. In particular, it formulates an optimization problem that minimizes the average relative difference between the piecewise-constant implicit step sizes associated with a layered coding strategy and the smooth step sizes given by a quantization scheme. The solution to this problem implies a coding order. Elegant, near-optimal solutions are presented to optimize step sizes over a variety of regions of rates, either continuous or discrete. This method can be used to create layers of coded data using any scalar quantization scheme combined with any wavelet bit-plane coder. It is illustrated using a variety of state-of-the-art coders and quantization schemes. In addition, the proposed method is verified with objective and subjective testing.

In a portable device, such as a digital camera, limitations on storage are an important consideration. In addition, due to constraints on the complexity of available hardware, image coding algorithms must be fairly simple in implementation. This work presents one such efficient method for coding multiple images of a scene, in a manner that complements a post-processing-based enhancement system. Super-resolution, image restoration and de-noising algorithms have demonstrated the ability to improve the quality of an image using multiple blurry, noisy copies of the same scene. This additional quality does not come without cost, however, since an image capture system must store each image. The proposed encoding scheme is derived from a general linear system model, and encodes multiple images of the same scene, with different amounts of blurring. It is also compared with a variety of methods based on current camera compression technology. For the tested images, this approach requires one-half the rate required by other methods at lower rates. In addition, for a small performance loss, it is essentially implementable without using any compression hardware

We consider the problem of constructing visually optimized balanced multiple descriptions of images, instead of being op- timized with the conventional measure of mean squared error (MSE). The recently proposed Modified Multiple Description Scalar Quantizer (MMDSQ) is used because its central and side quantizers all have convex and uniform cells. Since the majority of research results on the visual distortion caused by quantization noise assumes convex and uniform quantizer cells, they can be conveniently applied to the system based on MMDSQ. The technique provides substantially higher side reconstruction perceived quality when compared at the same central reconstruction quality, which is verified through per- ceptual tests.

Wavelet-based spatial quantization is a technique to compress image data that adapts the compression to the data in each region of an image. This approach is motivated because quantization with a single step-size does not result in a uniform visual effect across each spatial location; different types of image content mask quantization errors in different ways. While many spatial quantization techniques determine step-sizes via local activity measures, the proposed method induces local quantization distortion based on experiments that quantify human detection of this distortion as function of both the contrast and the type of the image data. Three types in particular, textures, structures and edges, are considered. A classifier is utilized to detect to which of these three categories a local region of image data belongs, and step-sizes are then derived based on the contrast and class of each region. Class and contrast data are conveyed to the coder with explicit side information. For images compressed at threshold, the proposed method requires 3-10 % less rate than a similar previous approach without classification, and on average produces images that are preferred by 2/3 of tested viewers.

Video distortion metrics based on models of the human visual system have traditionally used comparisons between the distorted signal and a reference signal to calculate distortions objectively. In video coding applications, this is not prohibitive. In quality monitoring applications, however, access to the reference signal is often limited. This paper presents a computationally efficient video distortion metric that can operate in full- or reduced-reference mode as required. The metric is based on a model of the human visual system implemented using the wavelet transform and separable filters. The visual model is parameterized using a set of video frames and the associated quality scores. The visual model's hierarchical structure, as well as the limited impact of fine scale distortions on quality judgments of severely impaired video, are exploited to build a framework for scaling the bitrate required to represent the reference signal. Two applications of the metric are also presented. In the first, the metric is used as the distortion measure in a rate-distortion optimized rate control algorithm for MPEG-2 video compression. The resulting compressed video sequences demonstrate significant improvements in visual quality over compressed sequences with allocations determined by the TM5 rate control algorithm operating with MPEG-2 at the same rate. In the second, the metric is used to estimate time series of objective quality scores for distorted video sequences using reference bitrates as low as 10 kb/s. The resulting quality scores more accurately model subjective quality recordings than do those estimated using the mean squared error as a distortion metric, while requiring a fraction of the bitrate used to represent the reference signal. The reduced-reference metric's performance is comparable to that of the full-reference metrics tested in the first Video Quality Experts Group evaluation.

A rate control technique for video encoding under complex transmission scenarios is presented. A typical application for this method is the transmission of video over variable bit rate channels while accounting for restrictions on the end-to-end delay and decoder buffer size. That the resulting multiple constraints on the source and channel rates may be relaxed without loss of optimality into a set of linear inequality constraints-though they are usually expressed in nonlinear form-is a key insight of this paper. This allows for a systematic treatment of a large class of rate constraints and leads to a convex programming (CP) formulation for rate control. Approximation of the frame distortion-rate data by piecewise linear functions further facilitates an efficient solution based on linear programming (LP), a special case of CP. The LP method provides bounds for the deviation from optimality. Results for a standard video test set show that the proposed method provides solutions with mean square error (MSE) distortion value within 2% of the global minimum across a range of rates. The proposed technique is also applied in conjunction with a perceived distortion measure. Results exhibit significant reduction in blocking artifacts and flicker compared to the use of MSE

The multiple description (MD) problem has received considerable attention as a model of information transmission over unreliable channels. A general framework for designing efficient MD quantization schemes is proposed in this paper. We provide a systematic treatment of the El Gamal-Cover (EGC) achievable MD rate-distortion region, and show it can be decomposed into a simplified-EGC (SEGC) region and a superimposed refinement operation. Furthermore, any point in the SEGC region can be achieved via a successive quantization scheme along with quantization splitting. For the quadratic Gaussian case, the proposed scheme has an intrinsic connection with the Gram-Schmidt orthogonalization, which implies that the whole Gaussian MD rate-distortion region is achievable with a sequential dithered lattice-based quantization scheme as the dimension of the (optimal) lattice quantizers becomes large. Moreover, this scheme is shown to be universal for all independent and identically distributed (i.i.d.) smooth sources with performance no worse than that for an i.i.d. Gaussian source with the same variance and asymptotically optimal at high resolution. A class of MD scalar quantizers in the proposed general framework is also constructed and is illustrated geometrically; the performance is analyzed in the high-resolution regime, which exhibits a noticeable improvement over the existing MD scalar quantization schemes

This paper addresses the problem of bit allocation among frames in a predictively encoded video sequence. Finding optimal solutions to this problem potentially requires making an exponential number of calls to the encoder. To better understand the structure of the rate-distortion data output by video encoders, a simple model of a sequentially encoded autoregressive Gaussian random field is theoretically investigated. The rate-distortion data for the model exhibits an additive-separability property, i.e. the rate can be decomposed into a sum of independent functions of single distortion variables. This property implies the near-optimal behavior of a non-backtracking steepest-descent (SD) based bit allocation algorithm. The SD algorithm when applied to video coding produces near-optimal solutions by making a linear number of calls to the encoder. Results are presented for MPEG-2 encoding of standard video sequences.

We investigate using a lattice and a staggered version of it for balanced multiple description quantization at high-resolution. By letting one side quantizer use a lattice codebook, and the other side quantizer use a codebook based on the same lattice with a slight shift, the joint quantizer as the central quantizer can naturally achieve a performance improvement over the side quantizers. Application of such a structure is meaningful when the side quantizers' distortions are of (equal) primary concern. Another immediate application of such a structure is when a refinement stage is added to further reduce the central distortion. We analyze the performance of this structure for these two applications, and then evaluate the root lattices A/sub n/, D/sub n/ (with dimension less than 9), E/sub 6/, E/sub 7/, E/sub 8/ and their duals. The results suggest that these two applications in fact lead to different choices of lattices: while the former requires good lattices in terms of their normalized second moments of inertia, the latter appears to perform better when the staggering of lattices generates more cells by intersection of the Voronoi regions of the two lattices. These observations also suggest a suboptimal but simple multiple description quantization scheme using lattices.

Recent developments in psychovisual modeling have led to improvements in wavelet-based coder performance. A spatially selective quantizer based on texture masking sensitivities is introduced, which hides distortion in high-contrast portions of images. Unlike other spatial quantization schemes, this method requires explicit side information to convey stepsizes. A simple coder is presented which leverages this side information to reduce the rate required to code the quantized data. Side information coding is also discussed. With respect to visual quality, this compression scheme performs competitively with a CSF-optimized JPEG-2000 coder at equivalent rates.

Robust video coding is creating coded video data which provides received and decoded video of application-appropriate quality when transmitted over an imperfect channel. Video contains an enormous amount of redundancy, both from signal processing and psychological perspectives, which effective compression attempts to remove. However, ideal source coding leaves a coded stream which is extremely vulnerable to errors, and therefore compression and transmission provisions that avoid catastrophic failure caused by lost, delayed, or errant data are imperative. Robustness in video coding is obtained through source coding, channel coding, and error concealment following decoding. This paper first discusses issues specific to video in providing robustness and then poses the problem with two extremes of providing robustness and an example. Common strategies employed in robust video coding are then briefly described, and the specific application of these strategies in session papers are presented. The paper concludes with a discussion of future directions in robust video coding.

Real-time rate-control requires fast, accurate characterization of the relationship between wavelet subband step-size and rate. A rate-control algorithm is presented based on local properties of the function parameterizing this relationship, the rate-quantization or R-Q curve, for each subband. The behavior of this function is characterized by using the coder once, with step-sizes close to those that achieve the desired rate, which are determined entirely by a model-based technique. This solution requires a coder, a performance metric (which implies a subband-quantization strategy), and continuously differentiable (C/sup 1/) R-Q curves. This approach differs from some previous approaches in that it is based on local structure of R-Q curves instead of a global property, and does not require any training. The proposed scheme produces step-sizes that achieve rates typically less than half a percent away from the target rate. This method is fast, accurate, coder independent, requires no training, and can be combined with any subband-based quantization scheme.

We propose a new class of universal multiple description lattice quantizers based on the method of quantization splitting. For Gaussian sources and squared error distortion measure, our scheme can achieve the whole multiple description rate-distortion region, as the dimension of the (optimal) lattice quantizers becomes large

We introduce a new class of multiple description scalar quantizer (MDSQ), which will be referred to as the modified MDSQ (MMDSQ). The structure of MMDSQ is fundamentally different from the MDSQ proposed by Vaishampayan. Analysis shows that at high rates for sources with smooth pdfs, MMDSQ achieves the same performance as entropy-constrained MDSQ using a uniform central quantizer. Compared with MDSQ, MMDSQ features a more efficient central-side distortion tradeoff control mechanism, without requiring the design and implementation of a complicated index assignment scheme. The simple and efficient implementation of MMDSQ makes it suitable for image coding: It can be naturally incorporated into existing coders to form new multiple description coders. Such a system is constructed using an existing image coder, and the performance of the resulting system is compared to that of the same coder but using Vaishampayan's MDSQ. Simulation results on natural images reveal that the system based on MMDSQ outperforms that based on MDSQ.

This paper presents a contrast-based quantization strategy for use in lossy wavelet image compression that attempts to preserve visual quality at any bit rate. Based on the results of recent psychophysical experiments using near-threshold and suprathreshold wavelet subband quantization distortions presented against natural-image backgrounds, subbands are quantized such that the distortions in the reconstructed image exhibit root-mean-squared contrasts selected based on image, subband, and display characteristics and on a measure of total visual distortion so as to preserve the visual system's ability to integrate edge structure across scale space. Within a single, unified framework, the proposed contrast-based strategy yields images which are competitive in visual quality with results from current visually lossless approaches at high bit rates and which demonstrate improved visual quality over current visually lossy approaches at low bit rates. This strategy operates in the context of both nonembedded and embedded quantization, the latter of which yields a highly scalable codestream which attempts to maintain visual quality at all bit rates; a specific application of the proposed algorithm to JPEG-2000 is presented.

A steepest descent based bit allocation method with polynomial iteration complexity for minimizing the sum of frame distortions under a total bit rate constraint is presented for a predictive video coder. Previous algorithms that solve this problem have been based on Lagrangian relaxation followed by unconstrained optimization. Some of these methods have complexity exponential in the prediction depth in the worst case. Moreover, the convergence properties of the faster methods are difficult to analyze. A steepest descent method is utilized here for its comparatively low computational complexity, excellent performance and ease of analysis. Sufficient conditions for global optimality are presented without assuming independent encoding of frames. Results for dependent bit allocation through the assignment of quantizer step-sizes to frames in MPEG-2 encoded sequences suggest that these optimality conditions are to a large extent satisfied in practice. A PSNR improvement of up to 1.5 dB is obtained over the standard TM5 rate control algorithm for MPEG-2.

This paper introduces a new sequential design method for multiple description scalar quantizers (MDSQs) to generate two or more balanced descriptions. In this design method, a multiple description system is divided into multiple stages, and the m-th stage can be understood intuitively as minimizing the distortion of receiving any m of all the descriptions, while looking ahead to the next stage to reduce the distortion if more descriptions are present. Entropy-constrained sequential MDSQ with two descriptions is shown to achieve the same asymptotic performance as entropy-constrained MDSQ with uniform stepsize. Then this method is applied to the design of sequential MDSQ with three descriptions, for which two slightly different designs are given and compared with a three description system based on unequal loss protection at high rate. The results suggest that if the quality of the decoded source with two or more descriptions (rather than a single description) is most important, general multiple description systems should be favored over unequal loss protection systems. However, if the quality of the decoded source with a single description is most important (in the case of, for example, high channel failure rates), the difference in their performances is not terribly large.

Embedded image coding is a compression technique that yields bit streams robust to truncation. The first embedded wavelet-based compression algorithms were designed to optimize for mean-squared-error (MSE). More recent algorithms incorporate a variety of perceptually-based optimization criteria, which are implemented by selecting subband quantization step-sizes based on a target quality (and therefore rate). Since these step-sizes vary with rate, a bit stream optimized for a specific rate, if truncated, does not necessarily reflect the perceptual quality achievable at this lower rate. Constraints are given under which a wavelet-based perceptual model can generate an embedded image representation yielding perceptually derived results over a range of rates. This kind of embedding is implemented by strategically selecting the order in which subband bit planes are coded. When truncated, the resulting embedded streams provide perceptually tuned images.

Recently, image compression systems based on the tarp filter, a type of recursive filter, have attracted much attention in the image processing community. While providing very good performance when used in a non-embedded manner, the original tarp-filter-based algorithm performs less competitively when used in an embedded manner (in spite of its operation on bitplanes), because of its raster scan encoding order. We propose a Tarp-filter-based system which utilizes Classification of coefficients to achieve Embedding (TCE). The algorithm classifies the coefficients according to their statistical properties, and the tarp filter only runs on the single class on which it tends to generate accurate probability estimates. TCE can achieve much better rate-distortion embedding performance than the original tarp-filter-based system when used in an embedded manner; it achieves slightly better performance than SPIHT with arithmetic coding, and is comparable with JPEG-2000 performance on average.

When multiple channels are aggregated into a higher bandwidth logical channel, strategic product codes can be used to protect data against both channel losses and failures. Previous work in (W. Xu et al., Proc. ICME, vol.1, p.285-288, 2003) has considered a delay-optimal partitioning scheme given pre-determined error correction codes (ECCs). In order to achieve an improved distortion performance, image bit streams can be partitioned together with an appropriate selection of ECCs based on channel conditions. This paper addresses the problem of jointly partitioning image data over multiple channels and selecting product coding rates from a finite set to minimize the decoding distortion under delay constraints. Experimental results show that the image quality degrades very slightly when the delay constraint decreases over a wide range, with a gain up to 4.5 dB achieved over that using a sequential optimization method.

We introduce a special class of multiple description scalar quantizer (MDSQ), which is referred to as the modified MDSQ (MMDSQ). MMDSQ features simple implementations and an efficient central-side distortion trade-off control mechanism. These advantages make it suitable for multiple description image and video coding systems: the simplicity of MMDSQ facilitates the incorporation of multiple description coding into existing coding systems. Analysis shows that this class of MDSQ can achieve the same asymptotic high-rate performance as entropy-constrained MDSQ with uniform central quantizers, for sources with smooth pdf. The performance of MMDSQ on a unit-variance Gaussian i.i.d. source is compared with that of entropy-constrained MDSQ at different rates numerically, and the results confirm the satisfactory performance of MMDSQ at rates above 3 bps/description. Application of MMDSQ to image coding also shows promising results.

This paper presents a Rate-Distortion based optimization procedure for video coding that explicitly incorporates a perceptual distortion measure. A flexible MINMAX (minimizing the maximum distortion) optimization procedure based on Linear Programming (LP) is used to minimize the variation in distortion across frames. The distortion measure utilizes a multi-channel human vision model to generate a map of the perceived distortion at each pixel in a frame. This map is then collapsed into a single frame-level distortion. Lagrangian optimization is employed within each frame to assign macroblock-level quantization step-sizes that minimize this frame-level distortion. Coding results with MPEG-2 show a large reduction in peak perceived distortion with respect to TM5 as well as lower variation in distortion across the frames in the video sequence. Although results are presented for a specific metric, the LP based optimization procedure easily extends to other distortion measures.

This paper introduces a new high-rate analysis of the multiple description scalar quantizer (MDSQ) with balanced descriptions. The analysis provides insight into the structure of the MDSQ, suggesting the nonoptimality of uniform central quantizer cell lengths, as well as a method to approximate optimal cell lengths. For both level-constrained and entropy-constrained MDSQ, new upper bounds on the granular distortion for sources with smooth probability density functions (pdfs) are derived under the mean-squared error measure, which are 0.4 dB lower than previous results. Based on the insights, a universal multiple description scalar quantizer (UMDSQ) is proposed which, at high rate, can achieve nearly the same performance as the fully optimized entropy-constrained MDSQ (ECMDSQ), without requiring extensive training. The proposed UMDSQ has only two control parameters, and a continuum of tradeoff points between the central and side distortions can be achieved as the two parameters are varied.

The asymptotic analysis of multiple-description vector quantization (MDVQ) with a lattice codebook for sources with smooth probability density functions (pdfs) is considered in this correspondence. Goyal et al. (2002) observed that as the side distortion decreases and the central distortion correspondingly increases, the quantizer cells farther away from the coarse lattice points shrink in a spatially periodic pattern. In this correspondence, two special classes of index assignments are used along strategic groupings of central quantizer cells to derive a straightforward asymptotic analysis, which provides an analytical explanation for the aforementioned observation. MDVQ with a lattice codebook was shown earlier to be asymptotically optimal in high dimensions, with a curious converging property, that the side quantizers achieve the space filling advantage of an n-dimensional sphere instead of an n-dimensional optimal polytope. The analysis presented here explains this behavior readily. While central quantizer cells on a uniform lattice are asymptotically optimal in high dimensions, the present authors have shown that by using nonuniform rather than uniform central quantizer cells, the central-side distortion product in an MDSQ can be reduced by 0.4 dB at asymptotically high rate. The asymptotic analysis derived here partially unifies these previous results in the same framework, though a complete characterization is still beyond reach.

A linear programming (LP) technique that performs rate-distortion based optimization for constrained video coding is introduced. Given the assumption of piecewise linear and convex rate-distortion functions, LP is used to generate optimal solutions for the minmax problem under VBR buffer constraints. The LP solution for this problem avoids an explicit search for the maximum distortion value. To verify the linearity and convexity assumptions, this minmax optimization procedure is applied to the MPEG-2 framework with macroblock level rate control under VRB buffer constraints. The results show an improvement in average PSNR of approximately 0.5 dB to 2.0 dB over the initial TM5 solution, while maintaining an almost constant PSNR level. As a second LP application, a minimum rate problem with constraints on linear combinations of frame-level distortions is considered. Using the previous assumptions on the rate-distortion functions, it is proven that the LP solution is optimal. The LP technique yields a computationally efficient solution for this filtered distortion problem.

A new high-rate analysis of the entropy-constrained multiple description scalar quantizer (ECMDSQ) is introduced. The analysis provides insight into the structure of the ECMDSQ, suggesting the non-optimality of uniform central quantizer cells with finite diagonals in the index assignment matrix, as well as a method to approximate optimal cell sizes. Based on these insights, a universal multiple description scalar quantizer (UMDSQ) is proposed which can achieve nearly the same performance as the fully optimized ECMDSQ, at much lower design complexity. The design requires selection of only two parameters, and the resulting UMDSQ can provide a continuum of trade-off points between the central and side distortions as the two parameters are varied.

An efficient method for rate control is illustrated for joint source-channel rate selection and for encoding under multiple leaky bucket buffer constraints. This method shows advantages over existing methods in that it has polynomial time complexity and handles a class of rate constraints that include common constraints such as the leaky bucket and sliding window constraints. This technique is based on a linear relaxation of rate constraints and a piecewise linear approximation of the frame-level distortion-rate functions. Using linear programming, optimal solutions to the approximated versions of the problems are guaranteed in polynomial time. Moreover, bounds are provided for the deviation of the value of this solution from that of the optimal solution to the initial problem.

A double stimulus subjective evaluation was performed to determine preferred frame rates at a fixed bit rate for low bit rate video. Stimuli consisted of eight reference color video sequences of size 352/spl times/240 pixels. These were compressed at rates of 100, 200 and 300 kbps for low, medium, and high motion sequences, respectively, using three encoders and frame rates of 10, 15 and 30 frames per second. Twenty-two viewers ranked their frame rate preferences using an adjectival categorical scale. Their preferences were analyzed across sequence content, motion type, and encoder. Viewers preferred a frame rate of 15 frames per second across all categories, with several notable content-based exceptions.

Transmission of delay-sensitive images over wireless channels can be hampered by insufficient bandwidth and unreliable channels. These challenges can be overcome by aggregation of multiple channels into a single, higher bandwidth logical channel and by employing strategic error correction. This paper addresses the problem of partitioning image data over multiple channels with unequal characteristics to minimize delay, and the robustness to channel losses and failures is provided by product codes. Data is unequally error protected based on both decoding priorities and channel reliabilities. The proposed product codes outperform a conventional product code due to the adaptivity of the partitions to the channel conditions. Two schemes for product coding are considered and their relative merits are analyzed and compared.

Wavelet transform coefficients are defined by both a magnitude and a sign. While efficient algorithms exist for coding the transform coefficient magnitudes, current wavelet image coding algorithms are not as efficient at coding the sign of the transform coefficients. It is generally assumed that there is no compression gain to be obtained from entropy coding of the sign. Only recently have some authors begun to investigate this component of wavelet image coding. In this paper, sign coding is examined in detail in the context of an embedded wavelet image coder. In addition to using intraband wavelet coefficients in a sign coding context model, a projection technique is described that allows nonintraband wavelet coefficients to be incorporated into the context model. At the decoder, accumulated sign prediction statistics are also used to derive improved reconstruction estimates for zero-quantized coefficients. These techniques are shown to yield PSNR improvements averaging 0.3 dB, and are applicable to any genre of embedded wavelet image codec.

Reversible integer wavelet transforms are increasingly popular in lossless image compression, as evidenced by their use in the recently developed JPEG2000 image coding standard. In this paper, a projection-based technique is presented for decreasing the first-order entropy of transform coefficients and improving the lossless compression performance of reversible integer wavelet transforms. The projection technique is developed and used to predict a wavelet transform coefficient as a linear combination of other wavelet transform coefficients. It yields optimal fixed prediction steps for lifting-based wavelet transforms and unifies many wavelet-based lossless image compression results found in the literature. Additionally, the projection technique is used in an adaptive prediction scheme that varies the final prediction step of the lifting-based transform based on a modeling context. Compared to current fixed and adaptive lifting-based transforms, the projection technique produces improved reversible integer wavelet transforms with superior lossless compression performance. It also provides a generalized framework that explains and unifies many previous results in wavelet-based lossless image compression.

A visually-optimal quantization and rate-control strategy based on results of recent contrast sensitivity and suprathreshold summation experiments is proposed. At suprathreshold contrasts, masked detection thresholds for wavelet subband quantization distortions were approximately equal for scale-3, 4, and 5 distortions; approximately 52% greater for scale-2 distortions; and approximately 84% greater for scale-1 distortions. Based on a suprathreshold error-pooling model, contrasts for individual subbands are selected to match these contrast ratios, and are adjusted to account for changes in relative sensitivity at suprathreshold contrasts. Quantization step sizes are then computed from the adjusted base contrasts. A target contrast is estimated from the desired rate, and rate control is performed by adjusting this contrast until the rate is met. Images compressed with the proposed algorithm show improved visual quality at low bit rates.

The paper presents an objective metric for the perceived quality of low bit rate compressed video. Frames are decomposed into several frequency bands at multiple scales and a motion detection algorithm is used to segment moving and nonmoving regions. The distance between the decompositions of the reference and distorted video sequences is used to estimate a frame distortion. The pooled error across several frames and the correlation between successive estimates of perceived quality are used to estimate the current perceived quality at a rate of 2 samples per second. The results of a subjective evaluation of perceived quality are used to determine the model parameters. The variety of content and coding conditions used in the subjective evaluation permits a wide range of training conditions. The results indicate that the metric can capture many of the features of the measured perceived quality over time.

Reliable transmission of video over wireless networks must address the limited bandwidth and the possibility of loss. When the bandwidth is insufficient on a single channel, the video can be partitioned over multiple channels with possibly unequal characteristics at the expense of more complex channel coding (error correction). This paper addresses the problem of efficiently partitioning forward error protected, pre-encoded video data for transmission over multiple channels. The assumption of pre-encoding precludes adjustment of source rates to the channels, since it is assumed that channel characteristics are not known until immediately prior to the start of transmission. The proposed partitioning exploits the structure of MPEG video, and frames in each group-of-picture are reordered based on their decoding dependence. To be spectrally efficient, the frames of different types are unequally error protected taking different channel reliabilities into account. A pruned tree search algorithm is implemented to efficiently solve the problem. Simulation results are presented.

A subjective quality evaluation was performed to quantify viewer responses to various visual defects that appear in low bit rate video, both at full and reduced frame rates. The stimuli were eight video sequences compressed by three encoders operating at five bit rate/frame rate combinations. The sequences covered a broad range of typical streaming video content. Viewers continuously registered their responses to the stimuli as per the SSCQE method of ITU-R Recommendation BT.500-8. Data from 19 viewers was collected. An analysis of particularly disturbing artifacts is performed. The effects of changes in frame rate at a fixed bitrate on perceived quality are studied, demonstrating a preference for lower frame rates. A comparison of PSNR to the perceived quality ratings is made, revealing several notable trends.

An experiment quantifying human sensitivity to supra-threshold compression artifacts in wavelet-compressed images allows a computation and comparison of mean-squared errors producing equivalent visual distortion, and demonstrates that common assumptions used in wavelet image compression do not necessarily hold for low-rate images. The resulting MSE in the image is constant for a spatial frequency and is independent of orientation. Because the quantizers are not operating in the granular region, however, equal MSE does not imply equal quantizer step sizes. Computed threshold elevations for the second and third visible degradations are predicted well by the constant MSE model. Supra-threshold distortions in complex stimuli do not therefore behave according to sub-threshold assumptions, which suggest that the quantizer step size should be constant for a spatial frequency.

Rate control is essential to ensure high-quality video transmission over asynchronous transfer mode (ATM) networks; both source and channel rates must not violate buffer or network traffic constraints in order to guarantee decoded video quality. To date, source and channel rate selection has been performed jointly because the rates are related through buffer and network constraints and therefore appear to be interdependent. It is shown that selection of these rates can be separated and the rate control problem is therefore simplified, allowing sequential source and channel rate selection without the need for iteration. A rate control algorithm is proposed which implements noniterative, separate sequential selection of source and channel rates in order to minimize the distortion variation between frames, subject to all buffer and network constraints. In particular, the leaky bucket is used as the traffic policing mechanism. Achieving a target rate for individual frames is facilitated by using a rate-distortion optimized video coder which allows coding to an exactly specified rate or distortion. Channel rate selection is performed following source rate selection with the goal of providing smooth traffic to the network, thereby allowing a high statistical multiplexing gain. Experiments demonstrate that the proposed noniterative rate control algorithm, when implemented using the optimized coder, achieves consistent video quality while providing smooth traffic to the network.

Wavelet transform coefficients are defined by both a magnitude and a sign. While promising algorithms exist for efficiently coding the transform coefficient magnitudes, current wavelet image coding algorithms are not efficient at coding the sign of the transform coefficients. It is generally assumed that there is no compression gain to be obtained from entropy coding of the sign. Only recently have some authors begun to investigate this component of wavelet image coding. In this paper, sign coding is examined in detail in the context of an embedded wavelet image coder. It is shown that PSNR improvements of up to 0.7 dB are possible from an efficient modeling and entropy coding of the coefficient signs, combined with a new extrapolation technique, which is used to improve the final estimate of insignificant coefficients. These sign coding techniques are applicable to any genre (e.g., zero-tree, context-model) of the embedded wavelet image coder.

Summary form only given. Rate control is essential to ensure high quality video transmission over asynchronous transfer mode networks: source and channel rates cannot violate buffer or network traffic constraints. To date, source and channel rate selection have been performed jointly because the rates appear to be interdependent. However these rates can be separated, allowing sequential source and channel rate selection without the need for iteration. This work considers source and channel rate selection for variable-bit-rate (VBR) video coding subject to both buffer and channel constraints (known as C-VBR) for real-time transmission over an ATM network with a leaky bucket (LB) traffic policing mechanism. Source rates are selected to minimize distortion variation (MDV) to achieve consistent video quality while channel rates are selected to submit smooth traffic to the network. It has been previously shown that VBR rate control can be simplified to constant-bit-rate (CBR) rate control using the concept of the virtual buffer (i.e., the encoder buffer combined with the LB). This work shows that a similar simplification can be made for MDV rate control by additionally allowing virtual buffer underflow.

A robust image transmission system incorporating perceptually-based coding and error protection is proposed for wireless channels. The perceptually-based image coder consists of a wavelet transformation, uniform quantization, and variable-length entropy encoding. The coded data is placed in a spatially scalable stream divided into segments of varying perceptual importance and tolerance to transmission errors. A novel unequal error protection system is designed using turbo codes for the wireless channel, neural networks to predict decoder error and calculate erasures, and outer turbo-code concatenation of Reed-Solomon codes at different rates. A model of error tolerance is developed to minimize the added redundancy required to meet the desired throughput performance and visual quality level. The proposed system provides excellent reliability allowing the transmission of high visual quality images at a small additional redundancy and low signal-to-noise ratio (4.5 dB).

Many state-of-the-art wavelet image coders use nonorthogonal transforms for both lossy and lossless wavelet image coding. In this paper, a projection prediction is described that capitalizes on the non-orthogonality of wavelet transform basis vectors to improve the prediction of high-frequency coefficients. For lossy wavelet coders, the prediction yields improved context modeling and up to .2 dB coding improvement. For lossless integer wavelet coders, the prediction can be used as an extra lifting step during the transform, and results in a lower first-order entropy of wavelet coefficients and lower subsequent coding rate than current integer wavelet transforms.

With the proliferation of readily available image content, data hiding/watermarking has become a topic of considerable importance. In many applications, it is desirable to embed identifying data strings for authentication, or to embed contextual data that may be used for other purposes. It is also desirable that the image into which the data has been embedded be visually indistinguishable from the original. This paper presents a method in which psychovisual metrics are used in a wavelet based algorithm that can embed a set of bits into an image. The resulting image remains visually identical to the original. The algorithm can be used to embed very robust authentication data, or less robust data to be used for other purposes, where the robustness is a function of the number of bits to be embedded.

A psychovisually-based quantization strategy for wavelet coded images is proposed. The quantization strategy is based on the results of a psychovisual evaluation to quantify sensitivities to supra-threshold distortion caused by wavelet coefficient quantization in natural images. Quantizer step sizes to produce a minimum noticeable distortion are parameterized in terms of subband standard deviation and a quantization strategy is subsequently proposed. The resulting strategy is applied to both intraband and embedded wavelet coders and reveals a significant perceptual improvement when images exhibit supra-threshold distortion.

Resynchronizing variable-length codes (RVLCs) for large alphabets are designed by first creating resynchronizing Huffman codes and then adding an extended synchronizing codeword, and the RVLCs are applied to both JPEG and wavelet-based image compression. The RVLCs demonstrate the desired resynchronization properties, both at a symbol level and structurally so that decoded data can be correctly placed within an image following errors. The encoded images, when subject to both structural and statistical error detection and concealment, can tolerate BERs of up to 10/sup -4/ and are very tolerant of burst errors. The RVLC-JPEG images have negligible overhead at visually lossless bit rates, while the RVLC-wavelet overhead can be adjusted based on the desired tolerance to burst errors and typically ranges from 7 to 18%. The tolerance to both bit and burst errors demonstrates that images coded with such RVLCs can be transmitted over imperfect channels suffering bit errors or packet losses without channel coding for the image data, or with less channel coding than would be required if the encoded image data could tolerate no bit errors. While the overhead is nontrivial for the RVLC-wavelet images and the lower-rate RVLC-JPEG images, the encoded bitstreams do not have the firm restrictions on numbers or spacings of bit errors that some error correcting codes have, and hence provide more graceful degradation.

This paper addresses jointly rate-distortion optimal selection of coding parameters in a general motion-compensated video coder. The general coder uses variable-block-size motion estimation and multimode residual coding. This is essentially the optimal bit-allocation problem for an individual frame at a given rate constraint. This paper not only gives the general formulation and solution using the Lagrange multiplier method and dynamic programming, but also demonstrates how the general theory can he adapted and applied to both an MPEG-like coder and a motion-compensated wavelet coder. Simulations demonstrate that both proposed coders outperform MPEG (TM5) by 0.7-1.3 dB at a variety of bit rates, with the gain provided by both better motion estimation and the joint-parameter optimization. The technique is applicable to MPEG-compliant coders with fixed block-size motion estimation and provides a gain of 0.5-0.7 dB over TM5. The optimization approach can also be applied to distortion-constrained coding, and therefore allows a fine tuning of either the rate or distortion to follow any desired profile.

Summary form only given. This paper considers instantaneous prefix codes with possibly unequal codeword lengths but specific codewords, such as a Huffman code. Resynchronizing VLC (RVLC) contain one or more synchronizing codewords that resynchronize the decoder regardless of any previous data. Previous applications of optimal resynchronizing VLC have been limited to sources with alphabets of sizes less than 30, while only non-optimal VLC with ad-hoc marker codewords have been applied to image data. This paper adapts a general design algorithm for optimal resynchronizing VLC to JPEG image data, demonstrating its applicability to sources with large alphabets (>1000). In order to ensure that the decoded data is placed properly in the image following resynchronization, the resulting VLC are modified to contain extended synchronizing codewords to serve as markers. Minor modifications to the baseline JPEG coder increase the robustness to errors, and a concealment algorithm locates and repairs errant data. Images coded using the RVLC-JPEG combined with the concealment algorithm are robust to BER as high as 2/spl times/10/sup -4/ and are extremely tolerant of burst errors. The excellent tolerance to both bit and burst errors at high bit rates demonstrates that images coded with such RVLC can be transmitted over imperfect channels suffering bit errors or packet losses without channel coding for the image data. At lower bit rates, while the overhead is non-trivial, the encoded bitstream does not have firm restrictions on numbers or spacings of errors and hence provides more graceful degradation than traditional ECC.

A temporally scalable video coding algorithm allows extraction of video of multiple frame rates from a single coded stream. This paper compares the rate-distortion performance of three temporally scalable coding techniques: temporal subband coding (TSB), motion-compensated temporal subband coding (MC-TSB), and motion compensated prediction (MCP). The distortions are derived in terms of various temporal pixel correlations and the probability of occlusion. Measured values are used to predict performances at full and lower frame rates, and a comparison with experimental quantitative performances demonstrates that the performance ranking is correctly predicted and bounds the accuracy of the analyses at high rates. MCP and MC-TSB always outperform TSB. For high bit rate full-frame-rate video, MCP and MC-TSB perform equivalently. At lower frame rates, MCP provides better performance.

This paper proposes a frame rate-control scheme with a MINMAX distortion criterion in the framework of a rate-distortion optimized motion compensated embedded wavelet coder. The MINMAX criterion aims to minimize the maximum distortion in a group of pictures (GOP) for a given bit rate. The proposed frame level bit allocation scheme combined with R-D optimized within frame allocation allows precise rate control up to the exact bit. A simplifying fast algorithm is developed using an initial prediction of the operating distortion and adaptive adjustment during GOP coding. Experiments demonstrate that as much as 1.1 dB gain over MPEG-2 TM5 and 0.9 dB over H263 TMN5 can be achieved as well as more consistent visual quality.

A temporally scalable video coding algorithm allows extraction of video of multiple frame rates from a single coded stream. In recent years, several video coding techniques have been proposed that provide temporal scalability using subband coding, both without and with motion compensation. With a two-band subband decomposition applied hierarchically, frame rates halve after each filtering operation. Alternatively, motion-compensated prediction (as used in MPEG) can provide temporal scalability and the same frame rates as temporal subband coding through strategic placement of reference frames and selective decoding of frames. This paper compares three temporal coding techniques with respect to providing temporal scalability: temporal subband coding (TSB), motion-compensated temporal subband coding (MC-TSB), and motion-compensated prediction (MCP). Predicted rate-distortion performances at full- and lower frame rates and experimental quantitative and visual performances from coding several video sequences are compared. The comparison is explicitly for temporal coding when the dimensionality of the subsequent source coding is held constant; any spatial or higher dimensional source coding can follow. In theory and in practice, MCP and MC-TSB always outperform TSB. For high-bit-rate full-frame-rate video, the performances of MCP and MC-TSB are approximately equivalent. However, to provide temporal scalability, MCP clearly provides the best performance in terms of visual quality, quantitative quality, and bit rate of the lower frame-rate video.

Assumptions about image continuity lead to oversmoothed edges in common image interpolation algorithms. A wavelet-based interpolation method that imposes no continuity constraints is introduced. The algorithm estimates the regularity of edges by measuring the decay of wavelet transform coefficients across scales and preserves the underlying regularity by extrapolating a new subband to be used in image resynthesis. The algorithm produces visibly sharper edges than traditional techniques and exhibits an average peak signal-to-noise ratio (PSNR) improvement of 2.5 dB over bilinear and bicubic techniques.

Uniform quantization of wavelet coefficients introduces perceptually disturbing artifacts by decreasing the visual smoothness. In this paper, the local Holder regularity is used to quantify these changes in visual smoothness. A modified uniform quantization scheme that constrains local regularity reduction is proposed. The quantizer modification is applicable to any uniform quantization scheme and requires no additional side information to be transmitted to the decoder, unlike standard deadzone quantizers. The reconstructed images show a noticeable perceptual improvement as well as an increased PSNR relative to uniform quantization.

Spatial multi-resolution video sequences provide video at multiple frame sizes, allowing extraction of only the resolution or bit rate required by the user. This paper proposes fine-to-coarse motion estimation (ME) for multi-resolution video coding. While coarse-to-fine ME, used in previously proposed coding themes, can provide a better estimate at the coarsest resolution, it is outperformed by fine-to-coarse ME at finer resolutions due to the inability of coarse-to-fine ME to accurately track motion at finer resolutions. At the finest resolution, fine-to-coarse ME provides a PSNR improvement of up to 1 dB, for the sequences tested, and better visual quality at all resolutions. In addition, fine-to-coarse ME provides more accurate and thus more compressible motion estimates.

Common image interpolation methods assume that the underlying signal is continuous and may require that it possess one or more continuous derivatives. These assumptions are not generally true of natural images, most of which have instantaneous luminance transitions at the boundaries between objects. Continuity requirements on the interpolating function produce interpolated images with oversmoothed edges. To avoid this effect, a wavelet-based interpolation method that imposes no continuity constraints is introduced. The algorithm estimates the regularity of edges by measuring the decay of wavelet transform coefficients across scales and attempts to preserve the underlying regularity by extrapolating a new subband to be used in image resynthesis. The algorithm produces noticeably sharper edges than traditional techniques and exhibits an average PSNR improvement of 2.5 dB over bilinear and bicubic techniques.

Two temporally scalable video coding techniques, temporal subband coding (TSB) and predictive coding, are evaluated both theoretically and in practice to provide comparisons of compression and visual quality at differing frame rates. Results demonstrate that TSB coding has a higher coding gain at full frame rate than predictive coding if both algorithms use either no motion compensation or motion compensation (MC) that can be modeled as an invertible pre-distortion of the video sequence. However, predictive coding outperforms TSB coding at full frame rate when both schemes use block-based MC. In addition, visual results for lower-frame-rate video using TSB coding are unacceptable due to significant distortions from temporal filtering. These results are demonstrated through both theoretical evaluation of rate-distortion based coding gains and simulated coding of real video sequences.

Psychophysical studies have shown that there are three image components of distinct perceptual significance to human observers: strong edges, smooth regions, and textured or detailed regions. A psychophysical test was performed to evaluate the perceptual role of each region and a segmentation algorithm was developed to segment an image into the three regions. The segmentation algorithm identifies blocks of an image as belonging to one of the perceptual regions by analyzing high frequency coefficients of a 3-level hierarchical subband/wavelet decomposition. The segmentation algorithm performs well on a wide range of image content, including natural images and mixed images containing text, graphics, and natural scenes.

This paper proposes a video encoder with improved visual quality achieved through more accurate motion estimation and perceptual quantization. A quadtree-based variable size block motion estimation scheme jointly optimizes both motion vector coding and residual coding by minimizing the actual rate subject to a constraint on overall distortion. The residual coder includes perceptual quantization and direct block coding. Since the residual is encoded directly without any transform, the encoder achieves comparable speed with MPEG. The encoded video outperforms that of MPEG in terms of both PSNR and visual quality.

Transmission of digital subband-coded images over lossy packet networks presents a reconstruction problem at the decoder. This paper presents two techniques for reconstruction of lost subband coefficients, one for low-frequency coefficients and one for high-frequency coefficients. The low-frequency reconstruction algorithm is based on inherent properties of the hierarchical subband decomposition. To maintain smoothness and exploit the high intraband correlation, a cubic interpolative surface is fit to known coefficients to interpolate lost coefficients. Accurate edge placement, crucial for visual quality, is achieved by adapting the interpolation grid in both the horizontal and vertical directions as determined by the edges present. An edge model is used to characterize the adaptation, and a quantitative analysis of this model demonstrates that edges can be identified by simply examining the high-frequency bands, without requiring any additional processing of the low-frequency band. High-frequency reconstruction is performed using linear interpolation, which provides good visual performance as well as maintains properties required for edge placement in the low-frequency reconstruction algorithm. The complete algorithm performs well on loss of single coefficients, vectors, and small blocks, and is therefore applicable to a variety of source coding techniques.

Packetized image communication over lossy channels presents a reconstruction problem at the decoder. To date, reconstruction algorithms have been developed for fixed coding techniques. This paper examines the dual problem-a block-based coding technique, namely a family of lapped orthogonal transforms (LOTs), is designed to maximize the reconstruction performance of a fixed reconstruction algorithm. Mean-reconstruction, in which missing coefficient blocks are replaced with averages of their available neighbors, is selected for simplicity and implementation ease. A reconstruction criterion is defined in terms of mean-squared error, and is used to define the reconstruction gain. A family of LOTs is then designed to consider both reconstruction gain and coding gain. Reconstruction capability and compression are exchanged, and the LOT family consists of transforms that provide increasing reconstruction capability with lower coding gain. A transform can therefore be selected based on channel loss characteristics, desired reconstruction performance, and desired compression.

The uniform quantizers used in typical wavelet image compression algorithms can introduce perceptually disturbing artifacts by decreasing visual smoothness. In this paper, we use local Holder regularity to quantify changes in visual smoothness and propose a modified quantization scheme which constrains local regularity decreases. The reconstructed images show a significant perceptual improvement as well as an increased PSNR.

Wireless transmission of compressed visual information presents new challenges in image coding and reconstruction techniques. Wireless channels do not offer guaranteed transmission, and data loss over such channels can result in catastrophic errors in the decoded visual information. Visual data, however, can be reconstructed using lossy signal processing techniques. To date, reconstruction algorithms have been developed for fixed coding techniques. This paper presents and solves the dual problem-a block-based coding technique, namely a family of lapped orthogonal transforms (LOTs) is designed to maximize the reconstruction performance of a specified reconstruction algorithm. Mean-reconstruction, in which a missing coefficient block is replaced with the average of its available neighbors, is selected for its simplicity and ease of implementation. A reconstruction criterion is defined as the equal distribution of reconstruction errors across all transform coefficients, and a family of LOTs is then designed to meet the reconstruction criterion as well as consider the transform coding gain. Reconstruction capability and coding gain are traded off, and the LOT family consists of transforms that provide increasing reconstruction capability with lower coding gain. The reconstruction-optimized LOT family provides excellent reconstruction capability, and a transform can be selected based on the loss characteristics of the channel, the desired reconstruction performance, and the desired compression.

Packet-based transmission of subband coded images over lossy networks presents a reconstruction problem at the decoder. Accurate reconstruction of the high-energy low frequency subband coefficients is imperative in providing consumer-grade image quality. This paper introduces a family of one-dimensional quadrature mirror filters (QMFs) designed to minimize the mean-squared error of reconstructed low frequency coefficients for a given reconstruction algorithm to be implemented at the decoder. Mean-reconstruction, in which a missing coefficient is replaced with the average of its neighbors either horizontally or vertically, is selected for its simplicity and implementation ease. The resulting filters perform well as QMFs and provide the desired reconstruction properties in the event of loss. While the filters are developed using mean-reconstruction the filter design algorithm can be used with more sophisticated reconstruction techniques, providing that the mean-squared error can be expressed in the appropriate quadratic form.

Transmission of still images and video over lossy packet networks presents a reconstruction problem at the decoder. Specifically, in the case of block-based transform coded images, loss of one or more packets due to network congestion or transmission errors can result in errant or entirely lost blocks in the decoded image. This article proposes a computationally efficient technique for reconstruction of lost transform coefficients at the decoder that takes advantage of the correlation between transformed blocks of the image. Lost coefficients are linearly interpolated from the same coefficients in adjacent blocks subject to a squared edge error criterion, and the resulting reconstructed coefficients minimize blocking artifacts in the image while providing visually pleasing reconstructions. The required computational expense at the decoder per reconstructed block is less than 1.2 times a non-recursive DCT, and as such this technique is useful for low power, low complexity applications that require good visual performance.

A reconstruction technique for block-based transform-coded images transmitted over lossy packet networks is proposed and demonstrated. Lost blocks of transform coefficients are reconstructed via linear interpolation using the same coefficients from adjacent blocks, thus preserving high frequency details and local image characteristics. Determination of the interpolation coefficients, or weights, is combined with vector quantization in a single step at the encoder, and the resulting quantized weights are transmitted as overhead information. The required transmission overhead is less than 10% for typical JPEG-coded images, and the computation for reconstruction of a block is less than that required for simply decoding a block using an inverse discrete cosine transform. This technique is applicable to both luminance and chrominance components, and the reconstructed images maintain good visual quality at loss rates as high as 30%.>