This paper considers the entropy of highly correlated quantized samples. Two results are shown. The first concerns sampling and identically scalar quantizing a stationary continuous-time random process over a finite interval. It is shown that if the process crosses a quantization threshold with positive probability, then the joint entropy of the quantized samples tends to infinity as the sampling rate goes to infinity. The second result provides an upper bound to the rate at which the joint entropy tends to infinity, in the case of an infinite-level uniform threshold scalar quantizer and a stationary Gaussian random process. Specifically, an asymptotic formula for the conditional entropy of one quantized sample conditioned on the previous quantized sample is derived. At high sampling rates, these results indicate a sharp contrast between the large encoding rate (in bits/sec) required by a lossy source code consisting of a fixed scalar quantizer and an ideal, sampling-rate-adapted lossless code, and the bounded encoding rate required by an ideal lossy source code operating at the same distortion.

This paper considers basic questions regarding Markov random processes. It shows that continuous-time, continuous-valued, wide-sense stationary, Markov processes that have absolutely continuous second-order distribution and finite second moment are not bandlimited. It also shows that continuous-time, stationary, Markov processes that are continuous-valued or discrete-valued and satisfy additional mild conditions cannot be recovered from uniform sampling. Further it shows that continuous-time, continuous-valued, stationary, Markov processes that have absolutely continuous second-order distributions and are continuous almost surely, cannot be recovered without error after quantization. Finally, it provides necessary and sufficient conditions for stationary, discrete-time, Markov processes to have zero entropy rate, and relates this to information singularity.

This paper considers the problem, first introduced by Ahlswede and Korner in 1975, of lossless source coding with coded side information. Specifically, letXandYbe two random variables such thatXis desired losslessly at the decoder whileYserves as side information. The random variables are encoded independently, and both descriptions are used by the decoder to reconstructX. Ahlswede and Korner describe the achievable rate region in terms of an auxiliary random variable. This paper gives a partial solution for an optimal auxiliary random variable, thereby describing part of the rate region explicitly in terms of the distribution ofXandY.

Markov random processes and general random processes are considered. It is shown that continuous-time, continuous-valued, wide-sense stationary, Markov random processes that have absolutely continuous second order distributions are not bandlimited. It is also shown that when these processes are strictly stationary and continuous almost surely, they cannot be recovered without error from their quantized versions. Further, it is shown that continuous-time, discrete-valued Markov random processes, which are uniformly bounded and satisfy an additional condition, can be recovered with zero average distortion from an appropriate set of samples for a general distortion measure. A similar result is shown for general continuous-time random processes with rthpower distortion measure. Additionally, it is shown that under a milder condition on the Markov processes and a different condition on the sampling times (e.g., uniform sampling), such processes cannot be recovered with zero average distortion. Finally, the notion of information-singularity is extended to continuous-time random processes, and it is shown that both continuous- and discrete-time Markov processes are not information-singular.

This correspondence considers low-resolution scalar quantization for a memoryless Gaussian source with respect to absolute error distortion. It shows that slope of the operational rate-distortion function of scalar quantization is infinite at the point Dmaxwhere the rate becomes zero. Thus, unlike the situation for squared error distortion, or for Laplacian and exponential sources with squared or absolute error distortion, for a Gaussian source and absolute error, scalar quantization at low rates is far from the Shannon rate-distortion function, i.e., far from the performance of the best lossy coding technique

This paper considers low resolution scalar quantization for a memoryless Gaussian source with respect to absolute error distortion. It shows that slope of the operational rate-distortion function of scalar quantization is infinite at the point Dmaxwhere the rate becomes zero. Thus, unlike the situation for squared error distortion, or for Laplacian and exponential sources with squared or absolute error distortion, for a Gaussian source and absolute error, scalar quantization at low rates is far from the Shannon rate-distortion function, i.e. far from the performance of the best lossy coding technique

This paper considers the problem, first introduced by Ahlswede and Körner in 1975, of lossless source coding with coded side information. Specifically, let X and Y be two random variables such that X is desired losslessly at the decoder while Y serves as side information. The random variables are encoded independently, and both descriptions are used by the decoder to reconstruct X. Ahlswede and Körner describe the achievable rate region in terms of an auxiliary random variable. This paper gives a partial solution for the optimal auxiliary random variable, thereby describing part of the rate region explicitly in terms of the distribution of X and Y.

This correspondence analyzes the low-resolution performance of entropy-constrained scalar quantization. It focuses mostly on Gaussian sources, for which it is shown that for both binary quantizers and infinite-level uniform threshold quantizers, as D approaches the source variance /spl sigma//sup 2/, the least entropy of such quantizers with mean-squared error D or less approaches zero with slope -log/sub 2/e/2/spl sigma//sup 2/. As the Shannon rate-distortion function approaches zero with the same slope, this shows that in the low-resolution region, scalar quantization with entropy coding is asymptotically as good as any coding technique.

This paper considers the entropy of the highly correlated quantized samples resulting from sampling at high rate. Two results are shown. The first concerns sampling and identically scalar quantizing a stationary random process over a finite interval. It is shown that if the process crosses a quantization threshold with positive probability, then the joint entropy of the quantized samples tends to infinity as the sampling interval goes to zero. The second result provides an upper bound to the rate at which the joint entropy tends to infinity, in the case of infinite-level uniform threshold scalar quantizers and a stationary Gaussian random process whose mean lies at a midpoint of some quantization cell. Specifically, an asymptotic formula for the conditional entropy of one quantized sample conditioned on another quantized sample is derived

A uniform scalar quantizer with small step size, large support, and midpoint reconstruction levels is frequently modeled as adding orthogonal noise to the quantizer input. This paper rigorously demonstrates the asymptotic validity of this model when the input probability density function (pdf) is continuous and satisfies several other mild conditions. Specifically, as step size decreases, the correlation between input and quantization error becomes negligible relative to the mean-squared error (MSE). The model is even valid when the input density is discontinuous at the origin, but discontinuities elsewhere can prevent the correlation from being negligible. Though this invalidates the additive model, an asymptotic formula for the correlation is found in terms of the step size and the heights and positions of the discontinuities. For a finite support input density, such as uniform, it is shown that the support of the uniform quantizer can be matched to that of the density in ways that make the correlation approach a variety of limits. The derivations in this paper are based on an analysis of the asymptotic convergence of cell centroids to cell midpoints. This convergence is fast enough that the centroids and midpoints induce the same asymptotic MSE, but not fast enough to induce the same correlations.

The tradeoff between reliability and efficiency in distributed source coding for field-gathering sensor networks is examined. In the considered networks, sensors measure some underlying random field, quantize their measurements, encode the quantized values into bits and transmit these directly, or via relays, to a collector that reconstructs the field. The bits from one sensor's encoder are regarded as a packet. The minimum achievable coding rate can be attained if the sensors are ordered and each applies Slepian-Wolf distributed coding to its data assuming the decoder knows the data from all prior sensors. However, with such a coding scheme, losing even one sensor's packet would cause decoding failure for all subsequent sensors' values. Therefore, one might consider other ways of applying Slepian-Wolf coding, where in trade for increased coding rate, fewer sensor values are lost when a packet is lost. In this paper, the tradeoff between efficiency, i.e. coding rate, and reliability, characterized by a loss factor, is considered for several different Slepian-Wolf based coding schemes as a function of the packet error probability and the size of the network.

The operational rate-distortion function of entropy-constrained scalar quantizers in the asymptotic low resolution regime, for memoryless Gaussian sources, is investigated. It is shown that asymptotically, as distortion tends to variance, or equivalently as rate tends to zero, the operational rate-distortion function matches the Shannon rate-distortion function. This implies that scalar quantization is asymptotically optimal, a fact not previously known.

In this paper, we ignore transmission issues and focus on the total number of bits to transmit to the collector to form a reconstruction of the field with a given MSE. We assume that all sensors can transmit bits to the collector without error. With this assumption, with total number of bits as the cost measure, and with the style of coding, it can be argued that sensor-to-sensor relaying offers no advantages. This problem is similar to image coding and transmission, except that the quantization, encoding and transmission are constrained to take place separately at each sensor (pixel location), in contrast to traditional image coding and transmission, wherein the entire image is available for quantization, encoding, and transmission. Due to the need to separately encode values from separate sensors, we pursue a Slepian-Wolf style coding approach.