This paper presents a new achievable rate-distortion region for the general L channel multiple descriptions (MDs) problem. A well-known general region for this problem is due to Venkataramani, Kramer, and Goyal (VKG). Their encoding scheme is an extension of the El Gamal-Cover (EC) and Zhang-Berger (ZB) coding schemes to the L channel case and includes a combinatorial number of refinement codebooks, one for each subset of the descriptions. As in ZB, the scheme also allows for a single common codeword to be shared by all descriptions. This paper proposes a novel encoding technique involving combinatorial message sharing (CMS), where every subset of the descriptions may share a distinct common message. This introduces a combinatorial number of shared codebooks along with the refinement codebooks of. These shared codebooks provide a more flexible framework to tradeoff redundancy across the messages for resilience to descriptions loss. We derive an achievable rate-distortion region for the proposed technique, and show that it subsumes the VKG region for general sources and distortion measures. We further show that CMS provides a strict improvement of the achievable region for any source and distortion measures for which some two-description subset is such that ZB achieves points outside the EC region. We then show a more surprising result: CMS outperforms VKG for a general class of sources and distortion measures, including scenarios where the ZB and EC regions coincide for all two-description subsets. In particular, we show that CMS strictly improves on VKG, for the L-channel quadratic Gaussian MD problem, for all L ≥ 3, despite the fact that the EC region is complete for the corresponding two-descriptions problem. Consequently, the correlated quantization scheme (an extreme special case of VKG) that has been proven to be optimal for several cross sections of the L-channel quadratic Gaussian MD problem is strictly suboptimal in general. Using the encoding principles derived, we show that the CMS scheme achieves the complete rate-distortion region for several asymmetric cross sections of the L-channel quadratic Gaussian MD problem.

There has been considerable interest in distributed source coding (DSC) in recent years, primarily due to its potential contributions to low-power sensor networks. However, two major obstacles pose an existential threat to practical deployment of such techniques: the exponential growth of decoding complexity with network size and coding rates and the critical requirement of resilience to bit errors and erasures, given the severe channel conditions in many wireless sensor network applications. This article proposes a novel, unified approach for large-scale, error/erasure-resilient DSC that incorporates an optimally designed, nearest neighbor classifier-based decoding framework, where the design explicitly controls performance versus decoding complexity. Motivated by the highly nonconvex nature of the cost function, we present a deterministic annealing-based optimization algorithm for the joint design of the system parameters, which further enhances the performance over the greedy iterative descent technique. Simulation results on both synthetic and real sensor network data provide strong evidence for performance gains compared to other state-of-the-art techniques and may open the door to practical deployment of DSC in large sensor networks. Moreover, the framework provides a principled way to naturally scale to large networks while constraining decoder complexity, thereby enabling performance gains that increase with network size.

The hidden Markov model (HMM) is widely used to model processes in several real world applications, including speech processing and recognition, image understanding and sensor networks. A problem of concern is that of quantization of the sequence of observations generated by an HMM, which is referred as a hidden Markov source (HMS). Despite the importance of the problem, and the well-defined structure of the process, there has been very limited work addressing the optimal quantization of HMS, and conventional approaches focus on optimization of parameters of known quantization schemes. This paper proposes a method that directly tracks the state probability distribution of the underlying source and optimizes the encoder structure according to the estimated HMS status. Unlike existing approaches, no stationarity assumption is needed, and code parameters are updated on they: with each observation, both the encoder and the decoder refine the estimated probability distribution over the states. The main approach is then specialized to a practical variant involving switched quantizers, and an algorithm that iteratively optimizes the quantize codebooks is derived. Numerical results show superiority of the proposed approach over prior methods.

Scalable coders generate hierarchically layered bitstreams to serve content at different quality levels, wherein the base layer provides a coarse quality reconstruction and successive layers incrementally refine the quality. However, it is widely recognized that there is an inherent performance penalty due to the scalable coding structure, when compared to independently encoded copies. To mitigate this loss we propose a layered compression framework, having roots in information theoretic concepts, which relaxes the strict hierarchical constraints, wherein only a “subset” of the information of a lower quality level is shared with higher quality levels. In other words, there is flexibility to also have “private” information at each quality level, beside information that is common to multiple levels. We employ this framework within the MPEG Scalable AAC and propose an optimization scheme to jointly select parameters of all the layers. Experimental evaluation results demonstrate the utility of the flexibility provided by the proposed framework.

This paper studies the zero-delay source-channel coding problem, and specifically the problem of obtaining the vector transformations that optimally map between the m-dimensional source space and k-dimensional channel space, under a given transmission power constraint and for the mean square error distortion. The functional properties of the cost are studied and the necessary conditions for the optimality of the encoder and decoder mappings are derived. An optimization algorithm that imposes these conditions iteratively, in conjunction with the noisy channel relaxation method to mitigate poor local minima, is proposed. The numerical results show strict improvement over prior methods. The numerical approach is extended to the scenario of source-channel coding with decoder side information. The resulting encoding mappings are shown to be continuous relatives of, and in fact subsume as special case, the Wyner-Ziv mappings encountered in digital distributed source coding systems. A well-known result in information theory pertains to the linearity of optimal encoding and decoding mappings in the scalar Gaussian source and channel setting, at all channel signal-to-noise ratios (CSNRs). In this paper, the linearity of optimal coding, beyond the Gaussian source and channel, is considered and the necessary and sufficient condition for linearity of optimal mappings, given a noise (or source) distribution, and a specified a total power constraint are derived. It is shown that the Gaussian source-channel pair is unique in the sense that it is the only source-channel pair for which the optimal mappings are linear at more than one CSNR values. Moreover, the asymptotic linearity of optimal mappings is shown for low CSNR if the channel is Gaussian regardless of the source and, at the other extreme, for high CSNR if the source is Gaussian, regardless of the channel. The extension to the vector settings is also considered where besides the conditions inherited from the scalar case, additional constraints must be satisfied to ensure linearity of the optimal mappings.

The two most prevalent notions of common information (CI) are due to Wyner and Gács-Körner and both the notions can be stated as two different characteristic points in the lossless Gray-Wyner region. Although the information theoretic characterizations for these two CI quantities can be easily evaluated for random variables with infinite entropy (e.g., continuous random variables), their operational significance is applicable only to the lossless framework. The primary objective of this paper is to generalize these two CI notions to the lossy Gray-Wyner network, which hence extends the theoretical foundation to general sources and distortion measures. We begin by deriving a single letter characterization for the lossy generalization of Wyner's CI, defined as the minimum rate on the shared branch of the Gray-Wyner network, maintaining minimum sum transmit rate when the two decoders reconstruct the sources subject to individual distortion constraints. To demonstrate its use, we compute the CI of bivariate Gaussian random variables for the entire regime of distortions. We then similarly generalize Gács and Körner's definition to the lossy framework. The latter half of this paper focuses on studying the tradeoff between the total transmit rate and receive rate in the Gray-Wyner network. We show that this tradeoff yields a contour of points on the surface of the Gray-Wyner region, which passes through both the Wyner and Gács-Körner operating points, and thereby provides a unified framework to understand the different notions of CI. We further show that this tradeoff generalizes the two notions of CI to the excess sum transmit rate and receive rate regimes, respectively.

This paper considers the problem of minimum cost communication of correlated sources over a network with multiple sinks, which consists of distributed source coding followed by routing. We introduce a new routing paradigm called dispersive information routing (DIR), wherein the intermediate nodes are allowed to split a packet and forward subsets of the received bits on each of the forward paths. This paradigm opens up a rich class of research problems, which focus on the interplay between encoding and routing in a network. Unlike conventional routing methods such as in [1], DIR ensures that each sink receives just the information needed to reconstruct the sources it is required to reproduce. We demonstrate using simple examples that the proposed approach offers better asymptotic performance than conventional routing techniques. We show that, under certain assumptions on the cost function, the problem of finding the minimum cost under DIR essentially reduces to characterizing an achievable rate region for a new multiterminal information theoretic setup. While it is possible to derive an achievable region for this setup using prior results from general multiterminal source coding [3], these techniques do not exploit the underlying problem structure and thereby lead to suboptimal regions. In this paper, we propose a new coding scheme, using principles from multiple descriptions encoding [2], and show that it strictly improves upon a corresponding variant of coding scheme in [3]. We further show that the new coding scheme achieves the complete rate region for certain special cases of the general setup and thereby achieves the minimum communication cost under this routing paradigm.

In this paper, we study the zero-delay source-channel coding problem, and specifically the problem of obtaining the vector transformations that optimally map between the m-dimensional source space and the k-dimensional channel space, under a given transmission power constraint and for the mean square error distortion. We first study the functional properties of this problem and show that the objective is concave in the source and noise densities and convex in the density of the input to the channel. We then present the necessary conditions for optimality of the encoder and decoder mappings. A well known result in information theory pertains to the linearity of optimal encoding and decoding mappings in the scalar Gaussian source and channel setting, at all channel signal-to-noise ratios (CSNRs). In this paper, we study this result more generally, beyond the Gaussian source and channel, and derive the necessary and sufficient condition for linearity of optimal mappings, given a noise (or source) distribution, and a specified power constraint. We also prove that the Gaussian source-channel pair is unique in the sense that it is the only source-channel pair for which the optimal mappings are linear at more than one CSNR value. Moreover, we show the asymptotic linearity of optimal mappings for low CSNR if the channel is Gaussian regardless of the source and, at the other extreme, for high CSNR if the source is Gaussian, regardless of the channel.

This paper focuses on the problem of L-channel quadratic Gaussian multiple description (MD) coding. We recently introduced a new encoding scheme in [1] for general L-channel MD problem, based on a technique called `Combinatorial Message Sharing' (CMS), where every subset of the descriptions shares a distinct common message. The new achievable region subsumes the most well known region for the general problem, due to Venkataramani, Kramer and Goyal (VKG) [2]. Moreover, we showed in [3] that the new scheme provides a strict improvement of the achievable region for any source and distortion measures for which some 2-description subset is such that the Zhang and Berger (ZB) scheme achieves points outside the El-Gamal and Cover (EC) region. In this paper, we show a more surprising result: CMS outperforms VKG for a general class of sources and distortion measures, which includes scenarios where for all 2-description subsets, the ZB and EC regions coincide. In particular, we show that CMS strictly extends VKG region, for the L-channel quadratic Gaussian MD problem for all L ≥ 3, despite the fact that the EC region is complete for the corresponding 2-descriptions problem. Using the encoding principles derived, we show that the CMS scheme achieves the complete rate-distortion region for several asymmetric cross-sections of the L-channel quadratic Gaussian MD problem, which have not been considered earlier.

There has been considerable interest in distributed source coding within the compression and sensor network research communities in recent years, primarily due to its potential contributions to low-power sensor networks. However, two major obstacles pose an existential threat on practical deployment of such techniques in real world sensor networks, namely, the exponential growth of decoding complexity with network size and coding rates, and the critical requirement for error-resilience given the severe channel conditions in many wireless sensor networks. Motivated by these challenges, this paper proposes a novel, unified approach for large scale, error-resilient distributed source coding, based on an optimally designed classifier-based decoding framework, where the design explicitly controls the decoding complexity. We also present a deterministic annealing (DA) based global optimization algorithm for the design due to the highly non-convex nature of the cost function, which further enhances the performance over basic greedy iterative descent technique. Simulation results on data, both synthetic and from real sensor networks, provide strong evidence that the approach opens the door to practical deployment of distributed coding in large sensor networks. It not only yields substantial gains in terms of overall distortion, compared to other state-of-the-art techniques, but also demonstrates how its decoder naturally scales to large networks while constraining the complexity, thereby enabling performance gains that increase with network size.

This paper proposes a new multiple description (MD) coding method and an associated achievable rate-distortion region for L ≥ 2 channels. The proposed scheme randomly bins codebooks chosen from the codebook structure, similar to that of the recently proposed combinatorial message sharing (CMS) scheme designed for conditional codebook encoding. The proposed scheme effectively performs multilayer random binning for each subset of the description, which hence enables to utilize the symmetry of a “subset” of the description rates wherever it exists. The new scheme specializes in to the conventional multilayer random binning as an extreme special case.

The two most prevalent notions of common information are due to Wyner and Gács-Körner and both the notions can be stated as two different characteristic points in the lossless Gray-Wyner region. Although these quantities can be easily evaluated for random variables with infinite entropy (eg. continuous random variables), the operational significance underlying their definition is applicable only to the lossless framework. The primary objective of this paper is to generalize these two notions of common information to the lossy Gray-Wyner network, which extends the theoretical intuition underlying their definitions for general sources and distortion measures. We begin with the lossy generalization of Wyner's common information, defined as the minimum rate on the shared branch of the Gray-Wyner network at minimum sum rate when the two decoders reconstruct the sources subject to individual distortion constraints. We derive a complete single letter information theoretic characterization for this quantity and use it to compute the common information of symmetric bivariate Gaussian random variables. We then derive similar results to generalize Gács-Körner's definition to the lossy framework. These two characterizations allow us to carry the practical insight underlying the two notions of common information to general sources and distortion measures.

There are two common types of encoding paradigms in multiple descriptions (MD) coding: i) an approach based on conditional codebook generation, which was originally initiated by El-Gamal and Cover for the 2 channel setting and later extended to more than 2 channels by Venkataramani, Kramer and Goyal (VKG), ii) and an approach based on Slepian and Wolf's random binning technique, proposed by Pradhan, Puri and Ramchandran (PPR) for L >; 2 descriptions. It is well known that the achievable region due to PPR subsumes the VKG region for the symmetric Gaussian MD problem. Motivated by several practical advantages of random binning based methods over the conditional codebook encoding, this paper focuses on the following important questions: Does a random binning based scheme achieve the performance of conditional codebook encoding, even for the 2 descriptions scenario? Are random binning based approaches beneficial for settings that are not fully symmetric? This paper answers both these questions in the affirmative. Specifically, we propose a 2 descriptions coding scheme, based on random binning, which subsumes the currently known largest region for this problem due to Zhang and Berger. Moreover, we propose its extensions to L >; 2 channels and derive the associated achievable regions. The proposed scheme enjoys the advantages of both encoding paradigms making it particularly useful when there is symmetry only within a subset of the descriptions.

This paper focuses on a new framework for scalable coding of information based on principles derived from common information of two dependent random variables. In the conventional successive refinement setting, the encoder generates two layers of information called the base layer and the enhancement layer. The first decoder, which receives only the base layer, produces a coarse reconstruction of the source, whereas the second decoder, which receives both the layers, uses the enhancement layer to refine the information further leading to a finer reconstruction. It is popularly known that asymptotic rate-distortion optimality at both the decoders is possible if and only if the source-distortion pair is successively refinable. However when the source is not successively refinable under the given distortion metric, it is impossible to achieve rate-distortion optimality at both the layers simultaneously. For this reason, most practical system designers resort to storing two individual representations of the source leading to significant overhead in transmission/storage costs. Inspired by the breadth of applications, in this paper, we propose a new framework for scalable coding wherein a subset of the bits sent to the first decoder is not sent to the second decoder. That is, the encoder generates one common bit stream which is routed to both the decoders, but unlike the conventional successive refinement setting, both the decoders receive an additional individual bitstream. By relating the proposed framework with the problem of common information of two dependent random variables, we derive a single letter characterization for the minimum sum rate achievable for the proposed setting when the two decoders are constrained to receive information at their respective rate-distortion functions. We show using a simple example that the proposed framework provides a strictly better asymptotic sum rate as opposed to the conventional scalable coding setup when the source-distortion pair is not successively refinable.

This paper addresses the L-channel multiple descriptions problem for a Gaussian source under mean squared error (MSE) distortion metric. We focus on particular cross-sections of the general rate-distortion region where a subset the 2L- 1 distortion constraints are not imposed. Specifically, we assume that certain descriptions are never received simultaneously at the decoder and thereby the transmitted codewords require joint typicality only within prescribed subsets. We derive a new encoding scheme and an associated rate-distortion region wherein joint typicality of codewords only within the prescribed subsets is maintained. We show that enforcing joint typicality of all the codewords entails strict suboptimality in the achievable rate-distortion region. Specifically, we consider a 3 descriptions scenario wherein descriptions 1 and 3 are never received simultaneously at the decoder and show that a strictly larger achievable region is obtained when the encoder maintains joint typicality of codewords only within the required subsets. To prove these results, we derive a lemma called the `subset typicality lemma' which plays a critical role in establishing the new achievable region.

When is optimal estimation linear? It is well known that when a Gaussian source is contaminated with Gaussian noise, a linear estimator minimizes the mean square estimation error. This paper analyzes, more generally, the conditions for linearity of optimal estimators. Given a noise (or source) distribution, and a specified signal-to-noise ratio (SNR), we derive conditions for existence and uniqueness of a source (or noise) distribution for which theLpoptimal estimator is linear. We then show that if the noise and source variances are equal, then the matching source must be distributed identically to the noise. Moreover, we prove that the Gaussian source-channel pair is unique in the sense that it is the only source-channel pair for which the mean square error (MSE) optimal estimator is linear at more than one SNR values. Furthermore, we show the asymptotic linearity of MSE optimal estimators for low SNR if the channel is Gaussian regardless of the source and, vice versa, for high SNR if the source is Gaussian regardless of the channel. The extension to the vector case is also considered where besides the conditions inherited from the scalar case, additional constraints must be satisfied to ensure linearity of the optimal estimator.

Canonical distributed quantization schemes do not scale to large sensor networks due to the exponential decoder storage complexity that they entail. Prior efforts to tackle this issue have largely been limited to the suboptimal schemes of source grouping and decoding, thus failing to use all available information at the decoder. We propose a new decoding paradigm where all received bits are used in decoding. Essentially, to decode each source, we partition the space of received bit-tuples using a nearest neighbor quantizer at a decoding rate consistent with the allowed complexity and each partition is then mapped to a reconstruction value for that source. To avoid local minima in design, we resort to deterministic annealing to determine the nearest neighbor partition function (the partitioning prototypes) from the training set. Results on several data-sets show substantial gains over naive and other competing approaches.

This paper considers the problem of optimal multi-hop routing of correlated sources over a network with multiple sinks and arbitrary network demands. We recently introduced a new routing paradigm in [10] called `dispersive information routing' (DIR), wherein the intermediate nodes are allowed to split a packet and forward a subset of the received bits on each forward path. DIR ensures that each sink receives just the information it requires to decode the sources it intends to reconstruct, and thereby outperforms conventional routing techniques in the literature. We proposed an encoding scheme called `power binning' which achieves complete rate region and the minimum cost under this paradigm when each sink is allowed to receive packets only from the sources it wants to reconstruct. This paper considers the optimum encoding scheme when every source can (possibly) communicate with every sink irrespective of what the sinks reconstruct. This generalization happens to be considerably more complex and we derive an achievable rate region and an associated achievable cost using principles from distributed source coding and multiple descriptions encoding.

This paper presents a new achievable rate-distortion region for the L-channel multiple descriptions problem. Currently, the most popular region for this problem is due to Venkataramani, Kramer and Goyal [3]. Their encoding scheme is an extension of the Zhang-Berger scheme to the L-channel case and includes a combinatorial number of refinement codebooks, one for each subset of the descriptions. All the descriptions also share a single common codeword, which introduces redundancy, but assists in better coordination of the descriptions. This paper proposes a novel encoding technique involving `Combinatorial Message Sharing', where every subset of the descriptions may share a distinct common message. This introduces a combinatorial number of shared codebooks along with the refinement codebooks of [3]. These shared codebooks provide a more flexible framework to trade off redundancy across the messages for resilience to descriptions loss. We derive an achievable rate-distortion region for the proposed technique, and show that it subsumes the achievable region of [3].

We recently proposed a new coding scheme for the L-channel multiple descriptions (MD) problem for general sources and distortion measures involving `Combinatorial Message Sharing' (CMS) [7] leading to a new achievable rate-distortion region. Our objective in this paper is to establish that this coding scheme strictly subsumes the most popular region for this problem due to Venkataramani, Kramer and Goyal (VKG) [3]. In particular, we show that for a binary symmetric source under Hamming distortion measure, the CMS scheme provides a strictly larger region for all L>; 2. The principle of the CMS coding scheme is to include a common message in every subset of the descriptions, unlike the VKG scheme which sends a single common message in all the descriptions. In essence, we show that allowing for a common codeword in every subset of descriptions provides better freedom in coordinating the messages which can be exploited constructively to achieve points outside the VKG region.

This paper considers the problem of characterizing the optimal tradeoff between the total transmit versus receive rate in the Gray-Wyner network. This tradeoff plays a crucial role in many important practical applications including establishing fundamental limits in databases for correlated sources and in minimum cost routing for networks. We develop the insight into this tradeoff by defining two quantities C(X, Y; R' ) and K(X,Y;R"), which quantify the shared rate as a function of the total transmit and receive rates respectively. Closely tied up with this tradeoff is the notion of common information of two dependent random variables. The two most influential definitions are due to Wyner [2] and Gács-Körner [1]. Though it is well know that these definitions can be characterized as two extreme points in the Gray-Wyner region, no contour with operational significance is known which connects them. We will show that the tradeoff between transmit and receive rates leads to a contour of points on the boundary of Gray-Wyner region which passes through the operating points of Wyner and Gács-Körner. We use these properties to derive alternate characterizations for the two definitions of common information under a broader unified framework.

This paper considers the problem of distributed source coding for a large sensor network. A typical shortcoming of current approaches to true distributed coding is the exponential growth of the decoder codebook size with the number of sources in the network. This growth in complexity renders many traditional approaches impractical for even moderately sized sensor networks. Inspired by our recent results on fusion coding for selective retrieval, we propose a new distributed coding approach that scales to a large number of sources. Central to our approach is a “bit-subset selector” module whose role is to judiciously extract an appropriate subset of the received bits for decoding per individual source. This, together with joint design of all system components, enables direct optimization of the decoder complexity-distortion tradeoff, and thereby the desired scalability. Experiments on both real and synthetic data-sets show considerable gains over heuristic schemes.

This paper considers the problem of minimizing the communication cost for a general multi-hop network with correlated sources and multiple sinks. For the single sink scenario, it has been shown that this problem can be decoupled, without loss of optimality, into two separate subproblems of distributed source coding and finding the optimal routing (transmission structure). It has further been established that, under certain assumptions, such decoupling also applies in the general case of multiple sinks and arbitrary network demands. We show that these assumptions are significantly restrictive, and further provide examples to substantiate the loss, including settings where removing the assumptions yields unbounded performance gains. Finally, an approach to solving the unconstrained problem, where routing and coding cannot be decoupled, is derived based on Han and Kobayashi's achievability region for multi-terminal coding.

When is optimal estimation linear? It is well-known that, in the case of a Gaussian source contaminated with Gaussian noise, a linear estimator minimizes the mean square estimation error. This paper analyzes more generally the conditions for linearity of optimal estimators. Given a noise (or source) distribution, and a specified signal to noise ratio (SNR), we derive conditions for existence and uniqueness of a source (or noise) distribution that renders the Lpnorm optimal estimator linear. We then show that, if the noise and source variances are equal, then the matching source is distributed identically to the noise. Moreover, we prove that the Gaussian source-channel pair is unique in that it is the only source-channel pair for which the MSE optimal estimator is linear at more than one SNR values.

In this paper, we consider the problem of minimum cost joint compression and routing for networks with multiple-sinks and correlated sources. We introduce a routing paradigm, called dispersive information routing, wherein the intermediate nodes are allowed to forward a subset of the received bits on subsequent paths. This paradigm opens up a rich class of research problems which focus on the interplay between encoding and routing in a network. What makes it particularly interesting is the challenge in encoding sources such that, exactly the required information is routed to each sink, to reconstruct the sources they are interested in. We demonstrate using simple examples that our approach offers better asymptotic performance than conventional routing techniques. We also introduce a variant of the well known random binning technique, called `power binning', to encode and decode sources that are dispersively transmitted, and which asymptotically achieves the minimum communication cost within this routing paradigm.