Onion peeling codes address a distributed data compression scenario related to Slepian-Wolf (SW) data compression. In a 2-user onion peeling code, as in a 2-user SW code, two dependent sources are compressed independently; while the SW code decodes both sources jointly using both source descriptions, the onion peeling code reconstructs the first source using only the first source description and then uses the first source reconstruction and the second source description to reconstruct the second source. The authors' prior results show that first-stage compressors that are almost identical in first-source reconstruction reliability and efficiency can exhibit extremely different best-case performance for the second source. This paper proposes the use of low density parity check (LDPC) source coding in the first stage of an onion peeling code and shows that, with high probability, the random LDPC code design used in the evaluation of this approach generates a first-stage code that performs well both in compressing the first source and in assisting the compressor of the second source. The method works universally for any conditional distribution on the second source given the first, meaning that one does not have to know the conditional distribution of the second source given the first to design a first-stage code with good performance on the second source.

A mechanism called switched feedback is introduced; under switched feedback, each channel output goes forward to the receiver(s) or back to the transmitter(s) but never both. By studying the capacity of the Multiple-Access Channel (MAC) with switched feedback, this work investigates the benefits of feedback, seeking to maximize that benefit under reliable and unreliable feedback scenarios. The study is used to explore the tradeoffs between cooperation and transmission in the context of communication systems. Results include upper and lower bounds on the capacity region of the MAC with switched feedback.

For a multi-source multi-terminal noiseless network, the multicastkey-disseminationproblem, here called thekey-castproblem, involves the task of multicasting a (secret) keyKfrom the network sources to its terminals. Unlike traditional communication, where messages must be delivered from source to destination(s) unchanged, key-cast is more flexible since key-cast need not require source reconstruction at destination nodes. Instead, the distributed key can be a mixture of sources from which the sources themselves may be unrecoverable. Key-cast (also known as secret key agreement) in memoryless networks has seen significant studies over the past decades. This work initiates the study of key-cast in noiseless networks, i.e., network coding, and addresses the similarities and differences between traditional forms of communication that require source reconstruction and the less constrained form of communication in the key-cast task. The study varies according to three criteria. The setting can be secure, in which case the shared key is not revealed to an eavesdropper (whose capabilities we constrain), or non-secure, in which case the key need not be hidden. The setting can have one or multiple source nodes capable of generating independent randomness to be used in the key generation process. And the setting can have one or multiple terminal sets, where the latter case is referred to as multiple key-cast. In multiple key-cast, one requires that terminals in each terminal set decode a different shared key. In the given settings, this work derives combinatorial conditions for key-cast and multiple key-cast and designs corresponding communication schemes. In addition, the study compares the key-cast rate with and without the restriction of source reconstruction that is needed in traditional forms of communication. Key-cast achieves a strict advantage in rate when source reconstruction is relaxed.

This work considers almost lossless onion peeling data compression. In onion peeling data compression, as in Slepian-Wolf data compression, multiple transmitters independently encode their respective sources and transmit their descriptions to a shared decoder. Onion peeling codes differ from Slepian-Wolf codes in that the onion peeling decoder must sequentially decode the individual sources rather than making a single joint decoding decision. This work considers an almost lossless two-stage code. The main result shows that when the first source is coded by a near-optimal code for a given first-stage error constraint, the conditional entropy rate of the second source, given this imperfect reconstruction, can vary within a gap that does not vanish when the blocklength grows without bound. It is also shown that the lower bound for the second-stage conditional entropy rate can be achieved using a classic random binning code in the first stage.

This work introduces a framework for an asynchronous random access source code (ARASC) and bounds the achievable performance under this framework. Like prior multiple access (or Slepian-Wolf) source codes, the ARASC enables multiple transmitters to efficiently, reliably, and independently describe dependent sources to a common receiver. As in prior "random access" codes, the number of active encoders is unknown a priori to both the transmitters and the receiver and single-bit stop-feedback from the receivers to the transmitters enables variable-rate coding. Unlike prior works, the proposed system eliminates all forms of block synchronization. The main result is a two-transmitter achievability bound demonstrating the achievability of a first-order average rate across blocks equal to the weighted average of the point-to-point source coding rate and multiple access achievable sum rate. The weights observed approach the fractions of time that separate and simultaneous observations are encoded. The result’s second order term bounds the speed at which the average rate approaches this weighted average.

Motivated by the design of low-complexity low-power coding solutions for the Gaussian relay channel, this work presents an upper bound on the minimum energy-per-bit achiev-able on the Gaussian relay channel using rank-1 linear relaying. Our study addresses high-dimensional relay codes and presents bounds that outperform prior known bounds using 2-dimensional schemes. A novelty of our analysis ties the optimization problem at hand to the solution of a certain differential equation which, in turn, leads to a low energy-per-bit achievable scheme.

Motivated by the study of multi-source multiterminal key-dissemination, here called “key-cast;’ the work at hand presents a combinatorial characterization of when positiverate secure multicast network coding in the presence of eaves-dropping nodes is possible. In key-cast, introduced by the authors in [ITW2022], network nodes hold independent random bits, and one seeks a communication scheme that allows all terminal nodes to share a secret key$K$. We here address positive (albeit, arbitrarily small) rate key-cast under the security requirement that no single non-terminal network node can gain information about the shared key$K$; this scenario is useful in cryptographic settings. The work at hand studies key-dissemination protocols based on secure network coding and presents a combinatorial characterization of networks that support positive-rate multicast resilient to eavesdroppers that control individual network nodes. The secure-multicast capacity solution in the same setting is a known open problem.

This paper investigates variable-length stop-feedback codes for memoryless channels in point-to-point, multiple access, and random access communication scenarios. The proposed codes employ $L$ decoding times $n_1, n_2, \dots, n_L$ for the point-to-point and multiple access channels and $KL + 1$ decoding times for the random access channel with at most $K$ active transmitters. In the point-to-point and multiple access channels, the decoder uses the observed channel outputs to decide whether to decode at each of the allowed decoding times $n_1, \dots, n_L$, at each time telling the encoder whether or not to stop transmitting using a single bit of feedback. In the random access scenario, the decoder estimates the number of active transmitters at time $n_0$ and then chooses among decoding times $n_{k, 1}, \dots, n_{k, L}$ if it believes that there are $k$ active transmitters. In all cases, the choice of allowed decoding times is part of the code design; given fixed value $L$, allowed decoding times are chosen to minimize the expected decoding time for a given codebook size and target average error probability. The number $L$ in each scenario is assumed to be constant even when the blocklength is allowed to grow; the resulting code therefore requires only sparse feedback. The central results are asymptotic approximations of achievable rates as a function of the error probability, the expected decoding time, and the number of decoding times. A converse for variable-length stop-feedback codes with uniformly-spaced decoding times is included for the point-to-point channel.

This paper studies the third-order characteristic of nonsingular discrete memoryless channels and the Gaussian channel with a maximal-power constraint. The third-order term in our expansions employs a new quantity here called the channel skewness, which affects the approximation accuracy more significantly as the error probability decreases. For the Gaussian channel, evaluating Shannon’s 1959 random coding bound and Vazquez-Vilar’s 2021 meta-converse bound in the central limit theorem (CLT) regime enables exact computation of the channel skewness. For discrete memoryless channels, this work generalizes Moulin’s 2017 bounds on the asymptotic expansion of the maximum achievable message set size for nonsingular channels from the CLT regime to include the moderate deviations (MD) regime, thereby refining Altuğ and Wagner’s 2014 MD result. For an example binary symmetric channel and most practically important$(n, \epsilon)$pairs, including$n \in [{100, 500}]$and$\epsilon \in [10^{-10}, 10^{-1}]$, an approximation up to the channel skewness is the most accurate among several expansions in the literature. A derivation of the third-order term in the type-II error exponent of binary hypothesis testing in the MD regime is also included; the resulting third-order term is similar to the channel skewness.

The multiple access channel (MAC) capacity with feedback is considered under feedback models designed to tease out which factors contribute to the MAC feedback capacity benefit. Comparing the capacity of a MAC with "perfect" feedback, which causally delivers to the transmitters the true channel output, to that of a MAC with "independent" feedback, which causally delivers to the transmitters an independent instance of that same channel output, allows separation of effects like cooperation from alternative feedback benefits such as knowledge of the channel instance. Proving that the Cover-Leung (CL) achievability bound, which is known to be loose for some channels, is achievable also under (shared or distinct) independent feedback at the transmitters shows that the CL bound does not require transmitter knowledge of the channel instance. Proving that each transmitter’s maximal rate under independent feedback exceeds that under perfect feedback highlights the potential power of an independent look at the channel output.

This work addresses the cooperation facilitator (CF) model, in which network nodes coordinate through a rate limited communication device. For multiple-access channel (MAC) encoders, the CF model is known to show significant rate benefits, even when the rate of cooperation is negligible. Specifically, the benefit in MAC sum-rate, as a function of the cooperation rate CCF, sometimes has an infinite slope at CCF= 0 when the CF enables transmitter dependence where none was possible otherwise. This work asks whether cooperation through a CF can yield similar infinite-slope benefits when dependence among MAC transmitters has no benefit or when it can be established without the help of the CF. Specifically, this work studies the CF model when applied to relay nodes of a single-source, single-terminal, diamond network comprising a broadcast channel followed by a MAC. In the relay channel with orthogonal receiver components, careful generalization of the partial-decode-forward/compress-forward lower bound to the CF model yields sufficient conditions for an infinite-slope benefit. Additional results include derivation of a family of diamond networks for which the infinite-slope rate-benefit derives directly from the properties of the corresponding MAC studied in isolation.

This paper studies a special case of the problem of source coding with side information. A single transmitter describes a source to a receiver that has access to a side information observation that is unavailable at the transmitter. While the source and true side information sequences are dependent, stationary, memoryless random processes, the side information observation at the decoder is unreliable, which here means that it may or may not equal the intended side information and therefore may or may not be useful for decoding the source description. The probability of side information observation failure, caused, for example, by a faulty sensor or source decoding error, is non-vanishing but is bounded by a fixed constant independent of the blocklength. This paper proposes a coding system that uses unreliable side information to get efficient source representation subject to a fixed error probability bound. Results include achievability and converse bounds under two different models of the joint distribution of the source, the intended side information, and the side information observation.

The channel coding problem in the moderate deviations regime is studied; here, the error probability sub-exponentially decays to zero, and the rate approaches the capacity slower than $O(1/\sqrt n )$. The main result refines Altuğ and Wagner’s moderate deviations result by deriving lower and upper bounds on the third-order term in the asymptotic expansion of the maximum achievable message set size. The third-order term of the expansion employs a new quantity called the channel skewness. For the binary symmetric channel and most practically important (n,ϵ) pairs, including n ∈ [100, 500] and ϵ ∈ [10−10,10−1], an approximation up to the channel skewness is the most accurate among several expansions in the literature.

For a multi-source multi-terminal noiseless network, the key-dissemination problem involves the task of multicasting a secret key K from the network sources to its terminals. As in secure multicast network-coding, in the key-dissemination problem the source nodes have access to independent randomness and, as the network is noiseless, the resulting key K is a function of the sources’ information. However, different from traditional forms of multicast, in key-dissemination the key K need not consist of source messages, but rather may be any function of the information generated at the sources, as long as it is shared by all terminals. Allowing the shared key K to be a mixture of source information grants a flexibility to the communication process which gives rise to the potential of increased key-rates when compared to traditional secure multicast. The multicast key-capacity is the supremum of achievable key-rates, subject to the security requirement that the shared key is not revealed to an eavesdropper with predefined eavesdropping capabilities. The key-dissemination problem (termed also, secret key-agreement) has seen significant studies over the past decades in memoryless network structures. In this work, we initiate the study of key-dissemination in the context of noiseless networks, i.e., network coding. In this context, we study similarities and differences between traditional secure-multicast and the more lenient task of key-dissemination.

The work at hand presents a finite-blocklength analysis of the multiple access channel (MAC) sum-rate under the cooperation facilitator (CF) model. The CF model, in which independent encoders coordinate through an intermediary node, is known to show significant rate benefits, even when the rate of cooperation is limited. We continue this line of study for cooperation rates which are sub-linear in the blocklength n. Roughly speaking, our results show that if the facilitator transmits log$K$bits, then there is a sum-rate benefit of order √log K/n compared to the best-known achievable rate. This result extends across a wide range of K: even a single bit of cooperation is shown to provide a sum-rate benefit of order 1/√n.

The edge-removal problem asks whether the removal of a$\lambda$-capacity edge from a given network can decrease the communication rate between source-terminal pairs by more than$\lambda$. We prove that for undirected networks, removing a$\lambda$capacity edge decreases the rate by$O(\lambda$. Through previously known reductive arguments, here newly applied to undirected networks, our result implies that the zero-error capacity region of an undirected network equals its vanishing-error capacity region. Whether it is possible to prove similar results for directed networks remains an open question.

This paper proposes a nested low-density parity-check (LDPC) code design. Combining this nested LDPC code with the random access coding strategy introduced by Yavas, Kostina, and Effros yields a random access LDPC (RA-LDPC) code for reliable communication in random access communication environments where neither the transmitters nor the receiver knows which or even how many transmitters wish to communicate at each moment. Coordination is achieved using sparse scheduled feedback. Bounds on the finite-blocklength performance of the RA-LDPC code under maximum likelihood (ML) decoding are derived using both error exponent and dispersion style analyses. Results include bounds on the penalty of the RA-LDPC code as a function of the LDPC code densities.

We investigate variable-length feedback (VLF) codes for the Gaussian point-to-point channel under maximal power, average error probability, and average decoding time constraints. Our proposed strategy chooses$K < \infty$decoding times$n_{1}, n_{2}, \ldots, n_{K}$rather than allowing decoding at any time$n=0,1,2, \ldots$. We consider stop-feedback, which is one-bit feedback transmitted from the receiver to the transmitter at times$n_{1}, n_{2},\ldots$only to inform her whether to stop. We prove an achievability bound for VLF codes with the asymptotic approximation$\ln M \approx \frac{NC(P)}{1-\epsilon}-\sqrt{N\ \ln_{(K-1)}(N)\frac{V(P)}{1-\epsilon}}$, where$\ln(K)(\cdot)$denotes the$K$-fold nested logarithm function,$N$is the average decoding time, and$C(P)$and$V(P)$are the capacity and dispersion of the Gaussian channel, respectively. Our achievability bound evaluates a non-asymptotic bound and optimizes the decoding times$n_{1}, \ldots, n_{K}$within our code architecture.

This paper investigates variable-length feedback codes for discrete memoryless channels in point-to-point, multiple access, and random access communication. The proposed nested code employs L decoding times $n_{1}, n_{2}, \ldots, n_{L}$ for the point-to-point and multiple access channels and KL decoding times $\left\{n_{k, \ell}: 1 \leq k \leq K, 1 \leq \ell \leq L\right\}$ for the random access channel with at most K active transmitters; in the latter case, decoding times $n_{k, \ell}, 1 \leq \ell \leq L$ are reserved for decoding in the scenario where the decoder believes that the number of active transmitters is k. The code has a nested structure, i.e., codewords used to decode messages from k active transmitters are prefix of codewords used to decode messages from $k+1$ active transmitters. The code employs single-bit, scheduled feedback from the receiver to the transmitters at each potential decoding time to inform the transmitters whether or not it is able to decode. Transmitters cease transmission, thereby truncating their codewords, when no further transmissions are required by the decoder. The choice of decoding times is optimized to minimize the expected decoding time subject to an error probability constraint, and second order achievability bounds are derived.

In communication networks, cooperative strategies are coding schemes where network nodes work together to improve performance metrics such as the total rate delivered across the network. This work studies encoder cooperation in the setting of a discrete multiple access channel (MAC) with two encoders and a single decoder. A network node, here called the cooperation facilitator (CF), that is connected to both encoders via rate-limited links, enables the cooperation strategy. Previous work by the authors presents two classes of MACs: (i) one class where the average-error sum-capacity has an infinite derivative in the limit where CF output link capacities approach zero, and (ii) a second class of MACs where the maximal-error sum-capacity is not continuous at the point where the output link capacities of the CF equal zero. This work contrasts the power of the CF in the maximal- and average-error cases, showing that a constant number of bits communicated over the CF output link can yield a positive gain in the maximal-error sum-capacity, while a far greater number of bits, even a number that grows sublinearly in the blocklength, can never yield a non-negligible gain in the average-error sum-capacity.

Consider a random access communication scenario over a channel whose operation is defined for any number of possible transmitters. As in the model recently introduced by Polyanskiy for the Multiple Access Channel (MAC) with a fixed, known number of transmitters, the channel is assumed to be invariant to permutations on its inputs, and all active transmitters employ identical encoders. Unlike the Polyanskiy model, in the proposed scenario, neither the transmitters nor the receiver knows which transmitters are active. We refer to this agnostic communication setup as the Random Access Channel (RAC). Scheduled feedback of a finite number of bits is used to synchronize the transmitters. The decoder is tasked with determining from the channel output the number of active transmitters, k, and their messages but not which transmitter sent which message. The decoding procedure occurs at a time ntdepending on the decoder's estimate, t, of the number of active transmitters, k, thereby achieving a rate that varies with the number of active transmitters. Single-bit feedback at each time ni, i ≤ t, enables all transmitters to determine the end of one coding epoch and the start of the next. The central result of this work demonstrates the achievability on a RAC of performance that is first-order optimal for the MAC in operation during each coding epoch. While prior multiple access schemes for a fixed number of transmitters require 2k- 1 simultaneous threshold rules, the proposed scheme uses a single threshold rule and achieves the same dispersion.

This paper presents finite-blocklength achievability bounds for the Gaussian multiple access channel (MAC) and random access channel (RAC) under average-error and maximal-power constraints. Using random codewords uniformly distributed on a sphere and a maximum likelihood decoder, the derived MAC bound on each transmitter’s rate matches the MolavianJazi-Laneman bound (2015) in its first- and second-order terms, improving the remaining terms to$\frac {1}2\frac {\log {n}}{n}+{O} \left ({\frac {1}{n}}\right)$bits per channel use. The result$\vphantom {\sum ^{R}}$then extends to a RAC model in which neither the encoders nor the decoder knows which of${K}$possible transmitters are active. In the proposed rateless coding strategy, decoding occurs at a time${n}_{t}$that depends on the decoder’s estimate${t}$of the number of active transmitters${k}$. Single-bit feedback from the decoder to all encoders at each potential decoding time${n}_{i}$,${i} \leq {t}$, informs the encoders when to stop transmitting. For this RAC model, the proposed code achieves the same first-, second-, and third-order performance as the best known result for the Gaussian MAC in operation.

This paper applies error-exponent and dispersionstyle analyses to derive finite-blocklength achievability bounds for low-density parity-check (LDPC) codes over the point-to-point channel (PPC) and multiple access channel (MAC). The error-exponent analysis applies Gallager's error exponent to bound achievable symmetrical and asymmetrical rates in the MAC. The dispersion-style analysis begins with a generalization of the random coding union (RCU) bound from random code ensembles with i.i.d. codewords to random code ensembles in which codewords may be statistically dependent; this generalization is useful since the codewords of random linear codes such as LDPC codes are dependent. Application of the RCU bound yields finite-blocklength error bounds and asymptotic achievability results for both i.i.d. random codes and LDPC codes. For discrete, memoryless channels, these results show that LDPC codes achieve first- and second-order performance that is optimal for the PPC and identical to the best prior results for the MAC.

This paper presents finite-blocklength achievabil- ity bounds for the Gaussian multiple access channel (MAC) and random access channel (RAC) under average-error and maximal-power constraints. Using random codewords uniformly distributed on a sphere and a maximum likelihood decoder, the derived MAC bound on each transmitter's rate matches the MolavianJazi-Laneman bound (2015) in its first- and second-order terms, improving the remaining terms to 1/2 (log n)/n + O(1/n) bits per channel use. The result then extends to a RAC model in which neither the encoders nor the decoder knows which of K possible transmitters are active. In the proposed rateless coding strategy, decoding occurs at a time ntthat depends on the decoder's estimate t of the number of active transmitters k. Single-bit feedback from the decoder to all encoders at each potential decoding time ni, i ≤ t, informs the encoders when to stop transmitting. For this RAC model, the proposed code achieves the same first-, second-, and third-order performance as the best known result for the Gaussian MAC in operation.

This work studies point-to-point, multiple access, and random access lossless source coding in the finite-blocklength regime. In each scenario, a random coding technique is developed and used to analyze third-order coding performance. Asymptotic results include a third-order characterization of the Slepian-Wolf rate region with an improved converse that relies on a connection to composite hypothesis testing. For dependent sources, the result implies that the independent encoders used by Slepian-Wolf codes can achieve the same third-order-optimal performance as a single joint encoder. The concept of random access source coding is introduced to generalize multiple access (Slepian-Wolf) source coding to the case where encoders decide independently whether or not to participate and the set of participating encoders is unknown a priori to both the encoders and the decoder. The proposed random access source coding strategy employs rateless coding with scheduled feedback. A random coding argument proves the existence of a single deterministic code of this structure that simultaneously achieves the third-order-optimal Slepian-Wolf performance for each possible active encoder set.

The edge-removal problem addresses the loss in capacity obtained by the removal of an edge of capacity λ from a given network. In the context of network coding, the edgere-moval problem has been solved for certain families of networks (such as instances with co-located sources). However, the problem is open for general instances. This work ties the edge-removal problem to two additional problems in the context of network communication: (a) the “zero-error” network coding problem, which asks whether the zero-error network coding capacity differs from the capacity when error probability is allowed to tend to zero as the blocklength grows; and (b) the “δ-dependence” problem, which measures the advantage of message dependence in network coding capacity.

Smart meters enable improvements in electricity distribution system efficiency at some cost in customer privacy. Users with home batteries can mitigate this privacy loss by applying charging policies that mask their underlying energy use. A battery charging policy is proposed and shown to provide universal privacy guarantees subject to a constraint on energy cost. The guarantee bounds our strategy's maximal information leakage from the user to the utility provider under general stochastic models of user energy consumption. The policy construction adapts coding strategies for non-probabilistic permuting channels to this privacy problem.

It is common in the study of secure multicast network coding in the presence of an eavesdropper that has access to z network links, to assume that the source node is the only node that generates random keys. In this setting, the secure multicast rate is well understood. Computing the secure multicast rate, or even the secure unicast rate, in the more general setting in which all network nodes may generate (independent) random keys is known to be as difficult as computing the (non-secure) capacity of multiple-unicast network coding instances - a well known open problem. This work treats an intermediate model of secure unicast in which only one node can generate random keys, however that node need not be the source node. The secure communication rate for this setting is characterized again with an eavesdropper that has access to z network links.

This paper treats point-to-point, multiple access and random access lossless source coding in the finite-blocklength regime. A random coding technique is developed, and its power in analyzing the third-order coding performance is demonstrated in all three scenarios. Results include a third-order-optimal characterization of the Slepian-Wolf rate region and a proof showing that for dependent sources, the independent encoders used by Slepian-Wolf codes can achieve the same third-order- optimal performance as a single joint encoder. The concept of random access source coding, which generalizes the multiple access scenario to allow for a subset of participating encoders that is unknown a priori to both the encoders and the decoder, is introduced. Contributions include a new definition of the probabilistic model for a random access-discrete multiple source, a general random access source coding scheme that employs a rateless code with sporadic feedback, and an analysis demonstrating via a random coding argument that there exists a deterministic code of the proposed structure that simultaneously achieves the third- order-optimal performance of Slepian-Wolf codes for all possible subsets of encoders.

The edge removal problem studies the loss in network coding rates that results when a network communication edge is removed from a given network. It is known, for example, that in networks restricted to linear coding schemes and networks restricted to Abelian group codes, removing an edge e* with capacity Re. reduces the achievable rate on each source by no more than Re*. In this work, we seek to uncover larger families of encoding functions for which the edge removal statement holds. We take a local perspective: instead of requiring that all network encoding functions satisfy certain restrictions (e.g., linearity), we limit only the function carried on the removed edge e*. Our central results give sufficient conditions on the function carried by edge e* in the code used to achieve a particular rate vector under which we can demonstrate the achievability of a related rate vector once e* is removed.

Consider a random access communication scenario over a channel whose operation is defined for any number of possible transmitters. Inspired by the model recently introduced for the Multiple Access Channel (MAC) with a fixed, known number of transmitters by Polyanskiy, we assume that the channel is invariant to permutations on its inputs, and that all active transmitters employ identical encoders. Unlike Polyanskiy, we consider a scenario in which neither the transmitters nor the receiver know which or how many transmitters are active. We refer to this agnostic communication setup as the Random Access Channel, or RAC. Limited feedback is used to ensure that the collection of active transmitters remains fixed during each epoch. The decoder is tasked with determining from the channel output the number of active transmitters (k) and their messages but not which transmitter sent which message. The central result of this work demonstrates the achievability on a RAC of performance that is first-order optimal for the MAC in operation during each coding epoch. While prior multiple access schemes for a fixed number of transmitters require 2k- 1 simultaneous threshold rules, the proposed scheme uses a single threshold rule and achieves the same dispersion.

In communication networks, cooperative strategies are coding schemes where network nodes work together to improve network performance metrics such as sum-rate. This work studies encoder cooperation in the setting of a discrete multiple access channel with two encoders and a single decoder. A node in the network that is connected to both encoders via rate-limited links, referred to as the cooperation facilitator (CF), enables the cooperation strategy. Previously, the authors presented a class of multiple access channels where the average-error sum-capacity has an infinite derivative in the limit where CF output link capacities approach zero. The authors also demonstrated that for some channels, the maximal-error sum-capacity is not continuous at the point where the output link capacities of the CF equal zero. This work shows that the average-error sum-capacity is continuous when CF output link capacities converge to zero; that is, the infinite derivative of the average-error sum-capacity is not a result of its discontinuity as in the maximal-error case.

Cooperation strategies allow communication devices to work together to improve network capacity. Consider a network consisting of k encoders, a multiple access channel (MAC), a decoder, and a node, referred to as a “cooperation facilitator” (CF), that is connected to each encoder via a pair of rate-limited links, with one link going from the encoder to the CF and the other link going back. Let the “cooperation rate” be the total outgoing rate of the CF. This paper demonstrates the existence of a class of MACs where the ratio of the sum-capacity gain to cooperation rate tends to infinity as the cooperation rate tends to zero. For any k ≥ 2, examples of channels in this class include the k-user binary adder MAC and the k-user Gaussian MAC.

In network cooperation strategies, nodes work together with the aim of increasing transmission rates or reliability. This paper demonstrates that enabling cooperation between the transmitters of a two-user multiple access channel via a cooperation facilitator that has access to both messages results in a network whose maximal- and average-error capacity regions are the same; this benefit ensues even when the information received by each transmitter is negligible. From this result, it follows that if a multiple access channel with no transmitter cooperation has different maximal- and average-error sum-capacities, then the maximal-error sum-capacity of the network consisting of this channel and a cooperation facilitator is not continuous with respect to the output edge capacities of the facilitator. Thus, there exist networks where adding negligible rate yields a non-negligible benefit.

In many communication networks, the availability of channel state information at various nodes provides an opportunity for network nodes to work together, or “cooperate.” This work studies the benefit of cooperation in the multiple access channel with a cooperation facilitator, distributed state information at the encoders, and full state information available at the decoder. Under various causality constraints, sufficient conditions are obtained such that encoder cooperation through the facilitator results in a gain in sum-capacity that has infinite slope in the information rate shared with the encoders. This result extends the prior work of the authors on cooperation in networks where none of the nodes have access to state information.

A key result of classical information theory states that if the rate of a randomly generated codebook is less than the mutual information between the channel's input and output, then the probability that that codebook has negligible error goes to one as the blocklength goes to infinity. In an attempt to bridge the gap between the probabilistic world of classical information theory and the combinatorial world of zero-error information theory, this work derives necessary and sufficient conditions on the rate so that the probability that a randomly generated codebook operated under list decoding (for any fixed list size) has zero error probability goes to one as the blocklength goes to infinity. Furthermore, this work extends the classical birthday problem to an information-theoretic setting, which results in the definition of a “noisy” counterpart of Rényi entropy, analogous to how mutual information can be considered a noisy counterpart of Shannon entropy.

We consider a delay-constrained streaming model for zero-error communications and show that under this model, network coding and index coding problems are code equivalent. That is, any streaming network coding instance can be efficiently mapped to a corresponding acyclic streaming index coding instance such that an index code for the latter can be efficiently transformed into a network code for the former. This reduction holds even for network coding instances that contain cycles, thereby proving the first known reduction from cyclic to finite acyclic network coding networks.

In this work we characterize the capacity region for multi-source multi-terminal acyclic network coding with dependent information sources. We show that a full characterization of the capacity region can be derived in terms of entropy functions.

Cooperation strategies that allow communication devices to work together can improve network capacity. This paper generalizes the “cooperation facilitator” (CF) model from the 2-user to the k-user multiple access channel (MAC), extending capacity bounds, characterizing all k-user MACs for which the sum-capacity gain of encoder cooperation exceeds the capacity cost that enables it, and demonstrating an infinite benefit-cost ratio in the limit of small cost.

In network cooperation strategies, nodes work together with the aim of increasing transmission rates or reliability. This paper demonstrates that enabling cooperation between the transmitters of a two-user multiple access channel via a cooperation facilitator that has access to both messages, always results in a network whose maximal- and average-error sum-capacities are the same-even when the information shared with the encoders is negligible. Thus, for a multiple access channel whose maximal- and average-error sum-capacities differ, the maximal-error sum-capacity is not continuous with respect to the output edge capacities of the facilitator. This shows that for some networks, sharing even a negligible number of bits per channel use with the encoders can yield a non-negligible benefit.

In this work, we study the Yeung network coding entropic function outer bound and prove an equivalence relationship between its tightness and the edge removal problem. In addition, we derive an implicit characterization of the 0-error capacity region using restricted sets of entropic vectors.

In cooperative communication, network nodes that would otherwise act independently instead coordinate their efforts with the aim of improving communication performance. To better understand cooperation, we consider communication over a multiple access channel using a “cooperation facilitator”, a node that receives rate-limited message descriptions from the transmitters and sends rate-limited message descriptions back. This model includes the conferencing encoders model and a prior model from the current authors as special cases. We characterize a class of multiple access channels for which there is no gain in sum-capacity under current or prior cooperation models. We then show that for all other multiple access channels, the gain in sum-capacity can be far greater than the capacity of the cooperation facilitator's output links. These channels violate the edge removal property. The Gaussian multiple access channel is an important special case for which we explicitly characterize the sum-rate cooperation gain.

In recent work, Kamath et al. show that network code design for any k-unicast network reduces to network code design for a related 2-unicast network. The proof assumes that codes achieve their desired rates precisely (rather than approaching them asymptotically) and that error probability equals zero. We study two questions posed in but left unanswered by the Kamath et al. paper. The first asks whether the reduction for 0-error code design can be extended to show an equivalence in 0-error network capacity, which includes rates approached asymptotically. The second asks whether the reduction can be generalized to show an equivalence in Shannon capacity, which requires that error probability approach (but not necessarily hit) zero. While we do not solve these questions, we show that finding the k-unicast capacity reduces to finding the 2-unicast capacity under this reduction if and only if the so called “edge removal statement” is true for all networks. This equivalence holds under both 0-error and asymptotic notions of reliability.

We show that the network coding and index coding problems are equivalent. This equivalence holds in the general setting which includes linear and nonlinear codes. Specifically, we present a reduction that maps a network coding instance to an index coding instance while preserving feasibility, i.e., the network coding instance has a feasible solution if and only if the corresponding index coding instance is feasible. In addition, we show that one can determine the capacity region of a given network coding instance with colocated sources by studying the capacity region of a corresponding index coding instance. Previous connections between network and index coding were restricted to the linear case.

In this paper, we prove the separation of source-network coding and channel coding in wireline networks. For the purposes of this paper, a wireline network is any network of independent, memoryless, point-to-point, and finite-alphabet channels used to transmit dependent sources either losslessly or subject to a distortion constraint. In deriving this result, we also prove that in a general memoryless network with dependent sources, lossless, and zero-distortion reconstruction are equivalent provided that the conditional entropy of each source given the other sources is nonzero. Furthermore, we extend the separation result to the case of continuous-alphabet and point-to-point channels, such as additive white Gaussian noise channels.

In this paper, we propose a new cooperation model for discrete memoryless multiple access channels. Unlike in prior cooperation models (e.g., conferencing encoders), where the transmitters cooperate directly, in this model the transmitters cooperate through a larger network. We show that under this indirect cooperation model, there exist channels for which the increase in sum-capacity resulting from cooperation is significantly larger than the rate shared by the transmitters to establish the cooperation. This result contrasts both with results on the benefit of cooperation under prior models and results in the network coding literature, where attempts to find examples in which similar small network modifications yield large capacity benefits have to date been unsuccessful.

An equivalence between multiple multicast and multiple unicast network codes is proven in a 2007 paper by Dougherty and Zeger. A related equivalence between multiple multicast and multiple unicast capacity for general (possibly noisy) memoryless networks is proven in a 2013 paper by the current authors. While the construction used in the proof from the earlier paper maps any linear code for one network to a linear code for the other network, the construction from the later paper does not necessarily preserve linearity. As a result, the 2013 result does not prove an equivalence between the capacity achievable by linear codes in memoryless multiple multicast and memoryless multiple unicast networks. The linear capacity equivalence for memoryless multiple multicast and multiple unicast networks is proven in this work.

In recent years, work has begun to emerge demonstrating intriguing relationships between seemingly disparate information theoretic problems. For example, recent results establish powerful ties between solutions for networks of memoryless channels and networks of noiseless links (network coding networks), between network coding networks in which every internal node can code and a particular subset of network coding networks in which only a single internal node can code (index coding networks), and between multiple multicast demands on memoryless networks and multiple unicast demands on memoryless networks. While the results vary widely, together, they hint at the potential for a unifying theory. In this work, we consider one possible framework for such a theory. Inspired by ideas from the field of computational complexity theory, the proposed framework generalizes definitions and techniques for reduction, completeness, and approximation to the information theoretic domain. One possible outcome from such a theory is a taxonomy of information theoretic problems where problems in the same taxonomic class share similar properties in terms of their code designs, capacities, or other forms of solution. Another potential outcome is the identification of small classes of network information theoretic problems whose solutions, were they available, would solve all information theoretic problems in a much larger class. A third potential outcome is the development of techniques by which approximate solution for one family of network information theoretic problems can be obtained from precise or approximate solution of another family of networks.

We consider the delay of network coding compared to routing with retransmissions in packet erasure networks with probabilistic erasures. We investigate the sublinear term in the block delay required for unicasting n packets and show that there is an unbounded gap between network coding and routing. In particular, we show that delay benefit of network coding scales at least as √n. Our analysis of the delay function for the routing strategy involves a major technical challenge of computing the expectation of the maximum of two negative binomial random variables. Previous characterizations of this expectation are approximate; we derive an exact characterization and analyze its scaling behavior, which may be of independent interest. We also use a martingale bounded differences argument to show that the actual coding delay is concentrated around its expectation.

A technique for bounding the capacities of networks of independent channels is introduced. Parts I and II treat point-to-point and multiterminal channels, respectively. Bounds are derived using a new tool called a bounding model. Channel 1 is an upper (lower) bounding model for channel 2 if replacing channel 2 by channel 1 in any network yields a new network whose capacity region is a superset (subset) of the capacity region of the original network. This paper derives bounding models from noiseless links, with lower bounding models corresponding to points in the channel's capacity region and upper bounding models corresponding to points in a new channel characterization called an emulation region. Replacing all channels in a network by their noiseless upper (lower) bounding models yields a network of lossless links whose capacity region is a superset (subset) of the capacity region for the original network. This converts a general (often stochastic) network into a network coding instance, enabling the application of tools and results derived in that domain. A channel's upper and lower bounding models differ when the channel can carry more information in some networks than in others. Bounding the difference between upper and lower bounding models bounds both the accuracy of the technique and the price of separating source-network coding from channel coding.

We show that the network coding and index coding problems are equivalent. This equivalence holds in the general setting which includes linear and non-linear codes. Specifically, we present an efficient reduction that maps a network coding instance to an index coding instance while preserving feasibility. Previous connections were restricted to the linear case.

In recent work by Effros, El Rouayheb, and Langberg, an equivalence of code feasibility between network and index coding is derived. The authors ask whether the capacity region of a network coding problem can be obtained by solving the capacity region of an index coding problem. We answer in the affirmative for the linear coding case. While the question is still open for the general case, we show that it is related to the edge removal problem, which has been studied recently.

We consider multi-resolution streaming in fully-connected peer-to-peer networks, where transmission rates are constrained by arbitrarily specified upload capacities of the source and peers. We fully characterize the capacity region of rate vectors achievable with arbitrary coding, where an achievable rate vector describes a vector of throughputs of the different resolutions that can be supported by the network. We then prove that all rate vectors in the capacity region can be achieved using pure routing strategies. This shows that coding has no capacity advantage over routing in this scenario.

In this paper, we investigate the impact of a single edge on the capacity region of a network of error-free, point-to-point links. A family of networks and edges is said to exhibit the “edge removal property” if for any network and edge in the family, removing a δ-capacity edge changes the capacity region by at most δ in each dimension. We derive a sufficient condition on network coding functions to guarantee that the edge removal property holds when the network is operated using functions satisfying the condition. Also, we extend the family of network capacity bounds for which it is known that removing a single edge of capacity δ changes the capacity bound by at most f(δ) in each dimension. Specifically, we show that removing a single δ-capacity edge changes the Generalized Network Sharing outer bound by at most δ in each dimension and the Linear Programming outer bound by at most a constant times δ in each dimension.

Traditional studies of multi-source, multi-terminal interference channels typically allow a vanishing probability of error in communication. Motivated by the study of network coding, this work addresses the task of quantifying the loss in rate when insisting on zero error communication in the context of interference channels.

We investigate the tradeoffs between rate (bits per channel use) and throughput (bits per time) in network coding over networks of independent noisy or noiseless channels under on-off scheduling. While some networks exhibit an inherent tradeoff between rate and throughput at finite blocklengths, our main result shows that rate and throughput can be simultaneously maximized in the limit of large blocklengths.

In this work, we address the capacity region of multi-source multi-terminal network communication problems, and study the change in capacity when one moves form independent to dependent source information. Specifically, we ask whether the trade off between capacity and source independence is of continuous nature. We tie the question at hand to that of edge removal which has seen recent interest.

This paper investigates the impact of noise dependence and signal delay on the capacities of networks. It is shown that statistical dependence between the noise in a network's component channels is helpful for communication. In particular, the capacity region of a network whose component channels exhibit dependent noise is a superset of the capacity region of a network built from the same component channels when the noise in those channels is independent. It is also shown that delay has no impact on capacity. That is, the capacity of a network of memoryless channels is unchanged by the addition of finite delays anywhere in the network. Both results are proven for all possible networks of memoryless point-to-point and multi-terminal channels under multiple multicast connections. All other connection types (e.g., single unicast, multiple unicast, single multicast, and mixed unicast and multicast connections) are special cases of the multiple multicast problem, and are therefore included in the given results. The results also generalize from reliable communication of independent sources to lossy or lossy description of possibly dependent sources (i.e., joint source-channel coding) across the same network.

A game-theoretic framework is introduced for studying selfish user behavior in shared wireless networks. The investigation treats an n-unicast problem in a wireless network that employs a restricted form of network coding called reverse carpooling. Unicast sessions independently choose routes through the network. The cost of a set of unicast routes is the number of wireless transmissions required to establish those connections using those routes. Game theory is employed as a tool for analyzing the impact of cost sharing mechanisms on the global system performance when each unicast independently and selfishly chooses its route to minimize its individual cost. The investigation focuses on the performance of stable solutions, where a stable solution is one in which no single unicast can improve its individual cost by changing its route. The results include bounds on the best- and worst-case stable solutions compared to the best performance that could be found and implemented using a centralized controller. The optimal cost sharing protocol is derived and the worst-case solution is bounded. That worst-case stable performance cannot be improved using cost-sharing protocols that are independent of the network structure.

We consider the problem of finding network coding capacities of networks of independent point-to-point channels in the presence of a Byzantine adversary. We assume that the adversary knows all messages, and noise values and the code used to communicate across the network. The adversary controls an unknown subset of edges and can replace the channel output vectors from those edges. We show that finding the capacity for the above network is equivalent to finding the capacity of a network that is obtained by replacing each finite input alphabet point-to-point channel by a noiseless link of the noisy channel capacity. Our result shows the asymptotic optimality of separation between channel coding for each link followed by network coding for the resulting network under the corresponding model of adversarial attack.

We prove a strong converse for a class of source coding problems. In each member of this class, there is one node that has access to all of the source sequences. This node has a direct link to every other node that can observe one or more source sequences. Source sequences are drawn i.i.d. according to a given distribution on a finite alphabet. The distortion measure is finite. The strong converse discussed here states that if a given rate vector R is not D-achievable, then for any sequence of block codes at rate R the probability of observing distortion no greater than D decreases exponentially to 0 as the code dimension grows without bound. The result can be applied to prove the strong converse for the Gray-Wyner problem and the multiple description problem.

Characterizing the set of achievable distortions in lossy transmission of dependent sources over multi-user channels is an open problem. Even for a channel such as the multiple access channel, where the capacity is well-understood, the complete characterization of achievable distortions is still unknown. Since separation of source coding and channel coding is sub-optimal for this case, bounds on achievable distortions require tools beyond prior channel capacities and rate-distortion bounds. In this paper we propose a systematic approach for finding lower bounds on the set of distortions achievable through joint source-channel coding.

An equivalence result is established between the Shannon capacity and the stable capacity of communication networks. Given a discrete-time network with memoryless, time-invariant, discrete-output channels, it is proved that the Shannon capacity equals the stable capacity. The results treat general demands (e.g., multiple unicast demands) and apply even when neither the Shannon capacity nor the stable capacity is known for the given demands. The result also generalize from discrete-alphabet channels to Gaussian channels.

The recently introduced network equivalence results are used to create bit-pipe models that can replace multi-terminal channels within a discrete memoryless network. The goal is to create a set of simple “components” or “blocks” that can be substituted for the channel in such a way that the resulting network is capable of emulating the operation of the original one. We develop general upper and lower bounding models for the multiple access channel and for a class of broadcast channels. These bounds are sharp in the sense that there exists networks where the original channel can achieve the maximum sum rate permissible through the upper or lower bounding models. This approach provides a simple method for analyzing the capacity of large networks, which we illustrate with an example.

This work considers the multiple-access multicast error-correction scenario over a packetized network withzmalicious edge adversaries. The network has min-cutmand packets of lengthl, and each sink demands all information from the set of sourcesS. The capacity region is characterized for both a “side-channel” model (where sources and sinks share some random bits that are secret from the adversary) and an “omniscient” adversarial model (where no limitations on the adversary's knowledge are assumed). In the “side-channel” adversarial model, the use of a secret channel allows higher rates to be achieved compared to the “omniscient” adversarial model, and a polynomial-complexity capacity-achieving code is provided. For the “omniscient” adversarial model, two capacity-achieving constructions are given: the first is based on random subspace code design and has complexity exponential inlm, while the second uses a novel multiple-field-extension technique and has O(lm|S|) complexity, which is polynomial in the network size. Our code constructions are “end-to-end” in that all nodes except the sources and sinks are oblivious to the adversaries and may simply implement predesigned linear network codes (random or otherwise). Also, the sources act independently without knowledge of the data from other sources.

In this paper, new upper and lower bounds are first presented for the rate loss of multiple description source codes (MDSCs). For a two-description MDSC (2DSC), the rate loss of descriptioniwith distortionDiis defined asLi=Ri-R(Di),i∈ {1,2}, whereRiis the rate of theith description andR(D) is the rate-distortion function for the given source; the joint rate loss associated with decoding the two descriptions together to achieve central distortionD0is measured asL0=R1+R2-R(D0). Consider an arbitrary memoryless source with variance σ2and let the distortion measure be the mean squared error distortion measure. It is shown that for any optimal 2DSC, (a)0 ≤L0≤ 0.8802 and 0 ≤L1,L2≤ 0.3802 ifD0≤D1+D2-σ2; (b) 0 ≤L1,L2≤ 0.4401 ifD0≥ (1/D1+1/D2-1/σ2)-1; (c) 0 ≤L1,L2≤ 0.3802 andR(max{D1,D2})-1 ≤L0≤R(max{D1,D2})+0.3802 for all otherD0. A tighter bound on the distance between the El Gamal-Cover inner bound and the achievable region is also presented. Inspired by these new bounds, a low-complexity composite 2DSC design first developed by Lastras-Montaño and Castelli is extended and the resulting code is shown to be near-optimal for all memoryless sources and all distortions. These results are also extended to MDSCs with an arbitrary number of packets. Finally, Gaussian sources are analyzed in detail to verify these bounds and examine their tightness.

This paper studies the capacity of single-source single-sink noiseless networks under adversarial or arbitrary errors on no more thanzedges. Unlike prior papers, which assume equal capacities on all links, arbitrary link capacities are considered. Results include new upper bounds, network error-correction coding strategies, and examples of network families where our bounds are tight. An example is provided of a network where the capacity is 50% greater than the best rate that can be achieved with linear coding. While coding at the source and sink suffices in networks with equal link capacities, in networks with unequal link capacities, it is shown that intermediate nodes may have to do coding, nonlinear error detection, or error correction in order to achieve the network error-correction capacity.

A family of equivalence tools for bounding network capacities is introduced. Given a networkNwith node setV, the capacity ofNis a set of non-negative vectors with elements corresponding to all possible multicast connections inN; a vector ℜ is in the capacity region forNif and only if it is possible to simultaneously and reliably establish all multicast connections acrossNat the given rates. Any other demand type with independent messages is a special case of this multiple multicast problem, and is therefore included in the given rate region. In Part I, we show that the capacity of a networkNis unchanged if any independent, memoryless, point-to-point channel inNis replaced by a noiseless bit pipe with throughput equal to the removed channel's capacity. It follows that the capacity of a network comprised entirely of such point-to-point channels equals the capacity of an error-free network that replaces each channel by a noiseless bit pipe of the corresponding capacity. A related separation result was known previously for a single multicast connection over an acyclic network of independent, memoryless, point-to-point channels; our result treats general connections (e.g., a collection of simultaneous unicasts) and allows cyclic or acyclic networks.

We consider the problem of zero error source coding with limited feedback when side information is present at the receiver. First, we derive an achievable rate region for arbitrary joint distributions on the source and the side information. When all pairs of source and side information symbols are observed with non-zero probability, we show that this characterization gives the entire rate region. Next, we demonstrate a class of sources for which asymptotically zero feedback suffices to achieve zero-error coding at the rate promised by the Slepian-Wolf bound for asymptotically lossless coding. Finally, we illustrate these results with the aid of three simple examples.

Polar codes have attracted much recent attention as one of the first codes with low computational complexity that provably achieve optimal rate-regions for a large class of information-theoretic problems. One significant drawback, however, is that for current constructions the probability of error decays sub-exponentially in the block-length more detailed designs improve the probability of error at the cost of significantly increased computational complexity. In this work we show how the the classical idea of code concatenation - using "short" polar codes as inner codes and a "high-rate" Reed-Solomon code as the outer code - results in substantially improved performance. In particular, code concatenation with a careful choice of parameters boosts the rate of decay of the probability of error to almost exponential in the block-length with essentially no loss in computational complexity. We demonstrate such performance improvements for three sets of information-theoretic problems - a classical point-to-point channel coding problem, a class of multiple-input multiple output channel coding problems, and some network source coding problems.

This paper derives new outer bounds on the capacities of networks comprised of broadcast and point-to-point channels. The results are tight in some cases, and methods for bounding their error in general are discussed. The results given demonstrate the simplicity of generalizing network coding results to networks of noisy channels using the family of network equivalence tools. The approach taken is not inherently a cut-set approach and frequently yields tighter bounds than those achieved by traditional cut-sets.

This paper proves the separation between source-network coding and channel coding in networks of noisy, discrete, memoryless channels. We show that the set of achievable distortion matrices in delivering a family of dependent sources across such a network equals the set of achievable distortion matrices for delivering the same sources across a distinct network which is built by replacing each channel by a noiseless, point-to-point bit-pipe of the corresponding capacity. Thus a code that applies source-network coding across links that are made almost lossless through the application of independent channel coding across each link asymptotically achieves the optimal performance across the network as a whole.

We consider the delay of network coding compared to routing for a family of simple networks with parallel links. We investigate the sub-linear term in the block delay required for unicasting n packets and show that there is an unbounded gap between network coding and routing. In particular, we show that delay benefit of network coding is scaling at least as fast as √n. The main technical contribution involves showing that the delay function for the routing retransmission strategy is unbounded. This problem is equivalent to computing the expected maximum of two negative binomial random variables. This problem has also been addressed previously and we derive the first exact characterization which might be of independent interest.

Motivated by applications to sensor networks and privacy preserving databases, we consider the problem of functional compression. The objective is to separately compress possibly correlated discrete sources such that an arbitrary but fixed deterministic function of those sources can be computed given the compressed data from each source. We consider both the lossless and lossy computation of a function. Specifically, we present results of the rate regions for three instances of the problem where there are two sources: 1) lossless computation where one source is available at the decoder; 2) under a special condition, lossless computation where both sources are separately encoded; and 3) lossy computation where one source is available at the decoder. For all of these instances, we present a layered architecture for distributed coding: first preprocess data at each source using colorings of certain characteristic graphs and then use standard distributed source coding (a laSlepian and Wolfs scheme) to compress them. For the first instance, our results extend the approach developed by Orlitsky and Roche (2001) in the sense that our scheme requires simpler structure of coloring rather than independent sets as in the previous case. As an intermediate step to obtain these results, we obtain an asymptotic characterization of conditional graph coloring for an OR product of graphs generalizing a result of Korner (1973), which should be of interest in its own right.

We consider three capacity definitions for composite channels with channel side information at the receiver. A composite channel consists of a collection of different channels with a distribution characterizing the probability that each channel is in operation. TheShannon capacityof a channel is the highest rate asymptotically achievable with arbitrarily small error probability. Under this definition, the transmission strategy used to achieve the capacity must achieve arbitrarily small error probability for all channels in the collection comprising the composite channel. The resulting capacity is dominated by the worst channel in its collection, no matter how unlikely that channel is. We, therefore, broaden the definition of capacity to allow for some outage. The capacity versus outage is the highest rate asymptotically achievable with a given probability of decoder-recognized outage. Theexpected capacityis the highest average rate asymptotically achievable with a single encoder and multiple decoders, where channel side information determines the channel in use. The expected capacity is a generalization of capacity versus outage since codes designed for capacity versus outage decode at one of two rates (rate zero when the channel is in outage and the target rate otherwise) while codes designed for expected capacity can decode at many rates. Expected capacity equals Shannon capacity for channels governed by a stationary ergodic random process but is typically greater for general channels. The capacity versus outage and expected capacity definitions relax the constraint that all transmitted information must be decoded at the receiver. We derive channel coding theorems for these capacity definitions through information density and provide numerical examples to highlight their connections and differences. We also discuss the implications of these alternative capacity definitions for end-to-end distortion, source-channel coding, and separation.

We consider source coding over networks with unlimited feedback from the sinks to the sources. We first show examples of networks where the rate region with feedback is a strict superset of that without feedback. Next, we find an achievable region for multiterminal lossy source coding with feedback. Finally, we evaluate this region for the case when one of the sources is fully known at the decoder and use the result to show that this region is a strict superset of the best known achievable region for the problem without feedback.

In this paper, we generalize the lossless coded side information problem from the three-node network of Ahlswede and Korner to more general network scenarios. We derive inner and outer bounds on the achievable rate region in the general network scenario and show that they are tight for some families of networks. Our approach demonstrates how solutions to canonical source coding problems can be used to derive bounds for more complex networks and reveals an interesting connection between networks with side information, successive refinement, and network coding.

We analyze a simple network where a source and a receiver are connected by a line of erasure channels of different reliabilities. Recent prior work has shown that random linear network coding can achieve the min-cut capacity and therefore the asymptotic rate is determined by the worst link of the line network. In this paper we investigate the delay for transmitting a batch of packets, which is a function of all the erasure probabilities and the number of packets in the batch. We show a monotonicity result on the delay function and derive simple expressions which characterize the expected delay behavior of line networks. Further, we use a martingale bounded differences argument to show that the actual delay is tightly concentrated around its expectation.

We prove a strong converse for particular source coding problems: the Ahlswede-Korner (coded side information) problem, lossless source coding for multicast networks with side-information at the end nodes, and the Gray-Wyner problem. Source and side-information sequences are drawn i.i.d. according to a given distribution on a finite alphabet. The strong converse discussed here states that when a given rate vector R is not D-achievable, the probability of observing distortion D for any sequence of block codes at rate R must decrease exponentially to 0 as the block length grows without bound. This strong converse implies the prior strong converses for the point-to-point network, Slepian-Wolf problem, and Ahlswede-Korner (coded side information) problem.

In this paper we derive capacity regions for network error correction with both known and unknown topologies (coherent and non-coherent network coding) under a multiple-source multicast transmission scenario. For the multiple-source non-multicast scenario, given any achievable network code for the error-free case, we construct a code with a reduced rate region for the case with errors.

We describe an equivalence result for network capacity. Roughly, our main result is as follows. Given a network of noisy, independent, memoryless links, a collection of demands can be met on the given network if and only if it can be met on another network where each noisy link is replaced by a noiseless bit pipe with throughput equal to the noisy link capacity. This result was previously known only for multicast connections.

We introduce a game theoretic framework for studying a restricted form of network coding in a general wireless network. The network is fixed and known, and the system performance is measured as the number of wireless transmissions required to meet n unicast demands. Game theory is here employed as a tool for improving distributed network coding solutions. We propose a framework that allows each unicast session to independently adjust his routing decision in response to local information. Specifically, we model the unicast sessions as self-interested decision makers in a noncooperative game. This approach involves designing both local cost functions and decision rules for the unicast sessions so that the resulting collective behavior achieves a desirable system performance in a shared network environment. We compare the performance of the resulting distributed algorithms to the best performance that could be found and implemented using a centralized controller. We focus on the performance of stable solutions - where stability here refers to a form of Nash equilibrium defined below. Results include bounds on the best- and worst-case stable solutions as compared to the optimal centralized solution. We show that our bounds on the best- and worst-case stable performance cannot be improved using cost functions that are independent of the network structure. Results in learning in games prove that the best-case stable solution can be learned by self-interested players with probability approaching 1.

This paper considers the problem, first introduced by Ahlswede and Korner in 1975, of lossless source coding with coded side information. Specifically, letXandYbe two random variables such thatXis desired losslessly at the decoder whileYserves as side information. The random variables are encoded independently, and both descriptions are used by the decoder to reconstructX. Ahlswede and Korner describe the achievable rate region in terms of an auxiliary random variable. This paper gives a partial solution for an optimal auxiliary random variable, thereby describing part of the rate region explicitly in terms of the distribution ofXandY.

We study the transmission of a stationary ergodic Gaussian source over a packet erasure channel, which is a composite channel with degraded states. A broadcast channel code can be applied to a composite channel to obtain different rates in the different channel states. However, we show that a non-broadcast direct transmission strategy achieves a higher expected rate than the broadcast code, although it does not meet Shannon's definition of reliable communication since it does not guarantee which bits will be received. Each channel code has a matching source code: the multiresolution source code allows the broadcast channel code to transmit prioritized information, and the symmetric multiple description source code enables direct transmission of unprioritized information. The end-to-end expected distortions of these schemes are also compared.

We investigate the network source coding rate region for networks with multiple sources and multicast demands in the presence of side information, generalizing earlier results on multicast rate regions without side information. When side information is present only at the terminal nodes, we show that the rate region is precisely characterized by the cut-set bounds and that random linear coding suffices to achieve the optimal performance. When side information is present at a non-terminal node, we present an achievable region. Finally, we apply these results to obtain an inner bound on the rate region for networks with general source-demand structures.

We derive new algorithms for approximating the rate regions for a family of source coding problems that includes lossy source coding, lossy source coding with uncoded side information at the receiver (the Wyner-Ziv problem), and an achievability bound for lossy source coding with coded side in formation at the receiver. The new algorithms generalize a recent approximation algorithm by Go and Effros from lossless to lossy coding. In each case, prior information theoretic descriptions of the desired regions are available but difficult to evaluate for example sources due to their dependence on auxiliary random variables. Our algorithm builds a linear program whose solution is no less than the desired lower bound and no greater than (1 + ∈) times that optimal value. These guarantees are met even when the optimal value is unknown. Here ∈ ≫ 0 is a parameter chosen by the user; the algorithmic complexity grows as O(∈−M) as e approaches 0, where M ≫ 4 is a constant that depends on the source coding problem and the alphabet sizes of the sources.

An information-theoretic approach for detecting Byzantine or adversarial modifications in networks employing random linear network coding is described. Each exogenous source packet is augmented with a flexible number of hash symbols that are obtained as a polynomial function of the data symbols. This approach depends only on the adversary not knowing the random coding coefficients of all other packets received by the sink nodes when designing its adversarial packets. We show how the detection probability varies with the overhead (ratio of hash to data symbols), coding field size, and the amount of information unknown to the adversary about the random code.

Network coding substantially increases network throughput. But since it involves mixing of information inside the network, a single corrupted packet generated by a malicious node can end up contaminating all the information reaching a destination, preventing decoding. This paper introduces distributed polynomial-time rate-optimal network codes that work in the presence of Byzantine nodes. We present algorithms that target adversaries with different attacking capabilities. When the adversary can eavesdrop on all links and jam links, our first algorithm achieves a rate of , where is the network capacity. In contrast, when the adversary has limited eavesdropping capabilities, we provide algorithms that achieve the higher rate of . Our algorithms attain the optimal rate given the strength of the adversary. They are information-theoretically secure. They operate in a distributed manner, assume no knowledge of the topology, and can be designed and implemented in polynomial time. Furthermore, only the source and destination need to be modified; nonmalicious nodes inside the network are oblivious to the presence of adversaries and implement a classical distributed network code. Finally, our algorithms work over wired and wireless networks.

An algorithm for scalar quantizer design on discrete-alphabet sources is proposed. The proposed algorithm can be used to design fixed-rate and entropy-constrained conventional scalar quantizers, multiresolution scalar quantizers, multiple description scalar quantizers, and Wyner-Ziv scalar quantizers. The algorithm guarantees globally optimal solutions for conventional fixed-rate scalar quantizers and entropy-constrained scalar quantizers. For the other coding scenarios, the algorithm yields the best code among all codes that meet a given convexity constraint. In all cases, the algorithm run-time is polynomial in the size of the source alphabet. The algorithm derivation arises from a demonstration of the connection between scalar quantization, histogram segmentation, and the shortest path problem in a certain directed acyclic graph.

We study a model for peer-to-peer file sharing. The goal is to distribute a file from a server to multiple peers. We assume the upload capacity of each peer is the only bottleneck. We examine the finish times of peers under different transmission strategies. Pareto optimality, min-max finish time, and optimal average finish time of the model are studied. We believe the results provide fundamental insights into practical peer-to-peer systems such as BitTorrent.

Abstract — We consider the problem of minimizing the resources used for network coding while achieving the desired throughput in a multicast scenario. Since this problem is NPhard, we seek a method for quickly finding sufficiently good solutions. To this end, we take evolutionary approaches based on a Genetic Algorithm. In this paper, we extend the evolutionary algorithm that we previously proposed in three perspectives. First, whereas the previous algorithm can be applied to only acyclic networks, we devise a modified evaluation method that works also with networks with cycles. Second, we introduce a new set of GA components that in our experiments outperforms the one used in the previous algorithm. Third, we present a new framework of the evolutionary approach, where fitness evaluation and population management are done in a decentralized manner with a limited amount of coordination. The new framework enables a network coding protocol where the resources used for coding are optimized in the setup phase as the proposed evolutionary algorithm being loaded and run at each node of the network. We demonstrate the effectiveness of our algorithms by carrying out simulations on a number of different sets of network topologies. I.

We investigate the network coding capacity for line networks. For independent sources and a special class of dependent sources, we fully characterize the capacity region of line networks for all possible demand structures (e.g., multiple unicast, mixtures of unicasts and multicasts, etc.) Our achievability bound is derived by first decomposing a line network into single-demand components and then adding the component rate regions to get rates for the parent network. For general dependent sources, we give an achievability result and provide examples where the result is and is not tight.

We consider three capacity definitions for general channels with channel side information at the receiver, where the channel is modeled as a sequence of finite dimensional conditional distributions not necessarily stationary, ergodic, or information stable. The Shannon capacity is the highest rate asymptotically achievable with arbitrarily small error probability. The outage capacity is the highest rate asymptotically achievable with a given probability of decoder-recognized outage. The expected capacity is the highest expected rate asymptotically achievable with a single encoder and multiple decoders, where the channel side information determines the decoder in use. Expected capacity equals Shannon capacity for channels governed by a stationary ergodic random process but is typically greater for general channels. These alternative definitions essentially relax the constraint that all transmitted information must be decoded at the receiver. We derive equations for these capacity definitions through information density. Examples are also provided to demonstrate their implications.

The lossless rate region for the coded side information problem is "solved," but its solution is expressed in terms of an auxiliary random variable. As a result, finding the rate region for any fixed example requires an optimization over a family of allowed auxiliary random variables. While intuitive constructions are easy to come by and optimal solutions are known under some special conditions, proving the optimal solution is surprisingly difficult even for examples as basic as a binary source with binary side information. We derive the optimal auxiliary random variables and corresponding achievable rate regions for a family of problems where both the source and side information are binary. Our solution involves first tightening known bounds on the alphabet size of the auxiliary random variable and then optimizing the auxiliary random variable subject to this constraint. The technique used to tighten the bound on the alphabet size applies to a variety of problems beyond the one studied here.

In many problems in wireline networks, it is known that achieving capacity on each link or subnetwork is optimal for the entire network operation. In this paper, we present examples of wireless networks in which decoding and achieving capacity on certain links or subnetworks gives us lower rates than other simple schemes, like forwarding. This implies that the separation of channel and network coding that holds for many classes of wireline networks does not, in general, hold for wireless networks. Next, we consider Gaussian and erasure wireless networks where nodes are permitted only two possible operations: nodes can either decode what they receive (and then re-encode and transmit the message) or simply forward it. We present a simple greedy algorithm that returns the optimal scheme from the exponential-sized set of possible schemes. This algorithm will go over each node at most once to determine its operation, and hence, is very efficient. We also present a decentralized algorithm whose performance can approach the optimum arbitrarily closely in an iterative fashion

We discuss two approaches for decoding at arbitrary rates in the Slepian-Wolf problem - time sharing and source splitting - both of which rely on constituent vertex decoders. We consider the error exponents for both schemes and conclude that source-splitting is more robust at coding at arbitrary rates, as the error exponent for time-sharing degrades significantly at rates near vertices. As a by-product of our analysis, we exhibit an interesting connection between minimum mean-squared error estimation and error exponents

Summary form only given. We study the source coding problem on a simple two-hop network with side information on the middle and end nodes. For the degraded case, where the side information at the end node is weaker than the side information at the middle node, a complete characterization of the rate-distortion region is derived.

For a family of network source coding problems, we prove that the lossless rate region is concave in the distribution of sources. While the proof of concavity is straightforward for the few examples where a single-letter characterization of the lossless source coding region is known, it is more difficult for the vast majority of networks where the lossless source coding region remains unsolved. The class of networks that we investigate includes both solved and unsolved examples. We further conjecture that the same property applies more widely and sketch an avenue for investigating that conjecture

In this paper we consider the per-node run-time complexity of network multicast codes. We show that the randomized algebraic network code design algorithms described extensively in the literature result in codes that on average require a number of operations that scales quadratically with the block-length m of the codes. We then propose an alternative type of linear network code whose complexity scales linearly in m and still enjoys the attractive properties of random algebraic network codes. We also show that these codes are optimal in the sense that any rate-optimal linear network code must have at least a linear scaling in run-time complexity

We describe a new tiling approach for network code design. The proposed method applies dynamic programming to find the best strategy among a restricted collection of network codes. We demonstrate the proposed strategy as a method for efficiently accommodating multiple unicasts in a wireless coding environment on a triangular lattice and discuss its generalization.

This paper considers the problem, first introduced by Ahlswede and Körner in 1975, of lossless source coding with coded side information. Specifically, let X and Y be two random variables such that X is desired losslessly at the decoder while Y serves as side information. The random variables are encoded independently, and both descriptions are used by the decoder to reconstruct X. Ahlswede and Körner describe the achievable rate region in terms of an auxiliary random variable. This paper gives a partial solution for the optimal auxiliary random variable, thereby describing part of the rate region explicitly in terms of the distribution of X and Y.

This paper discusses the Slepian-Wolf problem of distributed near-lossless compression of correlated sources. We introduce practical new tools for communicating at all rates in the achievable region. The technique employs a simple "source-splitting" strategy that does not require common sources of randomness at the encoders and decoders. This approach allows for pipelined encoding and decoding so that the system operates with the complexity of a single user encoder and decoder. Moreover, when this splitting approach is used in conjunction with iterative decoding methods, it produces a significant simplification of the decoding process. We demonstrate this approach for synthetically generated data. Finally, we consider the Slepian-Wolf problem when linear codes are used as syndrome-formers and consider a linear programming relaxation to maximum-likelihood (ML) sequence decoding. We note that the fractional vertices of the relaxed polytope compete with the optimal solution in a manner analogous to that observed when the "min-sum" iterative decoding algorithm is applied. This relaxation exhibits the ML-certificate property: if an integral solution is found, it is the ML solution. For symmetric binary joint distributions, we show that selecting easily constructable "expander"-style low-density parity check codes (LDPCs) as syndrome-formers admits a positive error exponent and therefore provably good performance

In this paper, a special class of wireless networks, called wireless erasure networks, is considered. In these networks, each node is connected to a set of nodes by possibly correlated erasure channels. The network model incorporates the broadcast nature of the wireless environment by requiring each node to send the same signal on all outgoing channels. However, we assume there is no interference in reception. Such models are therefore appropriate for wireless networks where all information transmission is packetized and where some mechanism for interference avoidance is already built in. This paper looks at multicast problems over these networks. The capacity under the assumption that erasure locations on all the links of the network are provided to the destinations is obtained. It turns out that the capacity region has a nice max-flow min-cut interpretation. The definition of cut-capacity in these networks incorporates the broadcast property of the wireless medium. It is further shown that linear coding at nodes in the network suffices to achieve the capacity region. Finally, the performance of different coding schemes in these networks when no side information is available to the destinations is analyzed

In this correspondence, we consider rate-distortion examples in the presence of side information. For a system with some side information known at both the encoder and decoder, and some known only at the decoder, we evaluate the rate distortion function for both Gaussian and binary sources. While the Gaussian example is a straightforward generalization of the corresponding result by Wyner, the binary example proves more difficult and is solved using a multidimensional optimization approach. Leveraging the insights gained from the binary example, we then solve the more complicated binary Heegard and Berger problem of decoding when side information may be present. The results demonstrate the existence of a new type of successive refinement in which the refinement information is decoded together with side information that is not available for the initial description.

We present a distributed random linear network coding approach for transmission and compression of information in general multisource multicast networks. Network nodes independently and randomly select linear mappings from inputs onto output links over some field. We show that this achieves capacity with probability exponentially approaching 1 with the code length. We also demonstrate that random linear coding performs compression when necessary in a network, generalizing error exponents for linear Slepian-Wolf coding in a natural way. Benefits of this approach are decentralized operation and robustness to network changes or link failures. We show that this approach can take advantage of redundant network capacity for improved success probability and robustness. We illustrate some potential advantages of random linear network coding over routing in two examples of practical scenarios: distributed network operation and networks with dynamically varying connections. Our derivation of these results also yields a new bound on required field size for centralized network coding on general multicast networks

This correspondence considers the problem of distributed source coding of multiple sources over a network with multiple receivers. Each receiver seeks to reconstruct all of the original sources. The work by Ho et al. 2004 demonstrates that random network coding can solve this problem at the potentially high cost of jointly decoding the source and the network code. Motivated by complexity considerations we consider the performance of separate source and network codes. Previous work by Effros et al. 2003 demonstrates the failure of separation between source and network codes for nonmulticast networks. We demonstrate that failure for multicast networks. We study networks with capacity constraints on edges. It is shown that the problem with two sources and two receivers is always separable. Counterexamples are presented for other cases.

Minimum-entropy decoding is a universal decoding algorithm used in decoding block compression of discrete memoryless sources as well as block transmission of information across discrete memoryless channels. Extensions can also be applied for multiterminal decoding problems, such as the Slepian-Wolf source coding problem. The 'method of types' has been used to show that there exist linear codes for which minimum-entropy decoders achieve the same error exponent as maximum-likelihood decoders. Since minimum-entropy decoding is NP-hard in general, minimum-entropy decoders have existed primarily in the theory literature. We introduce practical approximation algorithms for minimum-entropy decoding. Our approach, which relies on ideas from linear programming, exploits two key observations. First, the 'method of types' shows that that the number of distinct types grows polynomially in n. Second, recent results in the optimization literature have illustrated polytope projection algorithms with complexity that is a function of the number of vertices of the projected polytope. Combining these two ideas, we leverage recent results on linear programming relaxations for error correcting codes to construct polynomial complexity algorithms for this setting. In the binary case, we explicitly demonstrate linear code constructions that admit provably good performance.

Summary form only given. In this paper, we bound the rate-distortion region for a four-node network. The results are the first known expansion of rate-distortion theory from single-hop networks (every source has a direct connection to each of its destinations), to multihop networks, which allow intermediate nodes. While single-hop network source coding solutions may be applied in multihop networks, such applications require explicit rate allocation for each source-destination pair, and the resulting solutions may be suboptimal. We therefore tackle the multihop network source coding problem directly using a diamond network.

We address practical coding schemes for the Slepian-Wolf distributed data compression problem. We consider three approaches. First, we apply a source-splitting technique to code at any rate in the achievable rate region with low complexity. It is well known that vertices in the achievable rate region can be implemented with low complexity. The source-splitting approach transforms any achievable rate point into a vertex in a higher-dimensional Slepian-Wolf achievable rate region. Secondly, we consider linear programming relaxations of the maximum-likelihood decoding problem. We give a polynomial complexity construction for linear codes with a certificate property. Lastly, when the decoder does not have any knowledge of the source statistics, we present practical schemes for universal decoding, a topic heretofore confined primarily to theory.

We show that the deterministic broadcast channel, where a single source transmits to M receivers across a deterministic mechanism, may be reduced, via a rate-splitting transformation, to another (2M - 1)-receiver deterministic broadcast channel problem where a successive encoding approach suffices. Analogous to rate-splitting for the multiple access channel and source-splitting for the Slepian-Wolf problem, all achievable rates (including non-vertices) apply. This amounts to significant complexity reduction at the encoder

In this paper, we first define and bound the rate loss of source codes for broadcast channels. Our broadcast channel model comprises one transmitter and two receivers; the transmitter is connected to each receiver by a private channel and to both receivers by a common channel. The transmitter sends a description of source (X, Y) through these channels, receiver 1 reconstructs X with distortion D1, and receiver 2 reconstructs Y with distortion D2. Suppose the rates of the common channel and private channels 1 and 2 are R0, R1, and R2, respectively. The work of Gray and Wyner gives a complete characterization of all achievable rate triples (R0,R1,R2) given any distortion pair (D1,D2). In this paper, we define the rate loss as the gap between the achievable region and the outer bound composed by the rate-distortion functions, i.e., R0+R1+R2 ≥ RX,Y (D1,D2), R0 + R1 ≥ RX(D1), and R0 + R2 ≥ RY (D2). We upper bound the rate loss for general sources by functions of distortions and upper bound the rate loss for Gaussian sources by constants, which implies that though the outer bound is generally not achievable, it may be quite close to the achievable region. This also bounds the gap between the achievable region and the inner bound proposed by Gray and Wyner and bounds the performance penalty associated with using separate decoders rather than joint decoders. We then construct such source codes using entropy-constrained dithered quantizers. The resulting implementation has low complexity and performance close to the theoretical optimum. In particular, the gap between its performance and the theoretical optimum can be bounded from above by constants for Gaussian sources.

We derive the rate-distortion region for source coding on a simple multi-hop network with side information. The result represents the first complete solution to a multi-hop source coding problem. The proof technique combines ideas from Wyner-Ziv coding and coding with unreliable side information

We design codes to transmit information over a network, some subset of which is controlled by a malicious adversary. The computationally unbounded, hidden adversary knows the message to be transmitted, and can observe and change information over the part of the network being controlled. The network nodes do not share resources such as shared randomness or a private key. We first consider a unicast problem in a network with |epsiv parallel, unit-capacity, directed edges. The rate-region has two parts. If the adversary controls a fraction p < 0.5 of the |epsiv edges, the maximal throughput equals (1 - p) |epsiv|. We describe low-complexity codes that achieve this rate-region. We then extend these results to investigate more general multicast problems in directed, acyclic networks

In "On Coding for Reliable Communication over Packet Networks" (Lun, Medard, and Effros, Proc. 42nd Annu. Allerton Conf. Communication, Control, and Computing, 2004), a capacity-achieving coding scheme for unicast or multicast over lossy wireline or wireless packet networks is presented. We extend that paper's results in two ways: First, we extend the network model to allow packets received on a link to arrive according to any process with an average rate, as opposed to the assumption of Poisson traffic with i.i.d. losses that was previously made. Second, in the case of Poisson traffic with i.i.d. losses, we derive error exponents that quantify the rate at which the probability of error decays with coding delay

We motivate the consideration of deterministic broadcast channel coding as an interference management technique in wireless scenarios. We address practical coding strategies for such channels and discuss two approaches. The first relies upon enumerative source coding and can be applied for any deterministic broadcast channel problem as the first step in pipelined encoding for vertex rates. The second approach addresses a wireless interference management scenario and is a complete, practical, capacity-achieving strategy that dualizes the Luby transform code construction and encoding/decoding algorithms. This results in the first practical, nontrivial, capacity achieving code construction for the deterministic broadcast channel.

The rate loss of a multiresolution source code (MRSC) describes the difference between the rate needed to achieve distortion D/sub i/ in resolution i and the rate-distortion function R(D/sub i/). This paper generalizes the rate loss definition to multiple description source codes (MDSCs) and bounds the MDSC rate loss for arbitrary memoryless sources. For a two-description MDSC (2DSC), the rate loss of description i with distortion D/sub i/ is defined as L/sub i/=R/sub i/-R(D/sub i/), i=1,2, where R/sub i/ is the rate of the ith description; the joint rate loss associated with decoding the two descriptions together to achieve central distortion D/sub 0/ is measured either as L/sub 0/=R/sub 1/+R/sub 2/-R(D/sub 0/) or as L/sub 12/=L/sub 1/+L/sub 2/. We show that for any memoryless source with variance /spl sigma//sup 2/, there exists a 2DSC for that source with L/sub 1//spl les/1/2 or L/sub 2//spl les/1/2 and a) L/sub 0//spl les/1 if D/sub 0//spl les/D/sub 1/+D/sub 2/-/spl sigma//sup 2/, b) L/sub 12//spl les/1 if 1/D/sub 0//spl les/1/D/sub 1/+1/D/sub 2/-1//spl sigma//sup 2/, c) L/sub 0//spl les/L/sub G0/+1.5 and L/sub 12//spl les/L/sub G12/+1 otherwise, where L/sub G0/ and L/sub G12/ are the joint rate losses of a Gaussian source with variance /spl sigma//sup 2/.

The famous max-flow min-cut theorem states that a source node s can send information through a network (V, E) to a sink node t at a rate determined by the min-cut separating s and t. Recently, it has been shown that this rate can also be achieved for multicasting to several sinks provided that the intermediate nodes are allowed to re-encode the information they receive. We demonstrate examples of networks where the achievable rates obtained by coding at intermediate nodes are arbitrarily larger than if coding is not allowed. We give deterministic polynomial time algorithms and even faster randomized algorithms for designing linear codes for directed acyclic graphs with edges of unit capacity. We extend these algorithms to integer capacities and to codes that are tolerant to edge failures.

We introduce three new innovations for compression using LDPCs for the Slepian-Wolf problem. The first is a general iterative Slepian-Wolf decoding algorithm that incorporates the graphical structure of all the encoders and operates in a 'turbo-like' fashion. The second innovation introduces source-splitting to enable low-complexity pipelined implementations of Slepian-Wolf decoding at rates besides corner points of the Slepian-Wolf region. This innovation can also be applied to single-source block coding for reduced decoder complexity. The third approach is a linear programming relaxation to maximum-likelihood sequence decoding that exhibits the ML-certificate property. This can be used for decoding a single binary block-compressed source as well as decoding at vertex points for the binary Slepian-Wolf problem. All three of these innovations were motivated by recent analogous results in the channel coding domain.

In this paper, we build multiresolution source codes using entropy constrained dithered scalar quantizers. We demonstrate that for n-dimensional random vectors, dithering followed by uniform scalar quantization and then by entropy coding achieves performance close to the n-dimensional optimum for a multiresolution source code. Based on this result, we propose a practical code design algorithm and compare its performance with that of the set partitioning in hierarchical trees (SPIHT) algorithm on natural images.

It is shown that achieving an arbitrary rate-point in the achievable region of the M-source Slepian-Wolf [1] problem may be reduced via a practical source-splitting transformation to achieving a corner point in a 2M - 1 source Slepian-Wolf problem. Moreover, each source must be split at most once. This approach extends the ideas introduced in [2] to a practical setting: it does not require common randomness shared between splitters and the decoders, the cardinality of each source split is strictly smaller than the original, and practical iterative decoding methods can achieve rates near the theoretical bound.

We present a new algorithm for fixed-rate vector quantizer (VQ) design with deterministic data net. The algorithm also performs efficient VQ design for simply characterized continuous distributions. The algorithm also serves as an approximation algorithm for the d-dimensional fixed-rate operational distortion-rate function, extends to a variety of network VQ problems. The algorithm generalizes to give /spl epsiv/-approximation algorithms for many network VQ design problems. A few examples are multiresolution VQ (MRVQ), multiple description VQ (MDVQ), side information VQ (SIVQ), Broadcast VQ (BCVQ), joint source-channel VQ (JSCVQ) and remote source VQ (RSVQ).

We determine the capacity of a certain class of wireless erasure relay networks. We first find a suitable definition for the "cut-capacity" of erasure networks with broadcast at transmission and no interference at reception. With this definition, a maxflow mincut capacity result holds for the capacity of these networks.

Distributed randomized network coding, a robust approach to multicasting in distributed network settings, can be extended to provide Byzantine modification detection without the use of cryptographic functions is presented in this paper.

The problem of truly-lossless (Pe=0) distributed source coding requires knowledge of the joint statistics of the sources. In particular the locations of the zeroes of the probability mass functions (pmfs) are crucial for encoding at rates below (H(X),H(Y)). We consider the distributed computation of the empirical joint pmf Pnof a sequence of random variable pairs observed at physically separated nodes of a network. We consider both worst-case and average measures of information exchange and treat both exact calculation of Pnand a notion of approximation. We find that in all cases the communication cost grows linearly with the size of the input. Further, we consider the problem of determining whether the empirical pmf has a zero in a particular location and show that in most cases considered this also requires a communication cost that is linear in the input size

The Slepian-Wolf bound raises interest in lossless code design for multiple access networks. Previous work treats instantaneous codes. We generalize the Sardinas and Patterson test and bound the achievable rate region for uniquely decodable codes. The Kraft inequality is generalised to produce the necessary conditions on the codeword lengths for uniquely decodable-side information source code.

We compare two strategies for lossy source description across a pair of unreliable channels. In the first strategy, we use a broadcast channel code to achieve a different rate for each possible channel realization, and then use a multiresolution source code to describe the source at the resulting rates. In the second strategy, we use a channel coding strategy for two independent channels coupled with a multiple description source code. In each case, we choose the coding parameters to minimize the expected end-to-end distortion in the source reconstruction. We demonstrate that in point-to-point communication across a pair of non-ergodic channels, multiple description coding can provide substantial gains relative to multiresolution and broadcast coding. We then investigate this comparison in a simple MIMO channel. We demonstrate the inferior performance of space time coding with multiresolution source coding and broadcast channel coding relative to multiple description codes and a time sharing channel coding strategy. These results indicate that for non-ergodic channels, the traditional definition of channel capacity does not necessarily lead to the best channel code from the perspective of end-to-end source distortion.

Multiresolution source codes are data compression algorithms yielding embedded source descriptions. The decoder of a multiresolution code can build a source reproduction by decoding the embedded bit stream in part or in whole. All decoding procedures start at the beginning of the binary source description and decode some fraction of that string. Decoding a small portion of the binary string gives a low-resolution reproduction; decoding more yields a higher resolution reproduction; and so on. Multiresolution vector quantizers are block multiresolution source codes. This paper introduces algorithms for designing fixed- and variable-rate multiresolution vector quantizers. Experiments on synthetic data demonstrate performance close to the theoretical performance limit. Experiments on natural images demonstrate performance improvements of up to 8 dB over tree-structured vector quantizers. Some of the lessons learned through multiresolution vector quantizer design lend insight into the design of more sophisticated multiresolution codes.

We examine the performance of the Karhunen-Loeve transform (KLT) for transform coding applications. The KLT has long been viewed as the best available block transform for a system that orthogonally transforms a vector source, scalar quantizes the components of the transformed vector using optimal bit allocation, and then inverse transforms the vector. This paper treats fixed-rate and variable-rate transform codes of non-Gaussian sources. The fixed-rate approach uses an optimal fixed-rate scalar quantizer to describe the transform coefficients; the variable-rate approach uses a uniform scalar quantizer followed by an optimal entropy code, and each quantized component is encoded separately. Earlier work shows that for the variable-rate case there exist sources on which the KLT is not unique and the optimal quantization and coding stage matched to a "worst" KLT yields performance as much as 1.5 dB worse than the optimal quantization and coding stage matched to a "best" KLT. In this paper, we strengthen that result to show that in both the fixed-rate and the variable-rate coding frameworks there exist sources for which the performance penalty for using a "worst" KLT can be made arbitrarily large. Further, we demonstrate in both frameworks that there exist sources for which even a best KLT gives suboptimal performance. Finally, we show that even for vector sources where the KLT yields independent coefficients, the KLT can be suboptimal for fixed-rate coding.

We present an algorithm for designing locally optimal vector quantizers for general networks. We discuss the algorithm's implementation and compare the performance of the resulting "network vector quantizers" to traditional vector quantizers (VQs) and to rate-distortion (R-D) bounds where available. While some special cases of network codes (e.g., multiresolution (MR) and multiple description (MD) codes) have been studied in the literature, we here present a unifying approach that both includes these existing solutions as special cases and provides solutions to previously unsolved examples.

The performance of the KLT for transform coding applications was examined. The KLT has long been viewed as the best available block transform for transform coding. The fixed-rate and variable-rate transform codes were also presented. The fixed-rate approach uses an optimal fixed-rate scalar quantizer to describe the transform coefficients; the variable-rate approach uses a uniform scalar quantizer followed by an optimal entropy code. Earlier work shows that for the variable-rate case, there exist sources on which the KLT is not unique and the optimal transform code matched to a "worst" KLT yields performance as much as 1.5 dB worse than the optimal transform code matched to a "best" KLT. The results were strengthened to show that in both the fixed-rate and the variable-rate coding frameworks, there exist sources for which the performance penalty for using a "worst" KLT can be made arbitrarily large. Further demonstrations in both frameworks show that there exist sources for which even a best KLT gives suboptimal performance. Finally, the results show that even for vector sources where the KLT yields independent coefficients, the KLT can be suboptimal for fixed-rate coding.

Summary form only given. Assuming the squared error distortion measure, the performance achieved is bounded by using scalar entropy constrained dithered quantization (SECDQ) to build multi-resolution (MR), multiple access (MA) and broadcast system (BS) source codes. The resulting performances for arbitrary source distribution are discussed.

The instantaneous side of information source code (SISC) design is considered. In the SISC configuration, the encoder describes source X to the decoder; the decoder uses this description and side information Y, to reconstruct X. Prior work on lossless and near-lossless SISC design demonstrates that globally optimal design is NP-hard. A family of polynomial complexity code design algorithms is introduced that approximates the optimal solution for lossless and near-lossless SISCs. The algorithm may be used to design both Huffman and arithmetic SISCs for an arbitrary probability mass function p(x,y). Experimental results comparing the resulting performances to each other and to the theoretical limit are included.

We present new bounds for the rate loss of multiresolution source codes (MRSCs). Considering an M-resolution code, the rate loss at the ith resolution with distortion D/sub i/ is defined as L/sub i/=R/sub i/-R(D/sub i/), where R/sub i/ is the rate achievable by the MRSC at stage i. This rate loss describes the performance degradation of the MRSC compared to the best single-resolution code with the same distortion. For two-resolution source codes, there are three scenarios of particular interest: (i) when both resolutions are equally important; (ii) when the rate loss at the first resolution is 0 (L/sub 1/=0); (iii) when the rate loss at the second resolution is 0 (L/sub 2/=0). The work of Lastras and Berger (see ibid., vol.47, p.918-26, Mar. 2001) gives constant upper bounds for the rate loss of an arbitrary memoryless source in scenarios (i) and (ii) and an asymptotic bound for scenario (iii) as D/sub 2/ approaches 0. We focus on the squared error distortion measure and (a) prove that for scenario (iii) L/sub 1/<1.1610 for all D/sub 2/<D/sub 1/; (b) tighten the Lastras-Berger bound for scenario (ii) from L/sub 2//spl les/1 to L/sub 2/<0.7250; (c) tighten the Lastras-Berger bound for scenario (i) from L/sub i//spl les/1/2 to L/sub i/<0.3802, i/spl isin/{1,2}; and (d) generalize the bounds for scenarios (ii) and (iii) to M-resolution codes with M/spl ges/2. We also present upper bounds for the rate losses of additive MRSCs (AMRSCs). An AMRSC is a special MRSC where each resolution describes an incremental reproduction and the kth-resolution reconstruction equals the sum of the first k incremental reproductions. We obtain two bounds on the rate loss of AMRSCs: one primarily good for low-rate coding and another which depends on the source entropy.

A multiple access source code (MASC) is a source code designed for the following network configuration: a pair of correlated information sequences {X/sub i/}/sub i=1//sup /spl infin// and {Y/sub i/}/sub i=1//sup /spl infin// is drawn independent and identically distributed (i.i.d.) according to joint probability mass function (p.m.f.) p(x,y); the encoder for each source operates without knowledge of the other source; the decoder jointly decodes the encoded bit streams from both sources. The work of Slepian and Wolf describes all rates achievable by MASCs of infinite coding dimension (n/spl rarr//spl infin/) and asymptotically negligible error probabilities (P/sub e//sup (n)//spl rarr/0). In this paper, we consider the properties of optimal instantaneous MASCs with finite coding dimension (n

We consider the properties of optimal fixed-rate and entropy-constrained scalar quantizers for finite alphabet sources. In particular, we consider conditions under which the optimal scalar quantizer with contiguous codecells achieves performance no worse than the optimal scalar quantizer without the constraint of codecell contiguity. In addition to traditional scalar quantizers, we consider multi-resolution scalar quantizers and multiple description scalar quantizers and also look briefly at codes with decoder side information (Wyner-Ziv codes). While the conditions under which codecell contiguity is consistent with optimality in fixed-rate and entropy-constrained scalar quantization are quite broad, even with the squared error distortion measure, codecell contiguity in fixed-rate and entropy-constrained multi-resolution, multiple description, and Wyner-Ziv scalar quantization can preclude optimality for some sources.

We propose a polynomial-time algorithm for optimal scalar quantizer design on discrete-alphabet sources. Special cases of the proposed approach yield optimal design algorithms for fixed-rate and entropy-constrained scalar quantizers, multi-resolution scalar quantizers, multiple description scalar quantizers, and Wyner-Ziv scalar quantizers. The algorithm guarantees globally optimal solutions for fixed-rate and entropy-constrained scalar quantizers and constrained optima for the other coding scenarios. We derive the algorithm by demonstrating the connection between scalar quantization, histogram segmentation, and the shortest path problem in a certain directed acyclic graph.

We examine the rate-distortion performance and computational complexity of linear transforms for lossy data compression. The goal is to better understand the performance/complexity tradeoffs associated with using the Karhunen-Loeve transform (KLT) and its fast approximations. Since the optimal transform for transform coding is unknown in general, we investigate the performance penalties associated with using the KLT by examining cases where the KLT fails, developing a new transform that corrects the KLT's failures in those examples, and then empirically testing the performance difference between this new transform and the KLT. Experiments demonstrate that while the worst KLT can yield transform coding performance at least 3 dB worse than that of alternative block transforms, the performance penalty associated with using the KLT on real data sets seems to be significantly smaller, giving at most 0.5 dB difference in our experiments. The KLT and its fast variations studied here range in complexity requirements from O(n(2)) to O(n log n) in coding vectors of dimension n. We empirically investigate the rate-distortion performance tradeoffs associated with traversing this range of options. For example, an algorithm with complexity O(n(3/2)) and memory O(n) gives 0.4 dB performance loss relative to the full KLT in our image compression experiments.

The Burrows Wheeler transform (1994) is a reversible sequence transformation used in a variety of practical lossless source-coding algorithms. In each, the BWT is followed by a lossless source code that attempts to exploit the natural ordering of the BWT coefficients. BWT-based compression schemes are widely touted as low-complexity algorithms giving lossless coding rates better than those of the Ziv-Lempel codes (commonly known as LZ'77 and LZ'78) and almost as good as those achieved by prediction by partial matching (PPM) algorithms. To date, the coding performance claims have been made primarily on the basis of experimental results. This work gives a theoretical evaluation of BWT-based coding. The main results of this theoretical evaluation include: (1) statistical characterizations of the BWT output on both finite strings and sequences of length n /spl rarr/ /spl infin/, (2) a variety of very simple new techniques for BWT-based lossless source coding, and (3) proofs of the universality and bounds on the rates of convergence of both new and existing BWT-based codes for finite-memory and stationary ergodic sources. The end result is a theoretical justification and validation of the experimentally derived conclusions: BWT-based lossless source codes achieve universal lossless coding performance that converges to the optimal coding performance more quickly than the rate of convergence observed in Ziv-Lempel style codes and, for some BWT-based codes, within a constant factor of the optimal rate of convergence for finite-memory sources.

A network source code is an optimal source code for a network. To design network source codes, we require each node to have a single encoder, which jointly encodes all messages transmitted by that node, and a single decoder, which jointly decodes all messages arriving at that node. Given a distribution over the sources, the design of the network source code jointly optimizes all encoders and decoders to obtain the best performance with respect to a user-defined priority schedule over the rates and distortions of the system. In this paper we focus on fixed-rate codes and address the implementation of an existing design algorithm for optimal network vector quantizers. Implementing the design algorithm is not straightforward since each encoder must choose its reproduction based on the expected behavior of sources that are unknown to it. We describe a new implementation approach and demonstrate its performance on a three-node network. In addition, we extend the design algorithm to allow the decoder at each node to use side information (specifically, the messages that are to be encoded by the encoder at the same node).

A multiple access source code (MASC) is a source code designed for the following network configuration: a pair of correlated information sequences {X/sub i/}/sub i=1//sup /spl infin// and {Y/sub i/}/sub i=1//sup /spl infin// is drawn i.i.d. according to the joint probability mass function (p.m.f.) p(x,y); the encoder for each source operates without knowledge of the other source; the decoder jointly decodes the encoded bit streams from both sources. The work of Slepian and Wolf (1973) describes all rates achievable by MASCs with arbitrarily small but non-zero error probabilities but does not address truly lossless coding or code design. We consider practical code design for lossless and near lossless MASCs. We generalize the Huffman and arithmetic code design algorithms to attain the corresponding optimal MASC codes for arbitrary p.m.f. p(x,y). Experimental results comparing the optimal achievable rate region to the Slepian-Wolf region are included.

A multiresolution source code is a single code giving an embedded source description that can be read at a variety of rates and thereby yields reproductions at a variety of resolutions. The resolution of a source reproduction here refers to the accuracy with which it approximates the original source. Thus, a reproduction with low distortion is a "high-resolution" reproduction while a reproduction with high distortion is a "low-resolution" reproduction. This paper treats the generalization of universal lossy source coding from single-resolution source codes to multiresolution source codes. Results described in this work include new definitions for weakly minimax universal, strongly minimax universal, and weighted universal sequences of fixed- and variable-rate multiresolution source codes that extend the corresponding notions from lossless coding and (single-resolution) quantization to multiresolution quantizers. A variety of universal multiresolution source coding results follow, including necessary and sufficient conditions for the existence of universal multiresolution codes, rate of convergence bounds for universal multiresolution coding performance to the theoretical bound, and a new multiresolution approach to two-stage universal source coding.

We derive the capacity region for a broadcast channel with intersymbol interference (ISI) and colored Gaussian noise under an input power constraint. The region is obtained by first defining a similar channel model, the circular broadcast channel, which can be decomposed into a set of parallel degraded broadcast channels. The capacity region for parallel degraded broadcast channels is known. We then show that the capacity region of the original broadcast channel equals that of the circular broadcast channel in the limit of infinite block length, and we obtain an explicit formula for the resulting capacity region. The coding strategy used to achieve each point on the convex hull of the capacity region uses superposition coding on some or all of the parallel channels and dedicated transmission on the others. The optimal power allocation for any point in the capacity region is obtained via a multilevel water-filling. We derive this optimal power allocation and the resulting capacity region for several broadcast channel models.

This work combines a new fast context-search algorithm with the lossless source coding models of PPM to achieve a lossless data compression algorithm with the linear context-search complexity and memory of BWT and Ziv-Lempel codes and the compression performance of PPM-based algorithms. Both sequential and nonsequential encoding are considered. The proposed algorithm yields an average rate of 2.27 bits per character (bpc) on the Calgary corpus, comparing favorably to the 2.33 and 2.34 bpc of PPM5 and PPM/sup */ and the 2.43 bpc of BW94 but not matching the 2.12 bpc of PPMZ9, which, at the time of this publication, gives the greatest compression of all algorithms reported on the Calgary corpus results page. The proposed algorithm gives an average rate of 2.14 bpc on the Canterbury corpus. The Canterbury corpus Web page gives average rates of 1.99 bpc for PPMZ9, 2.11 bpc for PPM5, 2.15 bpc for PPM7, and 2.23 bpc for BZIP2 (a BWT-based code) on the same data set.

Multiple description codes are data compression algorithms designed with the goal of minimizing the distortion caused by data loss in packet-based or diversity communications systems. Recently, techniques that achieve multiple description coding by combining embedded source codes with unequal error protection channel codes have become popular in the literature. These codes allow for data reconstruction with any subset of the transmitted packets and achieve progressively better source reconstructions as more and more packets are decoded. The given methods may be applied to any embedded source description. While applicability to all embedded source codes provides great flexibility, this separation approach begs the question of whether better performance could be achieved by taking advantage of the internal structure of a particular embedded code. In this paper, we investigate an extremely simple method for using an embedded source code's internal state information in the construction of a multiple description code. In particular, we protect an embedded SPIHT bitstream by adding to that bitstream periodic descriptions of state information from the encoder, and we demonstrate how the state information can be used to recover lost bits. For low probabilities of network packet loss, the proposed algorithm achieves performance within 0.35 dB of the performance of a more sophisticated channel coding algorithm when both algorithms are applied to same SPIHT embedded source code.

Broadcast systems are a class of networks where one system node (transmitter) simultaneously sends both common and independent, information to multiple nodes (receivers) in the system. Compressing the messages transmitted in such systems using traditional (single-transmitter, single-receiver) techniques requires use of a collection of independent source codes, one for each message sent through the system. The result of this approach is a system with multiple independent encoders at the transmitter and multiple independent decoders at each receiver. An alternative approach is to design a single joint encoder at the transmitter and a single joint decoder at each receiver. We call the resulting code a "broadcast system source code". This paper treats the theory and practice of optimal lossless and lossy (fixed- and variable-rate) broadcast system source codes. The results given include: theoretical limits for lossless source code performance on broadcast systems; an optimal lossless source code design algorithm; an optimal lossy source code design algorithm that generalizes the generalized Lloyd algorithm to broadcast systems; and experimental results for fixed- and variable-rate code performance in a two-receiver broadcast system.

We study the performance of joint source and channel codes designed to minimize end-to-end distortion over a Rayleigh fading channel. We consider two joint code designs. The first joint code uses a sequential design: a standard vector quantizer (VQ) source code is designed for a perfect channel (noiseless and distortionless) and then an RCPC channel code is optimized relative to the VQ and the channel statistics. The second design jointly optimizes a channel optimized VQ (COVQ) and an RCPC channel code through an iterative design process. We consider both hard-decision and soft-decision decoding for the channel codes. In both designs the bit allocation between the source and channel codes is optimized. At this optimal bit allocation, the performance of the iterative joint design and the simpler sequential design are nearly the same over the range of SNR values that we considered. Both code designs outperform standard COVQ and by up to 6 dB, and this performance improvement is most pronounced at low SNRs.

This paper introduces a new data compression algorithm. The goal underlying this new code design is to achieve a single lossless compression algorithm with the excellent compression ratios of the prediction by partial mapping (PPM) algorithms and the low complexity of codes based on the Burrows Wheeler Transform (BWT). Like the BWT-based codes, the proposed algorithm requires worst case O(n) computational complexity and memory; in contrast, the unbounded-context PPM algorithm, called PPM*, requires worst case O(n/sup 2/) computational complexity. Like PPM*, the proposed algorithm allows the use of unbounded contexts. Using standard data sets for comparison, the proposed algorithm achieves compression performance better than that of the BWT-based codes and comparable to that of PPM*. In particular, the proposed algorithm yields an average rate of 2.29 bits per character (bpc) on the Calgary corpus; this result compares favorably with the 2.33 and 2.34 bpc of PPM5 and PPM* (PPM algorithms), the 2.43 bpc of BW94 (the original BWT-based code), and the 3.64 and 2.69 bpc of compress and gzip (popular Unix compression algorithms based on Lempel-Ziv (LZ) coding techniques) on the same data set. The given code does not, however, match the best reported compression performance-2.12 bpc with PPMZ9-listed on the Calgary corpus results web page at the time of this publication. Results on the Canterbury corpus give a similar relative standing. The proposed algorithm gives an average rate of 2.15 bpc on the Canterbury corpus, while the Canterbury corpus web page gives average rates of 1.99 bpc for PPMZ9, 2.11 bpc for PPM5, 2.15 bpc for PPM7, 2.23 bpc for BZIP2 (a popular BWT-based code), and 3.31 and 2.53 bpc for compress and gzip, respectively.

We here consider a theoretical evaluation of data compression algorithms based on the Burrows Wheeler transform (BWT). The main contributions include a variety of very simple new techniques for BWT-based universal lossless source coding on finite-memory sources and a set of new rate of convergence results for BWT-based source codes. The result is a theoretical validation and quantification of the earlier experimental observation that BWT-based lossless source codes give performance better than that of Ziv-Lempel-style codes and almost as good as that of prediction by partial mapping (PPM) algorithms.

Packet-based data communication systems suffer from packet loss under high network traffic conditions. As a result, the receiver is often left with an incomplete description of the requested data. Multiple description source coding addresses the problem of minimizing the expected distortion caused by packet loss. An equivalent problem is that of source coding for data transmission over multiple channels where each channel has some probability of breaking down. Recent work in practical multiple description coding explores the design of multiple description scalar and vector quantizers for the case of two channels or packets. This paper presents a new practical algorithm, based on a ternary tree structure, for the design of both fixed- and variable-rate multiple description vector quantizers for an arbitrary number of channels. Experimental results achieved by codes designed with this algorithm show that they perform well under a wide range of packet loss scenarios.

The weighted universal transform code (WUTC) is a two-stage transform code that replaces JPEG's single, non-optimal transform code with a jointly designed collection of transform codes to achieve good performance across a broader class of possible sources. Unfortunately, the performance gains of WUTC are achieved at the expense of significant increases in computational complexity and larger codes. We here present a faster, more space-efficient WUTC algorithm. The new algorithm uses separable coding instead of direct KLT. While separable coding gives performance comparable to that of WUTC, it uses only 1/8 of the floating-point multiplications and 1/32 of storage of direct KLT. Experimental results included in this work compare the performance of new separable WUTC with both the WUTC and other fast variations of that algorithm.

We describe a general coding strategy leading to a family of universal image compression systems designed to give good performance in applications where the statistics of the source to be compressed are not available at design time or vary over time or space. The basic approach considered uses a two-stage structure in which the single source code of traditional image compression systems is replaced with a family of codes designed to cover a large class of possible sources. To illustrate this approach, we consider the optimal design and use of two-stage codes containing collections of vector quantizers (weighted universal vector quantization), bit allocations for JPEG-style coding (weighted universal bit allocation), and transform codes (weighted universal transform coding). Further, we demonstrate the benefits to be gained from the inclusion of perceptual distortion measures and optimal parsing. The strategy yields two-stage codes that significantly outperform their single-stage predecessors. On a sequence of medical images, weighted universal vector quantization outperforms entropy coded vector quantization by over 9 dB. On the same data sequence, weighted universal bit allocation outperforms a JPEG-style code by over 2.5 dB. On a collection of mixed test and image data, weighted universal transform coding outperforms a single, data-optimized transform code (which gives performance almost identical to that of JPEG) by over 6 dB.

The source coding theorem for stationary sources describes the optimal performance theoretically achievable by fixed- and variable-rate block quantizers. The source coding theorem may be generalized by considering the problem of multiresolution or successive refinement source coding, which is the topic of this work. Given a distortion vector (D/sub 1/,...,D/sub L/), this work describes the family of achievable rate vectors (R/sub 1/,...,R/sub L/) for describing a stationary source at L resolutions, where the description at the first resolution is given at rate R/sub 1/ and achieves an expected distortion no greater than D/sub 1/, the description at the second resolution includes both the first description and a refining description of rate R/sub 2/ and achieves expected distortion no greater than D/sub 2/, and so on. The work includes performance bounds for both fixed- and variable-rate source codes on discrete-time stationary ergodic sources and discrete-time stationary nonergodic sources for any integer number of resolutions L/spl ges/1. For L=1, the source coding theorems for stationary sources result. For L>1, the results extend previous theorems for discrete-alphabet memoryless sources.

Consider a multi-resolution source code for describing a stationary source at L resolutions. The description at the first resolution is given at rate R/sub 1/ and achieves an expected distortion no greater than D/sub 1/. The description at the second resolution includes both the first description and a refining description of rate R/sub 2/ and achieves expected distortion no greater than D/sub 2/, and so on. Previously derived multi-resolution source coding bounds describe the family of achievable rate and distortion vectors ((R/sub 1/, R/sub 2/, ..., R/sub L/), (D/sub 1/, D/sub 2/, D/sub L/)). By examining these multi-resolution rate-distortion bounds, we gain insight into the problem of practical multi-resolution source coding. These insights lead to a new multi-resolution source code based on the tree-structured vector quantizer. This paper covers the algorithm, its optimal design, and preliminary experimental results.

We propose three new design algorithms for jointly optimizing source and channel codes. Our optimality criterion is to minimize the average end-to-end distortion. For a given channel SNR and transmission rate, our joint source and channel code designs achieve an optimal allocation of bits between the source and channel coders. Our three techniques include a source-optimized channel code, a channel-optimized source code, and an iterative descent technique combining the design strategies of the other two codes. The joint designs use channel-optimized vector quantization (COVQ) for the source code and rate compatible punctured convolutional (RCPC) coding for the channel code. The optimal bit allocation reduces distortion by up to 6 dB over suboptimal allocations and by up to 4 dB relative to standard COVQ for the source data set considered. We find that all three code designs have roughly the same performance when their bit allocations are optimized. This result follows from the fact that at the optimal bit allocation the channel code removes most of the channel errors, in which case the three design techniques are roughly equivalent. We also compare the robustness of the three techniques to channel mismatch. We conclude the paper by relaxing the fixed transmission rate constraint and jointly optimizing the transmission rate, source code, and channel code.

Effros and Chou (see Proceedings of the IEEE International Conference on Image Processing, Washington, DC, 1995) introduce a two-stage universal transform code called the weighted universal transform code (WUTC). By replacing JPEG's single, non-optimal transform code with a collection of optimal transform codes, the WUTC achieves significant performance gains over JPEG. The computational and storage costs of that performance gain are effectively the computation and storage required to operate and store a collection of transform codes rather than a single transform code. We consider two complexity- and storage-constrained variations of the WUTC. The complexity and storage of the algorithm are controlled by constraining the order of the bases. In the first algorithm, called the fast WUTC (FWUTC), complexity is controlled by controlling the maximum order of each transform. On a sequence of combined text and gray-scale images, the FWUTC achieves performance comparable to the WUTC. In the second algorithm, called the jointly optimized fast WUTC (JWUTC), the complexity is controlled by controlling the average order of the transforms. On the same data set and for the same complexity, the performance of the JWUTC always exceeds the performance of the FWUTC. The JWUTC and FWUTC algorithm are interesting both for their complexity and storage savings in data compression and for the insights that they lend into the choice of appropriate fixed- and variable-order bases for image representation.

We consider the use of second order statistics in two-stage universal source coding. Examples of two-stage universal codes include the weighted universal vector quantization (WUVQ), weighted universal bit allocation (WUBA), and weighted universal transform coding (WUTC) algorithms. The second order statistics are incorporated in two-stage universal source codes in a manner analogous to the method by which second order statistics are incorporated in entropy constrained vector quantization (ECVQ) to yield conditional ECVQ (CECVQ). In this paper, we describe an optimal two-stage conditional entropy constrained universal source code along with its associated optimal design algorithm and a fast (but nonoptimal) variation of the original code. The design technique and coding algorithm here presented result in a new family of conditional entropy constrained universal codes including but not limited to the conditional entropy constrained WUVQ (CWUVQ), the conditional entropy constrained WUBA (CWUBA), and the conditional entropy constrained WUTC (CWUTC). The fast variation of the conditional entropy constrained universal codes allows the designer to trade off performance gains against storage and delay costs. We demonstrate the performance of the proposed codes on a collection of medical brain scans. On the given data set, the CWUVQ achieves up to 7.5 dB performance improvement over variable-rate WUVQ and up to 12 dB performance improvement over ECVQ. On the same data set, the fast variation of the CWUVQ achieves identical performance to that achieved by the original code at all but the lowest rates (less than 0.125 bits per pixel).

We consider the problem of lossy source coding for transmission across an unknown or time-varying noisy channel. The objective is to design an optimal compression system for applications where the unknown channel characteristics are independently estimated at the channel encoder and decoder. Channel estimation reliability is allowed to vary from perfect channel identification to no channel identification. In each case, the goal in system design and operation is to achieve the best possible expected performance with respect to the unknown channel state and the accuracy of the channel estimators. We describe an optimal design technique and an algorithm for achieving optimal expected performance for the entire array of channel estimation accuracies. The resulting system achieves up to 9 dB improvement over the performance on a system designed assuming zero probability of error when used to encode a collection of medical brain scans for transmission across a finite state channel containing two equally probable binary symmetric channels with crossover probabilities .05 and .001.

We propose an iterative design algorithm for jointly optimizing source and channel codes. The joint design combines channel-optimized vector quantization (COVQ) for the source code with rate-compatible punctured convolutional (RCPC) coding for the channel code. Our objective is to minimize the average end-to-end distortion. For a given channel SNR and transmission rate, our joint source and channel code design achieves an optimal allocation of bits between the source and channel coders. This optimal allocation can reduce distortion by up to 6 dB over suboptimal allocations for the source data set considered. We also compare the distortion of our joint iterative design with that of two suboptimal design techniques: COVQ optimized for a given channel bit-error-probability, and RCPC channel coding optimized for a given vector quantizer. We conclude by relaxing the fixed transmission rate constraint and jointly optimizing the transmission rate, source code, and channel code.

We consider the problem of optimal, data-dependent zerotree design for use in weighted universal zerotree codes for image compression. A weighted universal zerotree code (WUZC) is a data compression system that replaces the single, data-independent zerotree of Said and Pearlman (see IEEE Transactions on Circuits and Systems for Video Technology, vol.6, no.3, p.243-50, 1996) with an optimal collection of zerotrees for good image coding performance across a wide variety of possible sources. We describe the weighted universal zerotree encoding and design algorithms but focus primarily on the problem of optimal, data-dependent zerotree design. We demonstrate the performance of the proposed algorithm by comparing, at a variety of target rates, the performance of a Said-Pearlman style code using the standard zerotree to the performance of the same code using a zerotree designed with our algorithm. The comparison is made without entropy coding. The proposed zerotree design algorithm achieves, on a collection of combined text and gray-scale images, up to 4 dB performance improvement over a Said-Pearlman zerotree.

A two-stage code is a block code in which each block of data is coded in two stages: the first stage codes the identity of a block code among a collection of codes, and the second stage codes the data using the identified code. The collection of codes may be noiseless codes, fixed-rate quantizers, or variable-rate quantizers. We take a vector quantization approach to two-stage coding, in which the first stage code can be regarded as a vector quantizer that "quantizes" the input data of length n to one of a fixed collection of block codes. We apply the generalized Lloyd algorithm to the first-stage quantizer, using induced measures of rate and distortion, to design locally optimal two-stage codes. On a source of medical images, two-stage variable-rate vector quantizers designed in this way outperform standard (one-stage) fixed-rate vector quantizers by over 9 dB. The tail of the operational distortion-rate function of the first-stage quantizer determines the optimal rate of convergence of the redundancy of a universal sequence of two-stage codes. We show that there exist two-stage universal noiseless codes, fixed-rate quantizers, and variable-rate quantizers whose per-letter rate and distortion redundancies converge to zero as (k/2)n/sup -1/ log n, when the universe of sources has finite dimension k. This extends the achievability part of Rissanen's theorem from universal noiseless codes to universal quantizers. Further, we show that the redundancies converge as O(n/sup -1/) when the universe of sources is countable, and as O(n/sup -1+/spl epsiv//) when the universe of sources is infinite-dimensional, under appropriate conditions.

We introduce a two-stage bit allocation algorithm analogous to the algorithm for weighted universal vector quantization (WUVQ). The encoder uses a collection of possible bit allocations (typically in the form of a collection of quantization matrices) rather than a single bit allocation (or single quantization matrix). We describe both an encoding algorithm for achieving optimal compression using a collection of bit allocations and a technique for designing locally optimal collections of bit allocations. We demonstrate performance on a JPEG style coder using the mean squared error (MSE) distortion measure. On a sequence of medical brain scans, the algorithm achieves up to 2.5 dB improvement over a single bit allocation system, up to 5 dB improvement over a WUVQ with first- and second-stage vector dimensions equal to 16 and 4 respectively, and up to 12 dB improvement over an entropy constrained vector quantizer (ECVQ) using 4 dimensional vectors.

We introduce a two-stage universal transform code for image compression. The code combines Karhunen-Loeve transform coding with weighted universal bit allocation (WUBA) in a two-stage algorithm analogous to the algorithm for weighted universal vector quantization (WUVQ). The encoder uses a collection of transform/bit allocation pairs rather than a single transform/bit allocation pair (as in JPEG) or a single transform with a variety of bit allocations (as in WUBA). We describe both an encoding algorithm for achieving optimal compression using a collection of transform/bit allocation pairs and a technique for designing locally optimal collections of transform/bit allocation pairs. We demonstrate the performance using the mean squared error distortion measure. On a sequence of combined text and gray scale images, the algorithm achieves up to a 2 dB improvement over a JPEG style coder using the discrete cosine transform (DCT) and an optimal collection of bit allocations, up to a 3 dB improvement over a JPEG style coder using the DCT and a single (optimal) bit allocation, up to 6 dB over an entropy constrained WUVQ with first- and second-stage vector dimensions equal to 16 and 4 respectively, and up to a 10 dB improvement over an entropy constrained vector quantizer (ECVQ) with a vector dimension of 4.

A new algorithm for variable dimension weighted universal coding is introduced. Combining the multi-codebook system of weighted universal vector quantization (WUVQ), the partitioning technique of variable dimension vector quantization, and the optimal design strategy common to both, variable dimension WUVQ allows mixture sources to be effectively carved into their component subsources, each of which can then be encoded with the codebook best matched to that source. Application of variable dimension WUVQ to a sequence of medical images provides up to 4.8 dB improvement in signal to quantization noise ratio over WUVQ and up to 11 dB improvement over a standard full-search vector quantizer followed by an entropy code. The optimal partitioning technique can likewise be applied with a collection of noiseless codes, as found in weighted universal noiseless coding (WUNC). The resulting algorithm for variable dimension WUNC is also described.>

The authors introduce a one-pass adaptive universal quantization technique for real, bounded alphabet, stationary sources. The algorithm is set on line without any prior knowledge of the statistics of the sources which it might encounter and asymptotically achieves ideal performance on all sources that it sees. The system consists of an encoder and a decoder. At increasing intervals, the encoder refines its codebook using knowledge about incoming data symbols. This codebook is then described to the decoder in the form of updates on the previous codebook. The accuracy to which the codebook is described increases as the number of symbols seen, and thus the accuracy to which the codebook is known, grows.>

For a stationary ergodic source, the source coding theorem and its converse imply that the optimal performance theoretically achievable by a fixed-rate or variable-rate block quantizer is equal to the distortion-rate function, which is defined as the infimum of an expected distortion subject to a mutual information constraint. For a stationary nonergodic source, however, the. Distortion-rate function cannot in general be achieved arbitrarily closely by a fixed-rate block code. We show, though, that for any stationary nonergodic source with a Polish alphabet, the distortion-rate function can be achieved arbitrarily closely by a variable-rate block code. We also show that the distortion-rate function of a stationary nonergodic source has a decomposition as the average of the distortion-rate functions of the source's stationary ergodic components, where the average is taken over points on the component distortion-rate functions having the same slope. These results extend previously known results for finite alphabets.>

The authors combine pruned tree-structured vector quantization (pruned TSVQ) with Itoh's (1987) universal noiseless coder. By combining pruned TSVQ with universal noiseless coding, they benefit from the "successive approximation" capabilities of TSVQ, thereby allowing progressive transmission of images, while retaining the ability to noiselessly encode images of unknown statistics in a provably asymptotically optimal fashion. Noiseless compression results are comparable to Ziv-Lempel and arithmetic coding for both images and finely quantized Gaussian sources.>

Weighted universal vector quantization uses traditional codeword design techniques to design locally optimal multi-codebook systems. Application of this technique to a sequence of medical images produces a 10.3 dB improvement over standard full search vector quantization followed by entropy coding at the cost of increased complexity. In this proposed variation each codebook in the system is given a mean or 'prediction' value which is subtracted from all supervectors that map to the given codebook. The chosen codebook's codewords are then used to encode the resulting residuals. Application of the mean-removed system to the medical data set achieves up to 0.5 dB improvement at no rate expense.>