This work considers new entropy-based proofs of some known, or otherwise refined, combinatorial bounds for bipartite graphs. These include upper bounds on the number of the independent sets, lower bounds on the minimal number of colors in constrained edge coloring, and lower bounds on the number of walks of a given length in bipartite graphs. The proofs of these combinatorial results rely on basic properties of the Shannon entropy.

We explore known integral representations of the logarithmic and power functions, and demonstrate their usefulness for information-theoretic analyses. We obtain compact, easily-computable exact formulas for several source and channel coding problems that involve expectations and higher moments of the logarithm of a positive random variable and the moment of order ρ>0 of a non-negative random variable (or the sum of i.i.d. positive random variables). These integral representations are used in a variety of applications, including the calculation of the degradation in mutual information between the channel input and output as a result of jamming, universal lossless data compression, Shannon and Rényi entropy evaluations, and the ergodic capacity evaluation of the single-input, multiple-output (SIMO) Gaussian channel with random parameters (known to both transmitter and receiver). The integral representation of the logarithmic function and its variants are anticipated to serve as a rigorous alternative to the popular (but non-rigorous) replica method (at least in some situations).

This paper provides upper and lower bounds on the optimal guessing moments of a random variable taking values on a finite set when side information may be available. These moments quantify the number of guesses required for correctly identifying the unknown object and, similarly to Arikan's bounds, they are expressed in terms of the Arimoto- Rényi conditional entropy. Although Arikan's bounds are asymptotically tight, the improvement of the bounds in this paper is significant in the non-asymptotic regime. Relationships between moments of the optimal guessing function and the MAP error probability are provided, characterizing the exact locus of their attainable values.

Bounds on optimal guessing moments serve to improve non-asymptotic bounds on the cumulant generating function of the codeword lengths for fixed-to-variable optimal lossless source coding without prefix constraints. Non-asymptotic bounds on the reliability function of discrete memoryless sources are presented as well. Lower bounds on the cumulant generating function of the codeword lengths are given, by means of the smooth Rényi entropy, for source codes that allow decoding errors.

This paper gives upper and lower bounds on the minimum error probability of Bayesian M-ary hypothesis testing in terms of the Arimoto-Rényi conditional entropy of an arbitrary order α. The improved tightness of these bounds over their specialized versions with the Shannon conditional entropy (α = 1) is demonstrated. In particular, in the case where M is finite, we show how to generalize Fano's inequality under both the conventional and list-decision settings. As a counterpart to the generalized Fano's inequality, allowing M to be infinite, a lower bound on the Arimoto-Rényi conditional entropy is derived as a function of the minimum error probability. Explicit upper and lower bounds on the minimum error probability are obtained as a function of the Arimoto-Rényi conditional entropy for both positive and negative α. Furthermore, we give upper bounds on the minimum error probability as functions of the Rényi divergence. In the setup of discrete memoryless channels, we analyze the exponentially vanishing decay of the Arimoto-Rényi conditional entropy of the transmitted codeword given the channel output when averaged over a random-coding ensemble.

This paper provides upper and lower bounds on the optimal guessing moments of a random variable taking values on a finite set when side information may be available. These moments quantify the number of guesses required for correctly identifying the unknown object and, similarly to Arikan's bounds, they are expressed in terms of the Arimoto-Rényi conditional entropy. Although Arikan's bounds are asymptotically tight, the improvement of the bounds in this paper is significant in the non-asymptotic regime. Relationships between moments of the optimal guessing function and the MAP error probability are also established, characterizing the exact locus of their attainable values. The bounds on optimal guessing moments serve to improve non-asymptotic bounds on the cumulant generating function of the codeword lengths for fixed-to-variable optimal lossless source coding without prefix constraints. Non-asymptotic bounds on the reliability function of discrete memoryless sources are derived as well. Relying on these techniques, lower bounds on the cumulant generating function of the codeword lengths are derived, by means of the smooth Rényi entropy, for source codes that allow decoding errors.

This paper gives upper and lower bounds on the minimum error probability of Bayesian M-ary hypothesis testing in terms of the Arimoto-Rényi conditional entropy of an arbitrary order α. The improved tightness of these bounds over their specialized versions with the Shannon conditional entropy (α = 1) is demonstrated. In particular, in the case where M is finite, we show how to generalize Fano's inequality under both the conventional and list-decision settings. As a counterpart to the generalized Fano's inequality, allowing M to be infinite, a lower bound on the Arimoto-Rényi conditional entropy is derived as a function of the minimum error probability. Explicit upper and lower bounds on the minimum error probability are obtained as a function of the Arimoto-Rényi conditional entropy.

We study the random-coding error exponent function of variable-length codes in the presence of a noiseless feedback channel, which is allowed to be used merely for a single bit feedback per each transmitted message. In this study, we harness results and analysis techniques from the theory of sequential hypothesis testing, and combine them with modern distance enumeration methods which are used in the literature on error exponents. For this setup, sometimes referred to as stop-feedback, we derive an exact single-letter expression for the random-coding error exponent over the binary symmetric channel. For symmetric discrete memoryless channels, the exact error exponent at zero rate is obtained, and a lower bound is provided for any other positive rate below capacity.

Motivated by a recent result by van Erven and Harremoës, we study a forward projection problem for the Rényi divergence on a particular α-convex set, termed α-linear family. The solution to this problem yields a parametric family of probability measures which turns out to be an extension of the exponential family, and it is termed α-exponential family. An orthogonality relationship between the α-exponential and α-linear families is first established and is then used to transform the reverse projection on an α-exponential family into a forward projection on an α-linear family. The full paper version of this work is available on the arXiv at http://arxiv.org/abs/1512.02515.

This paper gives improved Rényi entropy power inequalities (R-EPIs). Consider a sum Sn= Σk=1nXkof n independent continuous random vectors taking values on ℝd, and let α ∈ [1, ∞]. An R-EPI provides a lower bound on the order-α Rényi entropy power of Snthat, up to a multiplicative constant (which may depend in general on n, α, d), is equal to the sum of the order-α Rényi entropy powers of the n random vectors {Xk}k=1n. For α = 1, the R-EPI coincides with the wellknown entropy power inequality by Shannon. The first improved R-EPI is obtained by tightening the recent R-EPI by Bobkov and Chistyakov, which relies on the sharpened Young's inequality. A further improvement of the R-EPI also relies on convex optimization and results on rank-one modification of a real-valued diagonal matrix.

This paper studies forward and reverse projections for the Rényi divergence of order α E (0, ∞) on α-convex sets. The forward projection on such a set is motivated by some works of Tsallis et al. in statistical physics, and the reverse projection is motivated by robust statistics. In a recent work, van Erven and Harremoës proved a Pythagorean inequality for Rényi divergences on α-convex sets under the assumption that the forward projection exists. Continuing this study, a sufficient condition for the existence of a forward projection is proved for probability measures on a general alphabet. For αϵ(1, ∞), the proof relies on a new Apollonius theorem for the Hellinger divergence, and for α E (0, 1), the proof relies on the Banach- Alaoglu theorem from the functional analysis. Further projection results are then obtained in the finite alphabet setting. These include a projection theorem on a specific α-convex set, which is termed an α-linear family, generalizing a result by Csiszar to α ≠ 1. The solution to this problem yields a parametric family of probability measures, which turns out to be an extension of the exponential family, and it is termed an α-exponential family. An orthogonality relationship between the α-exponential and α-linear families is established, and it is used to turn the reverse projection on an α-exponential family into a forward projection on an α-linear family. This paper also proves a convergence result of an iterative procedure used to calculate the forward projection on an intersection of a finite number of α-linear families.

This paper develops systematic approaches to obtain f -divergence inequalities, dealing with pairs of probability measures defined on arbitrary alphabets. Functional domination is one such approach, where special emphasis is placed on finding the best possible constant upper bounding a ratio of f -divergences. Another approach used for the derivation of bounds among f -divergences relies on moment inequalities and the logarithmic-convexity property, which results in tight bounds on the relative entropy and Bhattacharyya distance in terms of χ2divergences. A rich variety of bounds are shown to hold under boundedness assumptions on the relative information. Special attention is devoted to the total variation distance and its relation to the relative information and relative entropy, including “reverse Pinsker inequalities,” as well as on the Eγdivergence, which generalizes the total variation distance. Pinsker's inequality is extended for this type of f -divergence, a result which leads to an inequality linking the relative entropy and relative information spectrum. Integral expressions of the Rényi divergence in terms of the relative information spectrum are derived, leading to bounds on the Rényi divergence in terms of either the variational distance or relative entropy.

This paper starts with a study of the minimum of the Rényi divergence, of an arbitrary order α > 0, subject to a fixed (or minimal) value of the total variation distance. Relying on the solution of this minimization problem, we determine the exact region of the points (D(Q∥P1), D(Q∥P2)) where P1and P2are any probability distributions whose total variation distance is not below a fixed value, and the probability distribution Q is arbitrary (none of these three distributions is assumed to be fixed). It is further shown that all the points of this convex region are attained by a triple of 2-element probability distributions. As a byproduct of this characterization, we provide a geometric interpretation of the minimal Chernoff information subject to a minimal total variation distance. A full paper version, which includes more results and proofs, is available at http://arxiv.org/abs/1501.03616.

Tight bounds for several symmetric divergence measures are introduced, given in terms of the total variation distance. Each of these bounds is attained by a pair of 2 or 3-element probability distributions. An application of these bounds for lossless source coding is provided, refining and improving a certain bound by Csiszár. A new inequality relating f-divergences is derived, and its use is exemplified. The last section of this conference paper is not included in the recent journal paper [16], as well as some new remarks that are linked to new references.

This paper presents polar coding schemes for the two-user discrete memoryless broadcast channel (DM-BC) which achieve Marton's region with both common and private messages. This is the best achievable rate region known to date, and it is tight for all classes of two-user DM-BCs whose capacity regions are known. To accomplish this task, we first construct polar codes for both the superposition as well as binning strategy. By combining these two schemes, we obtain Marton's region with private messages only. Finally, we show how to handle the case of common information. The proposed coding schemes possess the usual advantages of polar codes, i.e., they have low encoding and decoding complexity and a superpolynomial decay rate of the error probability. We follow the lead of Goela, Abbe, and Gastpar, who recently introduced polar codes emulating the superposition and binning schemes. To align the polar indices, for both schemes, their solution involves some degradedness constraints that are assumed to hold between the auxiliary random variables and channel outputs. To remove these constraints, we consider the transmission of k blocks and employ a chaining construction that guarantees the proper alignment of the polarized indices. The techniques described in this paper are quite general, and they can be adopted to many other multiterminal scenarios whenever there polar indices need to be aligned.

Tight bounds for several symmetric divergence measures are derived in terms of the total variation distance. It is shown that each of these bounds is attained by a pair of two- or three-element probability distributions. An application of these bounds for lossless source coding is provided, refining and improving a certain bound by Csiszár. Another application of these bounds has been recently introduced by Yardi et al. for channel-code detection.

We present polar coding schemes for the 2-user discrete memoryless broadcast channel (DM-BC) which achieve Marton's region with both common and private messages. This is the best achievable rate region up to date, and it is tight for all classes of 2-user DM-BCs whose capacity regions are known. Due to space limitations, this paper describes polar codes for the superposition strategy. The scheme for the achievability of Marton's region is presented in the longer version [1], and it is based on a combination of superposition coding and binning. We follow the lead of the recent work by Goela, Abbe, and Gastpar, who introduce polar codes emulating these two information-theoretic techniques. In order to align the polar indices, for both schemes, their solution involves some degradedness constraints that are assumed to hold between the auxiliary random variables and the channel outputs. To remove these constraints, we consider the transmission of k blocks, and employ chaining constructions that guarantee the proper alignment of polarized indices. The techniques described in this work are quite general, and they can be adopted in many other multi-terminal scenarios whenever there is the need for the aligning of polar indices.

We derive sharpened inequalities on the empirical output distribution of good channel codes with deterministic encoders and with non-vanishing maximal probability of decoding error. These inequalities refine recent bounds of Polyanskiy and Verdú by identifying closed-form expressions for certain asymptotic terms, which facilitates their calculation for finite blocklengths. The analysis relies on concentration-of-measure inequalities, specifically on McDiarmid's method of bounded differences and its close ties to transportation inequalities for weighted Hamming metrics. An operational implication of the new bounds is addressed.

This work provides new bounds on the difference between the entropies of two discrete random variables in terms of the local and total variation distances between their probability mass functions. The derivation of the bounds relies on maximal couplings, and the bounds apply to discrete random variables which are defined over finite or countably infinite alphabets. Loosened versions of these bounds are demonstrated to reproduce some previously reported results. The use of the new entropy bounds is exemplified for the Poisson approximation, where bounds on the local and total variation distances follow from Stein's method. The full paper version for this work is available at http://arxiv.org/abs/1209.5259.

Channel coding over arbitrarily permuted parallel channels was first studied by Willems and coworkers. This paper introduces capacity-achieving polar coding schemes for arbitrarily permuted parallel channels where the component channels are memoryless, binary-input, and output-symmetric.

This paper derives new bounds on the difference of the entropies of two discrete random variables in terms of the local and total variation distances between their probability mass functions. The derivation of the bounds relies on maximal coupling, and they apply to discrete random variables which are defined over finite or countably infinite alphabets. Loosened versions of these bounds are demonstrated to reproduce some previously reported results. The use of the new bounds is exemplified for the Poisson approximation, where bounds on the local and total variation distances follow from Stein's method.

This work refers to moderate-deviations analysis of binary hypothesis testing. It relies on a concentration inequality for discrete-parameter martingales with bounded jumps, which forms a refinement to the Azuma-Hoeffding inequality. Relations of the analysis to the moderate deviations principle for i.i.d. random variables and the relative entropy are considered.

This paper establishes new achievable rates for nonlinear Volterra communication channels using refined versions of the Azuma-Hoeffding inequality. The characteristics of these rates are illuminated in special cases of interest that include time invariant linear channels with memory, memoryless non-linear channels, and Volterra channel models.

This paper considers the entropy of the sum of (possibly dependent and non-identically distributed) Bernoulli random variables. Upper bounds on the error that follows from an approximation of this entropy by the entropy of a Poisson random variable with the same mean are derived. The derivation of these bounds combines elements of information theory with the Chen-Stein method for Poisson approximation. The resulting bounds are easy to compute, and their applicability is exemplified. This conference paper presents in part the first half of the paper entitled “An information-theoretic perspective of the Poisson approximation via the Chen-Stein method” (see: http://arxiv.org/abs/1206.6811). A generalization of the bounds that considers the accuracy of the Poisson approximation for the entropy of a sum of non-negative, integer-valued and bounded random variables is introduced in the full paper. It also derives lower bounds on the total variation distance, relative entropy and other measures that are not considered in this conference paper.

This work considers the concentration of measures for low-density parity-check (LDPC) code ensembles. The two results derived in this paper follow from Azuma's inequality for Doob martingales with bounded differences. The first result is a tightened concentration inequality for the conditional entropy (originally derived by Méasson et al.), and the second result is a concentration inequality for the cardinality of the fundamental systems of cycles of a bipartite graph from the ensemble.

A design of robust error-correcting codes that achieve reliable communication over various channels is of great theoretical and practical interest. Such codes are termed universal. This paper considers the universality of low-density parity-check (LDPC) code ensembles over families of memoryless binary-input output-symmetric (MBIOS) channels. Universality is considered both under belief-propagation (BP) and maximum-likelihood (ML) decoding. For the BP decoding case, we derive a density-evolution-based analytical method for designing LDPC code ensembles that are universal over various families of MBIOS channels. We also derive a necessary condition for universality of LDPC code ensembles under BP decoding; this condition is used to provide bounds on the universally achievable fraction of capacity. These results enable us to provide conditions for reliable/unreliable communications under BP decoding that are based on the Bhattacharyya parameter of the channel. For the ML decoding case, we prove that properly selected regular LDPC code ensembles are universally capacity-achieving for the set of equi-capacity MBIOS channels and extend this result to punctured regular LDPC code ensembles.

A universal design of low-density parity-check (LDPC) code ensembles which enables to operate reliably over various channels is of great theoretical and practical interest. This paper considers the universality of LDPC code ensembles over multitude memoryless binary-input output-symmetric (MBIOS) channels, addressing their universality under belief-propagation (BP) decoding. Based on the density evolution approach, analytical results related to the universality of LDPC code ensembles under BP decoding are derived; these results are expressed in closed form, and are easy to calculate. The full paper version related to this work provides further results, full proofs, additional discussions on the theorems, and it also considers the universality issue under ML decoding.

A capacity-achieving polar coding scheme is introduced for reliable communications over a set of parallel communication channels. They are assumed to be arbitrarily-permuted memoryless binary-input and output-symmetric (MBIOS) channels, and they form a set of (stochastically) degraded channels. The general case where the parallel channels are not necessarily degraded is addressed in the full paper version [3], though the suggested scheme is not capacity-achieving in the general case.

A message independence property and some new performance upper bounds are derived in this work for erasure, list, and decision-feedback schemes with linear block codes transmitted over memoryless symmetric channels. Similar to the classical work of Forney, this work is focused on the derivation of some Gallager-type bounds on the achievable tradeoffs for these coding schemes, where the main novelty is the suitability of the bounds for both random and structured linear block codes (or code ensembles). The bounds are applicable to finite-length codes and to the asymptotic case of infinite block length, and they are applied to low-density parity-check code ensembles.

This work introduces an information-theoretic lower bound on the number of fundamental cycles for bipartite graphs of low-density parity-check (LDPC) code ensembles. This information-theoretic bound is expressed in terms of the achievable gap to capacity when the transmission of the code ensemble takes place over a memoryless binary-input output-symmetric (MBIOS) channel. The bound shows quantitatively the necessity of cycles in bipartite graphs which represent good LDPC code ensembles. More explicitly, it shows that the number of fundamental cycles should grow at least like log 1/epsiv where epsiv designates the gap in rate to capacity, hence, it is unbounded as the gap to capacity vanishes. For the derivation of this bound, a new information-theoretic lower bound on the average right degree, which also behaves like log 1/epsiv, is derived. The interested reader is referred to the full paper version.

This paper considers information-theoretic lower bounds on the graphical complexity of finite-length LDPC codes. It is assumed that the transmission of the codes takes place over a memoryless binary-input output-symmetric (MBIOS) channel, and the bounds are expressed as a function of the code performance and their achievable gap to capacity (either under ML decoding or any sub-optimal decoding algorithm). The lower bounds on the graphical complexity are compared to some explicit LDPC codes (or code ensembles), showing that these bounds are informative for considering the fundamental tradeoff which exists between the performance and graphical complexity of finite-length LDPC codes. This work relies on the full paper version.

This work considers the behavior of the degree distributions of capacity-approaching low-density parity-check (LDPC) code ensembles via linear programming (LP) bounds. These LP bounds are information-theoretic, and they apply to finite-length LDPC codes and to the asymptotic case of an infinite block length. Analytical solutions of these bounds are given in closed form, and the bounds are compared for the BEC with some specific degree distributions of capacity-achieving sequences of LDPC code ensembles. These LP bounds are shown to be informative and are easy to calculate. Due to space limitations, the derivation of the LP bounds is outlined, and the reader is referred to the full paper version for complete proofs and further discussions.

The performance of nonbinary linear block codes is studied in this paper via the derivation of new upper bounds on the block error probability under maximum-likelihood (ML) decoding. The transmission of these codes is assumed to take place over a memoryless and symmetric channel. The new bounds, which are based on the Gallager bounds and their variations, are applied to the Gallager ensembles of nonbinary and regular low-density parity-check (LDPC) codes. These upper bounds are also compared with sphere-packing lower bounds. This study indicates that the new upper bounds are useful for the performance evaluation of coded communication systems which incorporate nonbinary coding techniques.

This paper is focused on the derivation of some universal properties of capacity-approaching low-density parity-check (LDPC) code ensembles whose transmission takes place over memoryless binary-input output-symmetric (MBIOS) channels. Properties of the degree distributions, graphical complexity, and the number of fundamental cycles in the bipartite graphs are considered via the derivation of information-theoretic bounds. These bounds are expressed in terms of the target block/bit error probability and the gap (in rate) to capacity. Most of the bounds are general for any decoding algorithm, and some others are proved under belief propagation (BP) decoding. Proving these bounds under a certain decoding algorithm, validates them automatically also under any suboptimal decoding algorithm. A proper modification of these bounds makes them universal for the set of all MBIOS channels which exhibit a given capacity. Bounds on the degree distributions and graphical complexity apply to finite-length LDPC codes and to the asymptotic case of an infinite block length. The bounds are compared with capacity-approaching LDPC code ensembles under BP decoding, and they are shown to be informative and are easy to calculate. Finally, some interesting open problems are considered.

This paper provides simple lower bounds on the number of iterations which is required for successful message-passing decoding of some important families of graph-based code ensembles (including low-density parity-check codes and variations of repeat-accumulate codes). The transmission of the code ensembles is assumed to take place over a binary erasure channel, and the bounds refer to the asymptotic case where we let the block length tend to infinity. The simplicity of the bounds derived in this paper stems from the fact that they are easily evaluated and are expressed in terms of some basic parameters of the ensemble which include the fraction of degree-2 variable nodes, the target bit erasure probability and the gap between the channel capacity and the design rate of the ensemble. This paper demonstrates that the number of iterations which is required for successful message-passing decoding scales at least like the inverse of the gap (in rate) to capacity, provided that the fraction of degree-2 variable nodes of these turbo-like ensembles does not vanish (hence, the number of iterations becomes unbounded as the gap to capacity vanishes).

We derive simple lower bounds on the number of iterations which are required to communicate over the binary erasure channel when graph-based code ensembles are used in conjunction with an iterative message-passing decoder. These bounds refer to the asymptotic case where we let the block length tend to infinity, and the bounds apply to general ensembles of low- density parity-check (LDPC) codes, irregular repeat-accumulate (IRA) and accumulate-repeat-accumulate (ARA) codes. It is demonstrated that, under a mild condition, the number of iterations required for successful decoding scales at least like the inverse of the gap (in rate) to capacity.

The performance analysis of non-binary linear block codes is studied under ML decoding where it is assumed that the transmission takes place over memoryless symmetric channels. Gallager-type bounds are derived, and the proposed bounds are exemplified for expurgated regular ensembles of non-binary low-density parity-check (LDPC) codes. These bounds are also compared with classical and recent improved sphere-packing bounds, indicating that these bounding techniques are informative for the performance evaluation of coded communication systems which incorporate non-binary coding techniques.

In the above titled paper (ibid., vol. 53, no. 2, pp. 580-598, Feb 07), there was a printing typo in the last line of eq. (30). Necessary revisions are presented here.

This paper derives an improved sphere-packing (ISP) bound for finite-length error-correcting codes whose transmission takes place over symmetric memoryless channels, and the codes are decoded with an arbitrary list decoder. We first review classical results, i.e., the 1959 sphere-packing (SP59) bound of Shannon for the Gaussian channel, and the 1967 sphere-packing (SP67) bound of Shannon et al. for discrete memoryless channels. An improvement on the SP67 bound, as suggested by Valembois and Fossorier, is also discussed. These concepts are used for the derivation of a new lower bound on the error probability of list decoding (referred to as the ISP bound) which is uniformly tighter than the SP67 bound and its improved version. The ISP bound is applicable to symmetric memoryless channels, and some of its applications are presented. Its tightness under maximum-likelihood (ML) decoding is studied by comparing the ISP bound to previously reported upper and lower bounds on the ML decoding error probability, and also to computer simulations of iteratively decoded turbo-like codes. This paper also presents a technique which performs the entire calculation of the SP59 bound in the logarithmic domain, thus facilitating the exact calculation of this bound for moderate to large block lengths without the need for the asymptotic approximations provided by Shannon.

This paper introduces ensembles of systematic accumulaterepeataccumulate (ARA) codes which asymptotically achieve capacity on the binary erasure channel (BEC) with bounded complexity, per information bit, of encoding and decoding. It also introduces symmetry properties which play a central role in the construction of new capacity-achieving ensembles for the BEC. The results here improve on the tradeoff between performance and complexity provided by previous constructions of capacity-achieving code ensembles defined on graphs. The superiority of ARA codes with moderate to large block length is exemplified by computer simulations which compare their performance with those of previously reported capacity-achieving ensembles of low-density parity-check (LDPC) and irregular repeataccumulate (IRA) codes. ARA codes also have the advantage of being systematic.

The transmission of coded communication systems is widely modeled to take place over a set of parallel channels. This model is used for transmission over block-fading channels, rate-compatible puncturing of turbo-like codes, multicarrier signaling, multilevel coding, etc. New upper bounds on the maximum-likelihood (ML) decoding error probability are derived in the parallel-channel setting. We focus on the generalization of the Gallager-type bounds and discuss the connections between some versions of these bounds. The tightness of these bounds for parallel channels is exemplified for structured ensembles of turbo codes, repeat-accumulate (RA) codes, and some of their recent variations (e.g., punctured accumulate-repeat-accumulate codes). The bounds on the decoding error probability of an ML decoder are compared to computer simulations of iterative decoding. The new bounds show a remarkable improvement over the union bound and some other previously reported bounds for independent parallel channels. This improvement is exemplified for relatively short block lengths, and it is pronounced when the block length is increased. In the asymptotic case, where we let the block length tend to infinity, inner bounds on the attainable channel regions of modern coding techniques under ML decoding are obtained, based solely on the asymptotic growth rates of the average distance spectra of these code ensembles.

A variety of communication scenarios can be modeled by a set of parallel channels. Upper bounds on the achievable rates under maximum-likelihood (ML) decoding, and lower bounds on the decoding complexity per iteration of ensembles of low-density parity-check (LDPC) codes are presented. The communication of these codes is assumed to take place over statistically independent parallel channels where the component channels are memoryless, binary-input, and output-symmetric. The bounds are applied to ensembles of punctured LDPC codes where the puncturing patterns are either random or possess some structure. Our discussion is concluded by a diagram showing interconnections between the new theorems and some previously reported results.

The performance of maximum-likelihood (ML) decoded binary linear block codes over the additive white Gaussian noise (AWGN) channel is addressed via the tangential sphere bound (TSB) and two of its recent improved versions. The correspondence is focused on the derivation of the error exponents of these bounds. Although it was shown that some recent improvements of the TSB tighten this bound for finite-length codes, it is demonstrated in this correspondence that their error exponents coincide. For an arbitrary ensemble of binary linear block codes, the common value of these error exponents is explicitly expressed in terms of the asymptotic growth rate of the average distance spectrum

The performance of maximum-likelihood (ML) decoded binary linear block codes is addressed via the derivation of tightened upper bounds on their decoding error probability. The upper bounds on the block and bit error probabilities are valid for any memoryless, binary-input and output-symmetric communication channel, and their effectiveness is exemplified for various ensembles of turbo-like codes over the additive white Gaussian noise (AWGN) channel. An expurgation of the distance spectrum of binary linear block codes further tightens the resulting upper bounds.

The moderate complexity of low-density parity-check (LDPC) codes under iterative decoding is attributed to the sparseness of their parity-check matrices. It is therefore of interest to consider how sparse parity-check matrices of binary linear block codes can be a function of the gap between their achievable rates and the channel capacity. This issue was addressed by Sason and Urbanke, and it is revisited in this paper. The remarkable performance of LDPC codes under practical and suboptimal decoding algorithms motivates the assessment of the inherent loss in performance which is attributed to the structure of the code or ensemble under maximum-likelihood (ML) decoding, and the additional loss which is imposed by the suboptimality of the decoder. These issues are addressed by obtaining upper bounds on the achievable rates of binary linear block codes, and lower bounds on the asymptotic density of their parity-check matrices as a function of the gap between their achievable rates and the channel capacity; these bounds are valid under ML decoding, and hence, they are valid for any suboptimal decoding algorithm. The new bounds improve on previously reported results by Burshteinand by Sason and Urbanke, and they hold for the case where the transmission takes place over an arbitrary memoryless binary-input output-symmetric (MBIOS) channel. The significance of these information-theoretic bounds is in assessing the tradeoff between the asymptotic performance of LDPC codes and their decoding complexity (per iteration) under message-passing decoding. They are also helpful in studying the potential achievable rates of ensembles of LDPC codes under optimal decoding; by comparing these thresholds with those calculated by the density evolution technique, one obtains a measure for the asymptotic suboptimality of iterative decoding algorithms.

The paper presents bounds on the achievable rates and the decoding complexity per iteration of low-density parity-check (LDPC) codes. It is assumed that the communication of these codes takes place over statistically independent parallel channels where these channels are memoryless, binary-input and output-symmetric. The bounds are applied to punctured LDPC codes. A diagram concludes our discussion by showing interconnections between the theorems in this paper and some previously reported results

The performance of maximum-likelihood (ML) decoded binary linear block codes is addressed via the derivation of tightened upper bounds on their decoding error probability. The upper bounds on the block and bit error probabilities are valid for any memoryless, binary-input and output-symmetric communication channel. The effectiveness of these bounds is exemplified for ensembles of turbo-like codes

We present two sequences of ensembles of nonsystematic irregular repeat-accumulate (IRA) codes which asymptotically (as their block length tends to infinity) achieve capacity on the binary erasure channel (BEC) with bounded complexity per information bit. This is in contrast to all previous constructions of capacity-achieving sequences of ensembles whose complexity grows at least like the log of the inverse of the gap (in rate) to capacity. The new bounded complexity result is achieved by puncturing bits, and allowing in this way a sufficient number of state nodes in the Tanner graph representing the codes. We derive an information-theoretic lower bound on the decoding complexity of randomly punctured codes on graphs. The bound holds for every memoryless binary-input output-symmetric (MBIOS) channel and is refined for the binary erasure channel.

We present two sequences of ensembles of nonsystematic irregular repeat-accumulate codes which asymptotically (as their block length tends to infinity) achieve capacity on the binary erasure channel (BEC) with bounded complexity. This is in contrast to all previous constructions of capacity-achieving sequences of ensembles whose complexity grows at least like the log of the inverse of the gap to capacity. The new bounded complexity result is achieved by allowing a sufficient number of state nodes in the Tanner graph representing the codes.

This paper analyzes achievable rate regions for the Gaussian IFC (interference channel). The achievable rate region is based on reducing one user's transmit power to a point such that its message is decodeable to both receivers. The second receiver then does interference reduction. The main point of the paper is that the time-sharing of Sato's region exceeds the convex closure of a sub-region of Han and Kobayashi for symmetric interference channel under moderate interference, and also the achievable rate region by TDM/ FDM. It follows that on degraded Gaussian IFC, the sum-capacity is achieved with the stronger user transmitting at its maximum rate (ignoring the interference), while the weaker user treats the stronger user as interference. For a one-sided Gaussian IFC with weak or moderate interference, the sum-capacity is achieved if the transmitter, which is not interfered sends its data at the maximal achievable rate of a single-user, and the second transmitter sends its data at the maximal possible rate where the interfering signal is treated as an additive Gaussian noise. We show that the sum-capacity of a one-sided Gaussian IFC implies the upper bound on the sum-capacity of a two-user Gaussian IFC.

The complete characterization of the capacity region of a two-user Gaussian interference channel (IC) is still an open problem unless the interference is strong. In this work, we derive an achievable rate region for this channel. It includes the rate region which is achieved by time/frequency division multiplexing (TDM/ FDM), and it also includes the rate region which is obtained by time sharing between the two rate pairs where one of the transmitters sends its data reliably at the maximal possible rate (i.e., the maximum rate it can achieve in the absence of interference), and the other transmitter decreases its data rate to the point where both receivers can reliably decode its message. The suggested rate region is easily calculable, though it is a particular case of the celebrated achievable rate region of Han and Kobayashi whose calculation is, in general, prohibitively complex. In the high-power regime, a lower bound on the sum-capacity (i.e., the maximal achievable total rate) is derived, and we show its superiority over the maximal total rate which is achieved by the TDM/FDM approach with moderate interference. For degraded and one-sided Gaussian ICs, we rely on some observations of Costa and Sato, and obtain directly their sum-capacities. We conclude our discussion by pointing out two interesting open problems.

We derive upper and lower bounds on the encoding and decoding complexity of two capacity-achieving ensembles of irregular repeat-accumulate (IRA1 and IRA2) codes on the binary erasure channel (BEC). These bounds are expressed in terms of the gap between the channel capacity and the rate of a typical code from this ensemble for which reliable communications is achievable under message-passing iterative (MPI) decoding. The complexity of the ensemble of IRA1 codes grows like the negative logarithm of the gap to capacity. On the other hand, the complexity of the ensemble of IRA2 codes with any choice of the degree distribution grows at least like the inverse square root of the gap to capacity, and at most like the inverse of the gap to capacity.

We derive tight exponential upper bounds on the decoding error probability of block codes which are operating over fully interleaved Rician fading channels, coherently detected and maximum-likelihood decoded. It is assumed that the fading samples are statistically independent and that perfect estimates of these samples are provided to the decoder. These upper bounds on the bit and block error probabilities are based on certain variations of the Gallager bounds. These bounds do not require integration in their final version and they are reasonably tight in a certain portion of the rate region exceeding the cutoff rate of the channel. By inserting interconnections between these bounds, we show that they are generalized versions of some reported bounds for the binary-input additive white Gaussian noise channel.

We derive lower bounds on the density of parity-check matrices of binary linear codes which are used over memoryless binary-input output-symmetric (MBIOS) channels. The bounds are expressed in terms of the gap between the rate of these codes for which reliable communications is achievable and the channel capacity; they are valid for every sequence of binary linear block codes if there exists a decoding algorithm under which the average bit-error probability vanishes. For every MBIOS channel, we construct a sequence of ensembles of regular low-density parity-check (LDPC) codes, so that an upper bound on the asymptotic density of their parity-check matrices scales similarly to the lower bound. The tightness of the lower bound is demonstrated for the binary erasure channel by analyzing a sequence of ensembles of right-regular LDPC codes which was introduced by Shokrollahi, and which is known to achieve the capacity of this channel. Under iterative message-passing decoding, we show that this sequence of ensembles is asymptotically optimal (in a sense to be defined in this paper), strengthening a result of Shokrollahi. Finally, we derive lower bounds on the bit-error probability and on the gap to capacity for binary linear block codes which are represented by bipartite graphs, and study their performance limitations over MBIOS channels. The latter bounds provide a quantitative measure for the number of cycles of bipartite graphs which represent good error-correction codes.

We present a general method for computing the asymptotic input-output weight distribution of convolutional encoders. In some instances, one can derive explicit analytic expressions. In general, though, to determine the growth rate of the input-output weight distribution for a particular normalized input weight /spl kappa/ and output weight /spl omega/, a system of polynomial equations has to be solved. This method is then used to determine the asymptotic weight distribution of various concatenated code ensembles and to derive lower bounds on the thresholds of these ensembles under maximum-likelihood (ML) decoding.

There has been renewed interest in deriving tight bounds on the error performance of specific codes and ensembles, based on their distance spectrum. We discuss many reported upper bounds on the maximum-likelihood (ML) decoding error probability and demonstrate the underlying connections that exist between them. In addressing the Gallager bounds and their variations, we focus on the Duman and Salehi (see IEEE Trans. Commun., vol.46, p.717-723, 1998)variation, which originates from the standard Gallager bound. A large class of efficient bounds (or their Chernoff versions) is demonstrated to be a special case of the generalized second version of the Duman and Salehi bounds. Implications and applications of these observations are pointed out, including the fully interleaved fading channel, resorting to either matched or mismatched decoding. The proposed approach can be generalized to geometrically uniform nonbinary codes, finite-state channels, bit interleaved coded modulation systems, and it can be also used for the derivation of upper bounds on the conditional decoding error probability.

We derive here improved upper bounds on the decoding error probability of block codes which are transmitted over fully interleaved Rician fading channels, coherently detected and maximum-likelihood (ML) decoded. We assume that the fading coefficients during each symbol are statistically independent (due to a perfect channel interleaver), and that perfect estimates of these fading coefficients are provided to the receiver. The improved upper bounds on the block and bit error probabilities are derived for fully interleaved fading channels with various orders of space diversity, and are found by generalizing some previously introduced upper bounds for the binary-input additive white Gaussian nose (AWGN) channel. The advantage of these bounds over the ubiquitous union bound is demonstrated for some ensembles of turbo codes and low-density parity-check (LDPC) codes, and it is especially pronounced in a portion of the rate region exceeding the cutoff rate. Our generalization of the Duman and Salehi bound (Duman and Salehi 1998, Duman 1998) which is based on certain variations of Gallager's (1965) bounding technique, is demonstrated to be the tightest reported upper bound. We therefore apply it to calculate numerically upper bounds on the thresholds of some ensembles of turbo-like codes, referring to the optimal ML decoding. For certain ensembles of uniformly interleaved turbo codes, the upper bounds derived here also indicate good match with computer simulation results of efficient iterative decoding algorithms.

The ensemble performance of ML decoded turbo codes using coherent BPSK signaling on fully interleaved (memoryless) Rician fading channels is considered, where the ensemble is generated by a uniform choice of the interleaver. The improved bound proposed here is advantageous over the ubiquitous union bound, and it is especially pronounced in the rate region exceeding the cutoff rate (where the performance of turbo codes is most appealing but the union bounds become useless). The upper bounds are compared to simulation results of the log-MAP iterative decoding algorithm for various degrees of space diversity, demonstrating a good match. Hence the improved bounds can be used also as a fast technique to approximately assess the performance of efficient iterative decoding.

The ensemble performance of parallel and serial concatenated turbo codes is considered, where the ensemble is generated by a uniform choice of the interleaver and of the component codes taken from the set of time-varying recursive systematic convolutional codes. Following the derivation of the input-output weight enumeration functions of the ensembles of random parallel and serial concatenated turbo codes, the tangential sphere upper bound is employed to provide improved upper bounds on the block and bit error probabilities of these ensembles of codes for the binary-input additive white Gaussian noise (AWGN) channel, based on coherent detection of equi-energy antipodal signals and maximum-likelihood decoding. The influence of the interleaver length and the memory length of the component codes is investigated. The improved bounding technique proposed here is compared to the conventional union bound and to a alternative bounding technique by Duman and Salehi (1998) which incorporates modified Gallager bounds. The advantage of the derived bounds is demonstrated for a variety of parallel and serial concatenated coding schemes with either fixed or random recursive systematic convolutional component codes, and it is especially pronounced in the region exceeding the cutoff rate, where the performance of turbo codes is most appealing. These upper bounds are also compared to simulation results of the iterative decoding algorithm.

We study a serially interleaved concatenated code construction, where the outer code is a standard convolutional code, and the inner code is a recursive convolutional code of rate 1. We focus on the ubiquitous inner differential encoder (used, in particular, to resolve phase ambiguities), double differential encoder (used to resolve both phase and frequency ambiguities), and another rate 1 recursive convolutional code of memory 2. We substantiate analytically the rather surprising result, that the error probabilities corresponding to a maximum-likelihood (ML) coherently detected antipodal modulation over the additive white Gaussian noise (AWGN) channel for this construction are advantageous as compared to the stand-alone outer convolutional code. This is in spite of the fact that the inner code is of rate 1. The analysis is based on the tangential sphere upper bound of an ML decoder, incorporating the ensemble weight distribution (WD) of the concatenated code, where the ensemble is generated by all random and uniform interleavers. This surprising result is attributed to the WD thinning observed for the concatenated scheme which shapes the WD of the outer convolutional code to resemble more closely the binomial distribution (typical of a fully random code of the same length and rate). This gain is maintained regardless of a rather dramatic decrease, as demonstrated here, in the minimum distance of the concatenated scheme as compared to the minimum distance of the outer stand-alone convolutional code. The advantage of the examined serially interleaved concatenated code, given in terms of bit and/or block error probability which is decoded by a practical suboptimal decoder, over the optimally decoded standard convolutional code is demonstrated by simulations, and some insights into the performance of the iterative decoding algorithm are also discussed. Though we have investigated only specific constructions of constituent inner (rate 1) and outer codes, we trust, hinging on the rational of the arguments here, that these results extend to many other constituent convolutional outer codes and rate 1 inner recursive convolutional codes.