The mutual information is bounded from above by a decreasing affine function of the square of the distance between the input distribution and the set of all capacity-achieving input distributions ΠA, on small enough neighborhoods of ΠA, using an identity due to Topsøe and the Pinsker’s inequality, assuming that the input set of the channel is finite and the constraint set A is polyhedral, i.e., can be described by (possibly multiple but) finitely many linear constraints. Counterexamples demonstrating nonexistence of such a quadratic bound are provided for the case of infinitely many linear constraints and the case of infinite input sets. Using Taylor’s theorem with the remainder term, rather than the Pinsker’s inequality and invoking Moreau’s decomposition theorem the exact characterization of the slowest decrease of the mutual information with the distance to ΠAis determined on small neighborhoods of ΠA. Corresponding results for classical-quantum channels are established under separable output Hilbert space assumption for the quadratic bound and under finite-dimensional output Hilbert space assumption for the exact characterization. Implications of these observations for the channel coding problem and applications of the proof techniques to related problems are discussed.

A new method to characterize Augustin information and mean is proposed. The proposed method allows for briefer and more direct proofs for the previously known results and leads to new observations related to this new characterization for classical channels. For classical-quantum channels, the proposed method extends the results, such as the existence of Augustin mean and Augustin fixed point property, to models with separable Hilbert spaces at the output.

For channels with finite input and output sets, under mild technical assumptions, the local behavior of the Augustin information as a function of the input distribution is characterized for all positive orders using the implicit function theorem and the characterization of the Augustin information in terms of the Augustin dual. For channels with (potentially multiple) linear constraints, the slowest decrease of Augustin information with increasing distance from the Augustin capacity-achieving input distributions is characterized within small neighborhoods around these distributions for all positive orders.

The mutual information is analyzed as a function of the input distribution using an identity due to Topsøe for channels with (possibly multiple) linear constraints and finite input and output sets. The mutual information is bounded above by a function decreasing quadratically with the distance to the set of all capacity-achieving input distributions for the case when the distance is less than a certain threshold. Explicit expressions for the threshold and the coefficient of the quadratic decrease are derived. A counter-example is provided demonstrating the non-existence of such a quadratic bound in the case of infinitely many linear cost constraints. Implications of these observations for the channel coding problem and applications of the proof technique to related problems are discussed.

On the fast-fading discrete memoryless channels (DMCs) with channel state information at the receiver a necessary and sufficient condition is determined for the tilted channel associated with an input distribution to be the product of a tilted channel state distribution and the tilted channel associated with the same input distribution on the non-fading DMC for the given channel state. On fading channels for which constituent channels associated with different channel states share a common Augustin capacity-achieving input distribution aforementioned necessary and sufficient condition is shown to hold, and a parametric form for the sphere packing exponent (SPE) is obtained in terms of the tilted channel state distribution and the tilted channel of the nonfading DMC for the given channel state. The SPE of fast-fading binary erasure and binary symmetric channels with channel state information at the receiver are analyzed as examples.

The existence of a unique Augustin mean and its invariance under the Augustin operator are established for arbitrary input distributions with finite Augustin information for channels with countably generated output $\sigma$-algebras. The existence is established by representing the conditional Rényi divergence as a lower semi-continuous and convex functional in an appropriately chosen uniformly convex space and then invoking the Banach-Saks property in conjunction with the lower semi-continuity and the convexity. A new family of operators is proposed to establish the invariance of the Augustin mean under the Augustin operator for orders greater than one. Some members of this new family strictly decrease the conditional Rényi divergence, when applied to the second argument of the divergence, unless the second argument is a fixed point of the Augustin operator.

A strong converse bound for constant composition codes of the form P(n)e≥ 1-An-0.5(1-E0sc(R,W,p))e-nEsc(R,W,p)is established using the Berry-Esseen theorem through the concepts of Augustin information and Augustin mean, where A is a constant determined by the channel W , the composition p, and the rate R, i.e., A does not depend on the block length n.

A judicious application of the Berry-Esseen theorem via the concepts of Augustin information and mean is demonstrated to be sufficient for deriving the sphere packing bound with a prefactor that is Ω (n-0.5(1-E'sp(R,W,p))) for the constant composition codes. The resulting non-asymptotic bounds have definite approximation error terms.

Establishing the sphere packing bound for block codes on the discrete stationary product channels with feedback—which are commonly called the discrete memoryless channels with feedback—was considered to be an open problem until recently, notwithstanding the proof sketch provided by Augustin in 1978. A complete proof following Augustin’s proof sketch is presented to demonstrate its adequacy and to draw attention to two novel ideas that it employs. These novel ideas (i.e., the Augustin’s averaging and the use of subblocks) are likely to be applicable in other communication problems for establishing impossibility results.

A sphere packing bound (SPB) with a prefactor that is polynomial in the block length n is established for codes on a length n product channel W[1,n], assuming that the maximum order 1/2 Rényi capacity among the component channels, i.e. maxt∈[1,n]C1/2,Wt, is O(lnn). The reliability function of the discrete stationary product channels with feedback is bounded from above by the sphere packing exponent. Both results are proved by first establishing a non-asymptotic SPB. The latter result continues to hold under a milder stationarity hypothesis.

Rényi's information measures-the Rényi information, mean, capacity, radius, and center-are analyzed relying on the elementary properties of the Rényi divergence and the power means. The van Erven- Harremoës conjecture is proved for any positive order and for any set of probability measures on a given measurable space and a generalization of it is established for the constrained variant of the problem. The finiteness of the order Rényi capacity is shown to imply the continuity of the Rényi capacity on (0, α] and the uniform equicontinuity of the Rényi information, both as a family of functions of the order indexed by the priors and as a family of functions of the prior indexed by the orders.

For any channel with a convex constraint set and finite Augustin capacity, existence of a unique Augustin center and associated Erven-Harremoes bound are established. Augustin-Legendre capacity, center, and radius are introduced and proved to be equal to the corresponding Renyi-Gallager entities. Sphere packing bounds with polynomial prefactors are derived for codes on two families of channels: (possibly non-stationary) memoryless channels with multiple additive cost constraints and stationary memoryless channels with convex constraints on the empirical distribution of the input codewords.

The bit-wise unequal error protection problem, for the case when the number of groups of bits${\ell}$is fixed, is considered for variable-length block codes with feedback. An encoding scheme based on fixed-length block codes with erasures is used to establish inner bounds to the achievable performance for finite expected decoding time. A new technique for bounding the performance of variable-length block codes is used to establish outer bounds to the performance for a given expected decoding time. The inner and the outer bounds match one another asymptotically and characterize the achievable region of rate-exponent vectors, completely. The single-message message-wise unequal error protection problem for variable-length block codes with feedback is also solved as a necessary step on the way.

Inner and outer bounds are derived on the optimal performance of fixed-length block codes on discrete memoryless channels with feedback and errors-and-erasures decoding. First, an inner bound is derived using a two-phase encoding scheme with communication and control phases together with the optimal decoding rule for the given encoding scheme, among decoding rules that can be represented in terms of pairwise comparisons between the messages. Then, an outer bound is derived using a generalization of the straight-line bound to errors-and-erasures decoders and the optimal error-exponent tradeoff of a feedback encoder with two messages. In addition, upper and lower bounds are derived, for the optimal erasure exponent of error-free block codes in terms of the rate. Finally, a proof is provided for the fact that the optimal tradeoff between error exponents of a two-message code does not improve with feedback on discrete memoryless channels (DMCs).

A lower bound bound is established on the error probability of fixed-length block-coding systems with finite memory feedback, which can be described in terms of a time dependent finite state machine. It is shown that the reliability function of such coding systems over discrete memoryless channels is upper-bounded by the sphere-packing exponent.

Bit-wise unequal error protection problem with two layers is considered for variable length block-codes with feedback. Inner and outer bounds are derived for achievable performance for finite expected decoding time. These bounds completely characterize the error exponent of the special bits as a function of overall rate R, overall error exponent E and the rate of the special bits Rs. Single message Message-wise unequal protection problem is also solved as a step on the way.

Schalkwijk and Kailath (1966) developed a class of block codes for Gaussian channels with ideal feedback for which the probability of decoding error decreases as a second-order exponent in block length for rates below capacity. This well-known but surprising result is explained and simply derived here in terms of a result by Elias (1956) concerning the minimum mean-square distortion achievable in transmitting a single Gaussian random variable over multiple uses of the same Gaussian channel. A simple modification of the Schalkwijk-Kailath scheme is then shown to have an error probability that decreases with an exponentialorderwhich is linearly increasing with block length. In the infinite bandwidth limit, this scheme produces zero error probability using bounded expected energy at all rates below capacity. A lower bound on error probability for the finite bandwidth case is then derived in which the error probability decreases with an exponential order which is linearly increasing in block length at the same rate as the upper bound.

A new technique is proposed for upper bounding the error probability of fixed length block codes with feedback. Error analysis is inspired by Gallager's error analysis for block codes without feedback. Zigangirov-D'yachkov (Z-D ) encoding scheme is analyzed with the technique on binary input channels and k-ary symmetric channels. A strict improvement is obtained for k-ary symmetric channels.

In a remarkable paper published in 1976, Burnashev determined the reliability function of variable-length block codes over discrete memoryless channels (DMCs) with feedback. Subsequently, an alternativeachievabilityproof was obtained by Yamamoto and Itoh via a particularly simple and instructive scheme. Their idea is to alternate between a communication and a confirmation phase until the receiver detects the codeword used by the sender to acknowledge that the message is correct. We provide aconversethat parallels the Yamamoto-Itoh achievability construction. Besides being simpler than the original, the proposed converse suggests that a communication and a confirmation phase are implicit in any scheme for which the probability of error decreases with the largest possible exponent. The proposed converse also makes it intuitively clear why the terms that appear in Burnashev's exponent are necessary.

An information-theoretic framework for unequal error protection is developed in terms of the exponential error bounds. The fundamental difference between thebit-wiseandmessage-wiseunequal error protection (UEP) is demonstrated, for fixed-length block codes on discrete memoryless channels (DMCs) without feedback. Effect of feedback is investigated via variable-length block codes. It is shown that, feedback results in a significant improvement in bothbit-wiseandmessage-wise UEPs(except the single message case for missed detection). The distinction between false-alarm and missed-detection formalizations formessage-wise UEPis also considered. All results presented are at rates close to capacity.

Various formulations are considered where some information is more important than other and needs better protection. Our information theoretic framework in terms of exponential error bounds provides some fundamental limits and optimal strategies for such problems of unequal error protection. Even for data-rates approaching the channel capacity, it shows how a crucial part of information can be protected with exponential reliability. Channels without feedback are analyzed first, which is useful later in analyzing channels with feedback. A new channel parameter, called the Red-Alert Exponent, is fundamentally important in such problems.

Fixed length block codes on discrete memoryless channels with feedback are considered for errors and erasures decoding. Upper and lower bounds are derived for the error exponent in terms of the rate and the erasure exponents. In addition the converse result of Burnashev for variable length block codes is extended to include list decoding.

Variable-length block-coding schemes are investigated for discrete memoryless channels with ideal feedback under cost constraints. Upper and lower bounds are found for the minimum achievable probability of decoding error Pe,min as a function of constraints R, P, and tau on the transmission rate, average cost, and average block length, respectively. For given R and P, the lower and upper bounds to the exponent -( ln Pe,min)/tau are asymptotically equal as tau rarr infin. The resulting reliability function,limtaurarrinfin(-In Pe,min)/tau as a function of R and V, is concave in the pair (R,P) and generalizes the linear reliability function of Burnashev to include cost constraints. The results are generalized to a class of discrete-time memoryless channels with arbitrary alphabets, including additive Gaussian noise channels with amplitude and power constraints.

Variable-length block-coding schemes are investigated for discrete memoryless channels (DMC) with perfect feedback under cost constraints. Upper and lower bounds are found for the minimum achievable probability of decoding error Pepsi,minas a function of transmission rate R, cost constraint P, and expected block length taumacr. For given P and R, the lower and upper bounds to the exponent -(InPepsi,min)/taumacr are asymptotically equal as taumacr rarrinfin. The reliability function, limtaurarrinfin(-ln Pepsi,min)/taumacr, as a function of P and R, is concave in the pair (P, R) and generalizes the linear reliability function of Burnashev (M.V. Burnashev, 1976) to include cost constraints