The Körner-Marton modulo two-sum problem is described as a variation of the lossless two-help-one source coding problem.

Principles behind lossless and lossy coding are usually considered related, yet distinct. In contrast, we show that the direct statements of the rate-distortion theorem and the lossless coding theorem are consequences of a common distortion-abstracted phenomenon. Significantly, we extend such distortion abstraction to a more general multiterminal framework, and derive a canonical direct theorem that subsumes known results. Further, we show that the converse holds if all but at most one encoded sources are perfectly reconstructed, and, thereby, not only generalize known results but settle open problems such as the single-helper problem. More generally, a canonical sequence of inner bounds approaches the achievable region, and points to a new problem, whose solution, if found, would lead to a desirable computable description.

In this correspondence, the stopping set of turbo codes with iterative decoding in the binary erasure channel is defined. Block and bit erasure probabilities of turbo codes are studied by using the stopping set analysis. It is found that block and bit erasure probabilities of turbo codes with iterative decoding are higher than those with maximum-likelihood decoding, where the differences are negligible in the error floor region. It is shown that in the error floor region, block and bit erasure probabilities of turbo codes with iterative decoding are dominated by small stopping sets and are asymptotically dominated by low weight codewords.

We propose the signal-to-noise ratio (SNR) transfer characteristic band (TCB) chart of extrinsic information (EI) as a basic tool for the probabilistic convergence analysis of iterative decoding. The proposed SNR TCB chart is a general version of the widely used SNR transfer chart, and it can model the behavior of EI in iterative decoding with any blocklength. We propose an algorithm using the SNR TCB chart to obtain the information, which is both quantitative and qualitative, on the probabilistic convergence behavior of iterative decoding. We also propose the bit-error rate (BER) estimate, in the form of a lower bound, of turbo codes over an additive white Gaussian noise channel. The proposed estimate is especially useful in estimating the BER of finite-length turbo codes in a waterfall region, while it is meaningful for all blocklengths and all Eb/N0

Euclidean distance is a discrepancy measure between two real-valued functions. Divergence is a discrepancy measure between two positive functions. Corresponding to these two well-known discrepancy measures, there are two inference principles; namely, the least-squares principle for choosing a real-valued function subject to linear constraints, and the minimum-divergence principle for choosing a positive function subject to linear constraints. To make the connection between these two principles more transparent, this correspondence provides an observation and a constructive proof that the minimum-divergence principle reduces to the least-squares principle asymptotically as the positivity requirements are de-emphasized. Hence, these two principles are asymptotically consistent.

Csiszár's I-divergence is adopted as a discrepancy measure for the nonnegative impulse response intersymbol interference (ISI) channel demodulation. Both blind and nonblind I-divergence demodulation algorithms are discussed and numerically studied. Simulation results show that they have bit error rate (BER) performance close to the Viterbi algorithm for the one-dimensional channel. Furthermore, the asymptotic optimality of the I-divergence demodulation algorithms is studied theoretically. It is shown that the I-divergence discrepancy measure agrees with the Euclidean distance asymptotically. The role of the progressive thresholding in improving the BER performance is discussed. Estimation of the channel impulse response function is asymptotically maximum-likelihood optimal under the nonnegativity constraints. The complexity of the I-divergence detection algorithms is significantly less than that of the Viterbi algorithm for long ISI channels.

The issue of source/channel duality is as old as information theory itself. Whereas type covering has been identified as a basis for source coding and sphere packing as the basis for channel coding, very little effort has been directed at tying covering and packing together. In this paper, a covering result has been derived which gives asymptotically optimal packing as well. We propose our covering/packing lemma as a mathematical basis of duality. Our lemma also sheds light on outstanding issues of source coding such as explicit rate-distortion tradeoff and unifying basis for lossless and lossy coding

We will illustrate the connection between the Ising problem in statistical mechanics and the problem of computing the constrained capacity of an array of the same dimension. Using this connection, we show that for a given amount of violation, a soft constrained capacity can be computed. The classical Shannon capacity of a constrained channel is merely an end point of the soft capacity curve, where no violations are allowed. Moreover we reduce the problem of computing the constrained capacity to that of computing the eigenvalues of a special matrix. We claim that an analytical solution to calculating the eigenvalues of interest corresponds to solving the special case of the two-dimensional constrained channel with the constraint (1,infin)

We present a single letter characterization of the secrecy capacity of the single-input multiple-outputs (SIMO) channel under Gaussian (and possibly colored) noise. To do so, we transform the channel into a scalar Gaussian wiretap channel using standard techniques of communications theory. The result is used to study the impact of slow fading on the secrecy capacity of the channel, and how the use of multiple receive antennas could improve the performance of the system

Multimode optical fiber links for local area networks have been challenged by the ever increasing demand for higher data rates. The understanding of their fundamental limits thus becomes crucial. Recent research results show that multimode optical fiber channels have much higher channel capacity from the information-theoretic point of view. A modulation method utilizing the space diversity of multimode optical fibers is introduced and studied in this paper. It asymptotically approaches the channel capacity promised by information theory. Experiments are also conducted to demonstrate the feasibility.

We propose a tool for the analysis of the finite-length iterative turbo decoding algorithm. The proposed tool is a generalized EXIT (extrinsic information transfer) chart based on the mutual information transfer characteristics of the extrinsic information in the iterative turbo decoding algorithm. The proposed tool can describe the probabilistic convergence behavior of the iterative decoding algorithm. By using this tool, we obtain the BER estimates of the finite-length turbo codes in the form of a lower bound, which shows a gentle waterfall over a wide waterfall region. The obtained lower bound is in reasonable agreement with the BER obtained by simulations of the iterative decoding algorithm.

A novel method of estimating the BER performance of finite length turbo codes in a binary AWGN channel is proposed. We consider the asymptotes of BER performance at low and high E/sub b//N/sub 0/ resulted from the small and large extrinsic values, respectively. The BER performance curve transitions between two asymptotes in the waterfall region, where the transition probability is approximately obtained by using the SNR transfer characteristic band of extrinsic values in the turbo decoding algorithm. Then, the BER estimate of finite length turbo codes is obtained by using two asymptotes at low and high E/sub b//N/sub 0/ and the transition probability between them.

The emergent role of information theory in image formation is surveyed. Unlike the subject of information-theoretic communication theory, information-theoretic imaging is far from a mature subject. The possible role of information theory in problems of image formation is to provide a rigorous framework for defining the imaging problem, for defining measures of optimality used to form estimates of images, for addressing issues associated with the development of algorithms based on these optimality criteria, and for quantifying the quality of the approximations. The definition of the imaging problem consists of an appropriate model for the data and an appropriate model for the reproduction space, which is the space within which image estimates take values. Each problem statement has an associated optimality criterion that measures the overall quality of an estimate. The optimality criteria include maximizing the likelihood function and minimizing mean squared error for stochastic problems, and minimizing squared error and discrimination for deterministic problems. The development of algorithms is closely tied to the definition of the imaging problem and the associated optimality criterion. Algorithms with a strong information-theoretic motivation are obtained by the method of expectation maximization. Related alternating minimization algorithms are discussed. In quantifying the quality of approximations, global and local measures are discussed. Global measures include the (mean) squared error and discrimination between an estimate and the truth, and probability of error for recognition or hypothesis testing problems. Local measures include Fisher information.

We define a checkerboard code as a two-dimensional binary code that satisfies some constraint, e.g., every binary one must be surrounded by eight zeros, and then investigate the capacity for each of several different constraints. Using a recursive construction we develop a series of loose bounds on the capacity. These bounds, in turn, lead to conjecturally precise estimates of the capacity by the use of a numerical convergence-speeding technique called Richardson extrapolation. Finally, using the value of the capacity, we define and compute a measure of coding gain which allows us to compare checkerboard codes to simple coding schemes.

The Gleason-Prange theorem describes a nontrivial automorphism of an extended quadratic residue code. By using the Fourier transform in an appropriate extension field, methods of digital signal processing are used to make the proof transparent and the automorphism intuitive. The theorem is proved both in a field of characteristic tow and in an arbitrary field and is stated simply as a general property of the discrete Fourier transform.>

This survey paper is intended to integrate the subjects of digital signal processing and error control codes by studying their common dependence on the properties of the discrete Fourier transform. The two subjects are traditionally studied in different algebraic fields. Usually, the computations of digital signal processing are done using the complex number system, while the computations of error control codes are done using the arithmetic of Galois fields. We will argue that this dichotomy may be partly a historical accident. By viewing the two problems in the opposite number system, we shall find that there are parallels and that many techniques can be shared by the two subjects. The new material included within the paper is introduced in order to extend known techniques used in one algebraic field into another algebraic field where those techniques are not yet used.

The performance of Channel block codes for a general channel is studied by examining the relationship between the rate of a code, the joint composition of pairs of codewords, and the probability of decoding error. At fixed rate, lower bounds and upper bounds, both on minimum Bhattacharyya distance between codewords and on minimum equivocation distance between codewords, are derived. These bounds resemble, respectively, the Gilbert and the Elias bounds on the minimum Hamming distance between codewords. For a certain large class of channels, a lower bound on probability of decoding error for low-rate channel codes is derived as a consequence of the upper bound on Bhattacharyya distance. This bound is always asymptotically tight at zero rate. Further, for some channels, it is asymptotically tighter than the straight line bound at low rates. Also studied is the relationship between the bounds on codeword composition for arbitrary alphabets and the expurgated bound for arbitrary channels having zero error capacity equal to zero. In particular, it is shown that the expurgated reliability-rate function for blocks of letters is achieved by a product distribution whenever it is achieved by a block probability distribution with strictly positive components.

A large class of lower bounds relating to the performance of hypothesis testers, channel codes, and source compression codes is developed. These are extensions of Fano's inequality on the one hand, and of the discrimination inequality of Kullback on the other. The hypothesis testing and channel coding bounds are interesting primarily for small blocklengths and, in general, are asymptotically inferior to the well-known exponentially decreasing bounds. The source compression results include new proofs of converse coding theorems. A lower bound is given to the probability that a source produces an output block which cannot be encoded within a desired maximum distortion.

The testing of binary hypotheses is developed from an information-theoretic point of view, and the asymptotic performance of optimum hypothesis testers is developed in exact analogy to the asymptotic performance of optimum channel codes. The discrimination, introduced by Kullback, is developed in a role analogous to that of mutual information in channel coding theory. Based on the discrimination, an error-exponent functione(r)is defined. This function is found to describe the behavior of optimum hypothesis testers asymptotically with block length. Next, mutual information is introduced as a minimum of a set of discriminations. This approach has later coding significance. The channel reliability-rate functionE(R)is defined in terms of discrimination, and a number of its mathematical properties developed. Sphere-packing-like bounds are developed in a relatively straightforward and intuitive manner by relatinge(r)andE (R). This ties together the aforementioned developments and gives a lower bound in terms of a hypothesis testing model. The result is valid for discrete or continuous probability distributions. The discrimination function is also used to define a source code reliability-rate function. This function allows a simpler proof of the source coding theorem and also bounds the code performance as a function of block length, thereby providing the source coding analog ofE (R).

By defining mutual information as a maximum over an appropriate space, channel capacities can be defined as double maxima and rate-distortion functions as double minima. This approach yields valuable new insights regarding the computation of channel capacities and rate-distortion functions. In particular, it suggests a simple algorithm for computing channel capacity that consists of a mapping from the set of channel input probability vectors into itself such that the sequence of probability vectors generated by successive applications of the mapping converges to the vector that achieves the capacity of the given channel. Analogous algorithms then are provided for computing rate-distortion functions and constrained channel capacities. The algorithms apply both to discrete and to continuous alphabet channels or sources. In addition, a formalization of the theory of channel capacity in the presence of constraints is included. Among the examples is the calculation of close upper and lower bounds to the rate-distortion function of a binary symmetric Markov source.