We investigate the joint source-channel coding (JSCC) excess distortion exponentEJ(the exponent of the probability of exceeding a prescribed distortion level) for some memoryless communication systems with continuous alphabets. We first establish upper and lower bounds forEJfor systems consisting of a memoryless Gaussian source under the squared-error distortion fidelity criterion and a memoryless additive Gaussian noise channel with a quadratic power constraint at the channel input. A necessary and sufficient condition for which the two bounds coincide is provided, thus exactly determining the exponent. This condition is observed to hold for a wide range of source-channel parameters. As an application, we study the advantage in terms of the excess distortion exponent of JSCC over traditional tandem (separate) coding for Gaussian systems. A formula for the tandem exponent is derived in terms of the Gaussian source and Gaussian channel exponents, and numerical results show that JSCC often substantially outperforms tandem coding. The problem of transmitting memoryless Laplacian sources over the Gaussian channel under the magnitude-error distortion is also carried out. Finally, we establish a lower bound forEJfor a certain class of continuous source-channel pairs when the distortion measure is a metric.

We study the transmission of two discrete memoryless correlated sources, consisting of a common and a private source, over a discrete memoryless multiterminal channel with two transmitters and two receivers. At the transmitter side, the common source is observed by both encoders but the private source can only be accessed by one encoder. At the receiver side, both decoders need to reconstruct the common source, but only one decoder needs to reconstruct the private source. We hence refer to this system by the asymmetric two-user source-channel coding system. We derive a universally achievable lossless joint source-channel coding (JSCC) error exponent pair for the two-user system by using a technique which generalizes Csiszar's type-packing lemma (1980) for the point-to-point (single-user) discrete memoryless source-channel system. We next investigate the largest convergence rate of asymptotic exponential decay of the system (overall) probability of erroneous transmission, i.e., the system JSCC error exponent. We obtain lower and upper bounds for the exponent. As a consequence, we establish a JSCC theorem with single-letter characterization and we show that the separation principle holds for the asymmetric two-user scenario. By introducing common randomization, we also provide a formula for the tandem (separate) source-channel coding error exponent. Numerical examples show that for a large class of systems consisting of two correlated sources and an asymmetric multiple-access channel with additive noise, the JSCC error exponent considerably outperforms the corresponding tandem coding error exponent.

Consider transmitting two discrete memoryless correlated sources, consisting of a common and a private source, over a discrete memoryless multi-terminal channel with two transmitters and two receivers. At the transmitter side, the common source is observed by both encoders but the private source can only be accessed by one encoder. At the receiver side, both decoders need to reconstruct the common source, but only one decoder needs to reconstruct the private source. We hence refer to this system by the asymmetric 2-user source-channel system. In this work, we derive a universally achievable joint source-channel coding (JSCC) error exponent pair for the 2-user system by using a technique which generalizes Csiszar's method (1980) for the point- to-point (single-user) discrete memoryless source-channel system. We next investigate the largest convergence rate of asymptotic exponential decay of the system (overall) probability of erroneous transmission, i.e., the system JSCC error exponent. We obtain lower and upper bounds for the exponent. As a consequence, we establish the JSCC theorem with single letter characterization.

We study the error exponent, EJ, for reliably transmitting a discrete stationary ergodic Markov (SEM) source Q over a discrete channel W with additive SEM noise via a joint source-channel (JSC) code. We first establish an upper bound for EJin terms of the Renyi entropy rates of the source and noise processes. We next investigate the analytical computation of EJby comparing our bound with Gallager's lower bound (1968) when the latter one is specialized to the SEM source-channel system. We also note that both bounds can be represented in Csiszar's form (1980), as the minimum of the sum of the source and channel error exponents. Our results provide us with the tools to systematically compare EJwith the tandem (separate) coding exponent EJ. We show that as in the case of memoryless source-channel pairs EJles 2Erand we provide explicit conditions for which EJ> ET. Numerical results indicate that EJap 2ETfor many SEM source-channel pairs, hence illustrating a substantial advantage of JSC coding over tandem coding for systems with Markovian memory.

For a memoryless Gaussian source under the squared-error distortion fidelity criterion and a memoryless additive Gaussian noise channel with a quadratic power constraint at the channel input, upper and lower bounds for the joint source-channel coding excess distortion exponent (which is the exponent of the probability of excess distortion) are established. A necessary and sufficient condition for which the two bounds coincide is provided, thus exactly determining the exponent. This condition is observed to hold for a wide range of source-channel parameters

We investigate the computation of Csisza/spl acute/r's bounds for the joint source-channel coding (JSCC) error exponent E/sub J/ of a communication system consisting of a discrete memoryless source and a discrete memoryless channel. We provide equivalent expressions for these bounds and derive explicit formulas for the rates where the bounds are attained. These equivalent representations can be readily computed for arbitrary source-channel pairs via Arimoto's algorithm. When the channel's distribution satisfies a symmetry property, the bounds admit closed-form parametric expressions. We then use our results to provide a systematic comparison between the JSCC error exponent E/sub J/ and the tandem coding error exponent E/sub T/, which applies if the source and channel are separately coded. It is shown that E/sub T//spl les/E/sub J//spl les/2E/sub T/. We establish conditions for which E/sub J/>E/sub T/ and for which E/sub J/=2E/sub T/. Numerical examples indicate that E/sub J/ is close to 2E/sub T/ for many source-channel pairs. This gain translates into a power saving larger than 2 dB for a binary source transmitted over additive white Gaussian noise (AWGN) channels and Rayleigh-fading channels with finite output quantization. Finally, we study the computation of the lossy JSCC error exponent under the Hamming distortion measure.

We establish an upper bound for the joint source-channel coding (JSCC) error exponent E/sub J/(Q, W) for a discrete stationary ergodic Markov (SEM) source Q and a discrete channel W with additive SEM noise. This bound, which is expressed in terms of the Renyi entropy rates of the source and noise processes, admits an identical form to Csiszar's sphere-packing upper bound for the JSCC error exponent for memoryless systems (I. Csiszar, Nov. 1982). In this regard, our result is a natural extension of Csiszar's upper bound of the JSCC error exponent from the case of memoryless systems to the case of SEM systems. We also investigate the analytical computation of E/sub J/(Q,W) by comparing our bound with Gallager's random-coding lower bound (R. G. Gallager, 1968), when the latter one is specialized to the SEM source-channel system.

We study the analytical computation of Csiszar's [1980] random-coding lower bound and sphere-packing upper bound for the lossless joint source-channel (JSC) error exponent, E/sub J/(Q, W), for a discrete memoryless source (DMS) Q and a discrete memoryless channel (DMC) W. We provide equivalent expressions for these bounds, which can be readily calculated for arbitrary (Q,W) pairs. We also establish explicit conditions under which the bounds coincide, thereby exactly determining E/sub J/(Q,W).

In this work, we provide a computable expression for the Kullback-Leibler divergence rate lim/sub n/spl rarr//spl infin//1/nD(p/sup (n)//spl par/q/sup (n)/) between two time-invariant finite-alphabet Markov sources of arbitrary order and arbitrary initial distributions described by the probability distributions p/sup (n)/ and q/sup (n)/, respectively. We illustrate it numerically and examine its rate of convergence. The main tools used to obtain the Kullback-Leibler divergence rate and its rate of convergence are the theory of nonnegative matrices and Perron-Frobenius theory. Similarly, we provide a formula for the Shannon entropy rate lim/sub n/spl rarr//spl infin//1/nH(p/sup (n)/) of Markov sources and examine its rate of convergence.

In this work, we examine the existence and the computation of the Renyi divergence rate, lim/sub n/spl rarr//spl infin// 1/n D/sub /spl alpha//(p/sup (n)//spl par/q/sup (n)/), between two time-invariant finite-alphabet Markov sources of arbitrary order and arbitrary initial distributions described by the probability distributions p/sup (n)/ and q/sup (n)/, respectively. This yields a generalization of a result of Nemetz (1974) where he assumed that the initial probabilities under p/sup (n)/ and q/sup (n)/ are strictly positive. The main tools used to obtain the Renyi divergence rate are the theory of nonnegative matrices and Perron-Frobenius theory. We also provide numerical examples and investigate the limits of the Renyi divergence rate as /spl alpha//spl rarr/1 and as /spl alpha//spl darr/0. Similarly, we provide a formula for the Renyi entropy rate lim/sub n/spl rarr//spl infin// 1/n H/sub /spl alpha//(p/sup (n)/) of Markov sources and examine its limits as /spl alpha//spl rarr/1 and as /spl alpha//spl darr/0. Finally, we briefly provide an application to source coding.

For channels which suffer predominantly from additive noise and intersymbol interference, the decision-feedback equalizer has provided a relatively simple solution for reducing the effects of interfering symbols at the input to the decision device. A technique is developed that enables fast, accurate calculation of the error performance of decision-feedback equalization for a number of channel models. The method is to calculate the n-step transition probability for an associated Markov process and then use this transition probability as an approximation to the stationary probability distribution. For systems with finite memory, it is proved that the method converges. If the signal-to-noise ratio (SNR) is high and the signal amplitude is more than twice the worst-case interference, it is shown that the convergence is rapid. Numerical results indicate that the convergence is rapid enough to make this an efficient method of calculation, even for channels for which the interference does not fully satisfy this condition. Two examples are given here, but the technique has been tested on most of the examples that have been presented in the literature. The method yields results in closer agreement with simulation results than previous results obtained using bounding techniques, especially at low to moderate SNRs, and requires less computation.

Given a prior estimate of a probability, q, and a constraint /spl Sigma/p/sub i/x/sub i/=a, one well-known way of estimating p is to minimize the cross-entropy I(p; q) subject to the constraint. A modification to this method is proposed for use when the value a is only approximately known. The modification is based on the penalty function method in constrained optimization. It has an interpretation in differential geometry methods in statistics and it sometimes gives a maximum-likelihood estimate.

Several treatments of interference problems are examined with a view to determine whether doubtful mathematical assumptions about the existence of probability density functions affect the conclusions. Most derivations can be recast so that the existence of a density function is not necessary for the conclusions to be valid.

It is pointed out that a uniform distribution on probability n-tuples is not necessarily the best distribution to use in calculating an average entropy. The noninformative prior of Bayesian statistics and certain-distributions which arise in differential-geometry approaches to statistics are other candidates. The mean and variance of the entropy are calculated when probability n-tuples are distributed according to these distributions.>

This paper presents some results on the cryptographic strength of Boolean functions from the information theoretic point of view. It is argued that a Boolean function is resistant to statistical analysis if there is no significant static and dynamic information leakage between its inputs and its output(s). In particular we relate information leakage to nonlinearity, higher order SAC, correlation immunity and resilient functions. It is shown that reducing information leakage increases resistance to the differential attack and the linear attack. We note that some conventional cryptographic criteria require zero static or dynamic information leakage in only one domain. Such a requirement can result in a large information leakage in another domain. To avoid this weakness, it is better to jointly constrain all kinds of information leakage in the function. In fact, we claim that information leakage can be used as a fundamental measure of the strength of a cryptographic algorithm.

Sums of infinite sequences of weighted binary random variables arise in communications problems involving signal-dependent interferences. In many cases of practical importance, the distribution functions of these sums are singular and often of Cantor type; they are continuous but do not have a density function. For this reason, special methods of calculating expectations are needed. Results of this type are derived. The method is used to compute error probabilities for differential detection of minimum shift keying, and for noncoherent detection of frequency shift keying. In each case the model assumed is a Rician fading channel.>

A derivation of the joint distribution of the amplitude and angle of a sinusoid in Gaussian noise is given. No assumptions about the structure of the noise at the output of the post-detection filter are required. However, it is assumed that the Gaussian noise at the input is generated by passing white noise through the bandpass equivalent of a single-pole low-pass filter. Hence, the noise is a two-dimensional Ornstein-Uhlenbeck or Gauss-Markov process. In practice, a higher-order bandpass filter would be encountered in FM reception. However, the approach is intended as a first step towards the goal of an understanding of the phase process. The method used is to derive and solve the Fokker-Planck partial differential equation that governs the joint distribution. An explicit integral formula for the solution is obtained. The integral is, in general, rather difficult to evaluate. A further integration over amplitude is required if the angle distribution rather than the joint distribution is required. For some special-cases, where the filtering time is large and the signal-to-noise power ratio is very large or very small, explicit approximate expressions are given.>

The sum of an infinite series of weighted binary random variables arises in communications problems involving intersymbol and adjacent-channel interference. If the weighting decays asymptotically at least exponentially and if the decay is not too slow, the sum has an unusual distribution which has neither a density nor a discrete mass function, and therefore cannot be manipulated with usual techniques. The distribution of the sum is given, and the calculus for dealing with the distribution is presented. It is shown that these Cantor-type random variables arise in a range of digital communications models, and exact explicit expressions for performance measures, such as the probability of error, may be obtained. Several examples are given.>

This paper generalizes some aspects of Jain and Blachman's analysis of the error probability for binary PSK transmission over a two-link channel having a hard-limiting repeater. In the present analysisM-ary PSK transmission through a repeater having both nonlinear amplitude and phase characteristics is considered. The error probability is given in terms of an infinite series whosenth term must be obtained by a one-dimensional numerical integration. A practical amplitude and phase characteristic from the literature is treated. Also presented is a study of the effect on error probability of "backing-off" the repeater from saturation.

In this paper we specify an optimal receiver filter for 2phase PSK transmission over a channel consisting of the tandem connection of a linear filter and a bandpass nonlinearity, with thermal noise added at the channel output. This filter minimizes the bit estimation error subject to the constraint that it be a linear filter. Our main aim is to consider satellite communication channels and as such the system nonlinearity is taken to have both AM/AM and AM/PM conversions. In these systems we assume the up-link signal-to-noise ratio is large and thus our work relates to large transmit terminal applications. In an example the probability of error performance of the optimal received filter is found to be 2 dB better in terms of the effective output SNR as compared with that of a standard choice for the receiver filter.

The problem of designing and evaluating the performance of a maximum likelihood sequence receiver for binary PSK transmission over bandlimited nonlinear channels is considered in this paper. The effects of intersymbol interference followed by AM/AM and AM/PM conversions are taken into account while optimizing the performance in the presence of white Gaussian noise. A new representation for the output of a bandpass nonlinearity is given when the input consists of a carrier signal modulated by a sum of binary overlapping pulses. The structure of a maximum likelihood sequence receiver for a bandlimited nonlinear channel is derived using this representation. The receiver uses a modified Viterbi algorithm to determine the most likely sequence of data symbols transmitted. An upperbound on the probability of symbol error for this receiver is obtained. Numerical results illustrating the applicability of the present work to optimizing the performance of a digital satellite communications link are also presented.

Letd_1(\alpha, \beta)be a distortion measure that is row balanced if the source alphabet is a finite set and that has the formd(\alpha - \beta)if the source alphabet is the real line. ForN-tuples, let the distortion measured_Nbe the single-letter measure. Consider a variable-length code in which there areDcode symbols and for which the length of the codeword for\alphaisn(\alpha). Codewordsw(u)forN-tuplesuare formed by concatenating codewords for the individual letters. LetE(N) = \sup d_N (u,v)where the supremum is over all pairs(u,v)for whichw(u) = w(v). Call the code\varepsilon-decodable if\lim E(N) = \varepsilon. If0 = d_1(\alpha,\beta) < d_1(\alpha,\beta)for\alpha \neq \beta, and if the code is uniquely decipherable, then\varepsilon = 0. For a discrete source it is shown that\sum D^{-n(i) -h(\varepsilon)} \leq 1, whereh(0) = 0andh(\varepsilon) > 0if\varepsioln > 0. For a continuous source for which values from the interval[-c,c]are encoded,\int_{-c}^c D^{-n(z)-h_1 (varepsilon)} dz \leq 1, whereh_1 (\varepsilon)is a known function. These inequalities are used to obtain lower bounds on the mean length of any\varepsilon-decodable code. In many cases, these lower bounds coincide with the rate-distortion functionR(\varepsilon)associated with the same distortion measure.

The problem of detection of a sinusoidal signal in the presence of white Gaussian noise and an interfering sinusoid at a nearby frequency is discussed. In the case of coherent detection, several possible receivers are analyzed and probability of error curves are calculated. In some cases it is possible to reduce the effect of cochannel interference significantly by proper choice of a receiver. In the case of incoherent detection, error probability curves have been calculated for the standard envelope detector for several values of frequency separation. The performance of the envelope detector can be degraded substantially by the presence of an interfering sinusoid.

A probability space of a special type is put into correspondence with a measure space. Under this correspondence, sets in the measure space correspond to partitions of the probability space and the measure of a set equals the entropy of the corresponding partition.

Consider a system which consists of two receivers, each containing a nonlinear device followed by a zonal filter. A general method is developed for calculating the cross-correlation function of the outputs of these receivers when the inputs are two related narrow-band Gaussian processes. In the course of the development some new results are obtained concerning the cross-correlation function of two pre-envelopes.

The so-called "pseudo-random" (p-r) sequences defined below have two properties which may make them useful in certain communication systems. Stated roughly, the first property is that the minimum distance between different signals of a special class is maximized and the second is that the probability that one member of this class be mistaken for some other member is minimized. The so-called "pseudo-random" (p-r) sequences defined below have two properties which may make them useful in certain communication systems. Stated roughly, the first property is that the minimum distance between different signals of a special class is maximized and the second is that the probability that one member of this class be mistaken for some other member is minimized.

An expression for the error rate in a system using a binary pulse position code is derived. In the system considered, the pulses amplitude modulate a carrier and the resultant signal is contaminated by additive Gaussian noise. At the receiver the pulses are recovered by an envelope detector. If synchronization errors and post-detection filtering are neglected, it is shown that the probability of a binary error is approximated well by1/2 \exp (-a^2/2), wherea^2is the peak input signal-to-noise power ratio. Finally, the error rate is derived for the case where the signal amplitude is subject to random fading. Some comparisons are made with error rates derived by Montgomery for other systems with and without carrier fading. It is found that when the signal is subject to fading the pulse position system is better than a comparable system using threshold detection.

The spectrum of the output of a half-wave rectifier is derived for an input which is the sum of random noise and two sinusoidal signals of different frequencies. The method used is the characteristic function method described by Rice. The components of the output spectrum are given as infinite series of hypergeometric functions. If both the input signals .are small compared with the noise, it is shown that the ratio of the output signal power at the difference frequency to the output noise power is proportional to the product of the input signal-to-noise power ratios at the two frequencies. If one of the input signals is very large compared with the noise, it is shown that the other signal and the noise are translated in frequency without alteration of the signal-to-noise ratio. A correction factor is obtained for the case where the large signal is not quite large enough. Finally, the output signal-to-noise ratio of a single-sideband detector is calculated as a function of the input signal-to-noise ratio, when the sideband amplitude is one-half the carrier amplitude.