We consider the estimation problem for jointly stable random variables. Under two specific dependency models: a linear transformation of two independent stable variables and a sub-Gaussian symmetric $α$-stable (S$α$S) vector, we show that the conditional mean estimator is linear in both cases. Moreover, we find dispersion optimal linear estimators. Interestingly, for the sub-Gaussian (S$α$S) vector, both estimators are identical generalizing the well-known Gaussian result of the conditional mean being the best linear minimum-mean square estimator.

We study the rate-distortion problem for both scalar and vector memoryless heavy-tailed α-stable sources (0 < α < 2). Using a recently defined notion of “strength” as a power measure, we derive the rate-distortion function for α-stable sources subject to a constraint on the strength of the error and show it to be logarithmic in the strength-to-distortion ratio. We show how our framework paves the way for finding optimal quantizers for α- stable sources and other general heavy-tailed ones. In addition, we study high-rate scalar quantizers and show that uniform ones are asymptotically optimal under the error-strength distortion measure. We compare uniform Gaussian and Cauchy quantizers and show that more representation points for the Cauchy source are required to guarantee the same quantization quality. Our findings generalize the well-known results of rate-distortion and quantization of Gaussian sources (α = 2) under a quadratic distortion measure.

A generalized notion of “relative entropy typicality” is introduced. The new definition accommodates non-i.i.d. scenarios and is parameterized by two families of parameters,$\left\{\delta^{[n]}\right\}_{n}$and$\left\{\epsilon^{[n]}\right\}_{n}$: the choice of$\left\{\delta^{[n]}\right\}_{n}$(the allowed “deviation” in the relative entropy typical set) can be optimized as a function of$\left\{\epsilon^{[n]}\right\}_{n}$, where ($1-\epsilon^{[n]}$) is the desired probability of the set. This generalized definition is shown to yield an extension of the Chernoff-Stein lemma, which characterizes the rate of decay of the type II error in hypothesis testing (under a fixed type I error constraint) as the KL divergence. In particular, the generalization accommodates both discrete and continuous random variables, non-i.i.d. settings, in addition to cases where the KL divergence grows non-linearly. Several example applications are discussed, including testing two correlated Gaussian distributions.

We propose a framework for the rate-distortion analysis that extends the Gaussian MSE setup for stable sources. Specifically, we study the rate-distortion function of$\alpha$-stable sources when the distortion constraint is specified using the recently defined notion of strength of the error variable. We also “generalize” the notion of$d$-tilted information under the new setup and use it to determine second-order terms for both fixed and variable-length codes.

We study the probability distribution of age of information (AoI) in arbitrary networks with memoryless service times. A source node generates packets following a Poisson process, and then the packets are forwarded across the network in such a way that newer updates preempt older ones. This model is equivalent to gossip networks that were recently studied by Yates, and for which he obtained a recursive formula allowing the computation for the average AoI. In this paper, we obtain a very simple characterization of the stationary distribution of AoI at every node in the network. This allows for the computation of the average of an arbitrary function of the age, such as the age-violation probabilities. Furthermore, we show how our simple characterization can yield substantial reductions in the computation time of average AoIs in some structured networks. Finally, we describe how it can yield faster and more accurate Monte Carlo simulations estimating the average AoI, or the average of an arbitrary function of the age.

We study the rates of source coding a T -seconds finite duration piece from a W -Hz band-limited real white Gaussian process with the mean squared error as a distortion measure. First we derive a discrete representation by projecting the signal of interest on the set of prolate spheroidal wave functions. We derive next a lower bound and an upper bound for the smallest rate that guarantees a distortion level with a probability (1 - ε) and we numerically evaluate these bounds. We also show that the derived bounds are asymptotically tight where they converge to Shannon's formula.

Practicalcommunication systems use finite duration pulses. As the time-limited pulses have infinite bandwidth, they lose some of their energy when passed through a band-limited channel and their support in time becomes infinite. We derive the achievable rates when using $T$ -seconds time-limited pulses in a combined PAM-OMM system over a $W$ -Hz band-limited Gaussian channel, and we numerically evaluate them. We show that these rates depend on the time-frequency factor $c=2WT$ . When $c \geq 1$ the loss of energy has no significant effect and the achievable rates are arbitrarily close to the channel capacity. Furthermore, we establish that there are at most 2WT independent data symbols.

In this paper we study optimal signaling when using time-limited pulses over band-limited Additive White Gaussian Noise channels. We adopt an information theoretic approach and quantify the achievable rates of such systems. We show that the Nyquist criterion cannot be satisfied and that signaling at faster than the Nyquist rate is significantly closer to optimality.

We study the classical problem of characterizing the channel capacity and its achieving distribution in a generic fashion. We derive a simple relation between three parameters: the input-output function, the input cost function, and the noise probability density function, one which dictates the type of the optimal input. In layman terms, we prove that the support of the optimal input is bounded whenever the cost grows faster than a “cutoff” growth rate equal to the logarithm of the inverse of the noise probability density function evaluated at the input-output function. Furthermore, we prove a converse statement that says whenever the cost grows slower than the “cutoff” rate, the optimal input has necessarily an unbounded support. In addition, we show how the discreteness of the optimal input is guaranteed whenever the triplet satisfy some analyticity properties. We argue that a suitable cost function to be imposed on the channel input is one that grows similarly to the “cutoff” rate. Our results are valid for any cost function that is super-logarithmic. They summarize a large number of previous channel capacity results and give new ones for a wide range of communication channel models, such as Gaussian mixtures, generalized-Gaussians, and heavy-tailed noise models, that we state along with numerical computations.

Recent studies found that many channels are affected by additive noise that is impulsive in nature and is best explained by heavy-tailed symmetric alpha-stable distributions. Dealing with impulsive noise environments comes with an added complexity with respect to the standard Gaussian environment: the alpha-stable probability density functions do not possess closed-form expressions except in few special cases. Furthermore, they have an infinite second moment and the “nice” Hilbert space structure of the space of random variables having a finite second moment is lost along with its tools and methodologies. This is indeed the case in estimation theory, where classical tools to quantify the performance of an estimator are tightly related to the assumption of having finite variance variables. In alpha-stable environments, expressions, such as the mean square error and the Cramer-Rao bound, are hence problematic. In this paper, we tackle the parameter-estimation problem in the impulsive noise environments and develop novel tools that are tailored to the alpha-stable and heavy-tailed noise environments, tools that coincide with the standard ones adopted in the Gaussian setup, namely, a generalized “power” measure and a generalized Fisher information. We generalize known information inequalities commonly used in the Gaussian context: the de Bruijn identity, the Fisher information inequality, the isoperimetric inequality for entropies and the Cramer-Rao bound. Additionally, we derive upper bounds on the differential entropy of independent sums having a stable component. Intermediately, the new power measure is used to shed some light on the additive alpha-stable noise channel capacity in a setup that generalizes the linear average power constrained additive white Gaussian noise channel. Our theoretical findings are paralleled with numerical evaluations of various quantities and bounds using developed MATLAB packages.

We consider infinitesimal perturbations along symmetric stable variables and define a new information measure. We derive a generalized de Bruijn's identity, prove that the new measure satisfies a data processing inequality and a generalized Fisher information inequality which are used to establish an upper bound on the differential entropy of independent sums when one of the variables is stable.

Evaluating the channel capacity is one of many key problems in information theory. In this work, we derive rather-mild sufficient conditions under which the capacity of continuous channels is finite and achievable. These conditions are derived for generic, memoryless, and possibly nonlinear additive noise channels. The results are based on a novel sufficient condition that guarantees the convergence of differential entropies under point-wise convergence of probability density functions. Perhaps surprisingly, the finiteness of channel capacity holds for the majority of setups, including those where inputs and outputs have possibly infinite second-moments.

In this work, we consider a discrete-time memoryless communication channel where the input is subjected to an independent additive Cauchy noise. We find the input constraint under which a Cauchy input is capacity achieving. The constraint is logarithmic and depends on a scalar parameter k which we interpret as a power measure. We draw a parallelism between this setup and that of the Gaussian channel under the second moment constraint. In fact, a Cauchy input yields a Cauchy output over this channel and achieves a capacity value of “log(1 + SNR)”.

We consider an additive multiple-access channel model where all users are constrained to use identical codebooks, and where single-user decoding is performed at the receiver. We study the sum-capacity of the channel for an arbitrarily large, but finite, number of users. For a noiseless n-user channel, we construct a signaling scheme that achieves rates per user that are arbitrarily large, proving that the sum-capacity is infinite, whether the users are average and/or peak power limited or not. We show that this result still holds whenever an arbitrary discrete-noise component is added, provided there exists a positive lower bound on the separation between noise samples. Whenever the noise is of bounded support, the non power-constrained sum-capacity is also proven to be infinite. The results are valid for an asynchronous multiple-access channel with single-user decoding, as the appropriate channel model is identical to the one studied in this work.

Many communication channels are reasonably modeled to be impaired by additive noise. Recent studies suggest that many of these channels are affected by additive noise that is best explained by alpha-stable statistics. We study in this work such channel models and we characterize the capacity-achieving input distribution for those channels under fractional order moment constraints. We prove that the optimal input is necessarily discrete with a compact support for all such channels. Interestingly, if the second moment is viewed as a measure of power, even when the channel input is allowed to have infinite second moment, the optimal one is found to have finite power.

This paper studies classes of generic deterministic, discrete time, memoryless, and “nonlinear” additive white Gaussian noise (AWGN) channels. Subject to multiple types of constraints such as the even-moment and compact-support constraints or a mixture, the optimal input is proved to be discrete with finite number of mass points in the vast majority of the cases. Only under the even-moment constraint and for special cases that emulate the average power constrained linear channel, capacity is found to be achieved by an absolutely continuous input. The results are extended to channels where the distortion is generally piecewise nonlinear where the discrete nature of the optimal input is conserved. These results are reached through the development of methodology and tools that are based on standard decompositions in a Hilbert space with the Hermite polynomials as a basis, and it is showcased how these bases are natural candidates for general information-theoretic studies of the capacity of channels affected by AWGN. Intermediately, novel results regarding the output rate of decay of Gaussian channels are derived. Namely, the output probability distribution of any channel subjected to additive Gaussian noise decays necessarily “slower” than the Gaussian itself. Finally, numerical computations are provided for some sample cases, optimal inputs are determined, and capacity curves are drawn. These results put into question the accuracy of adopting the widely used expression 1(1+ SNR) for computing capacities of Gaussian deterministic channels.

In communication theory, one of the best understood and commonly adopted channel model is the average-power con strained linear AWGN channel, the capacity of which is given by the expression 1/2 log (1 + SNR). But what if the channel is not linear? How bad is it to adopt a linear model for a non-linear channel? In this paper, we answer these questions by considering generic deterministic memoryless non-linear channel models. We study these models under an even-moment, a peak or a mixture input constraint. We prove that for the majority of the studied channels, the capacity achieving input distributions are of a discrete nature with a finite number of mass points. The linear model under the average power constraint and other "equivalent channels " being the only exceptions. We establish our results using Hermite bases in a standard Hilbert space decomposition of some relevant information quantities in channels affected by AWGN. We present numerical results for two sample sub-linear channels. We determine their optimal inputs and plot their capacity curves showing that adopting a linear model for slightly non-linear channels will have serious implications on achievable rates and their achieving distributions.

Pilot symbol assisted modulation (PSAM) is a standard approach for transceiver design for time-varying channels, with channel estimates obtained from pilot symbols being employed for coherent demodulation of the data symbols. In this paper, we show that PSAM schemes can be improved by adapting the coded modulation strategy at the sender to the quality of the channel measurement at the receiver, without requiring any channel feedback from the receiver. We consider performance in terms of achievable rate for binary signaling schemes. The transmitter employs interleaved codes, with data symbols coded according to their distance from the nearest pilot symbols. Symbols far away from pilot symbols encounter poorer channel measurements at the receiver and are therefore coded with lower rate codes, while symbols close to pilot symbols benefit from recent channel measurements and are coded with higher rate codes. The performance benefits from this approach are quantified in the context of binary signaling over time-varying Rayleigh fading channels described by a Gauss-Markov model. The spacing of the pilot symbols is optimized to maximize the mutual information between input and output in this setting. Causal and noncausal channel estimators of varying complexity and delay are considered. It is shown that, by appropriate optimization for the spacing between consecutive pilot symbols, the adaptive coding techniques proposed can improve achievable rate, without any feedback from the receiver to the sender. Moreover, channel estimation based on the two closest pilot symbols is generally close to optimal.

We derive, for a binary antipodal input signal, the optimal uncoded regenerator function when the channels at the ingress and at the egress of the regenerator are degraded by AWGN. We show that the optimal function is a Lambert W function parametrized on the energies of the noises and the input. For comparison, we derive the performance of systems in which the regenerator uses a hard limiter or an amplifier.

In communication over time-varying Rayleigh fading channels, adaptive coded modulation for pilot symbol assisted modulation (PSAM) without feedback has been shown to yield significant benefits in terms of achievable rates (M. Medard et al., 2000). This technique adapts transmission rate at the sender to the quality of the channel estimate at the receiver but keeps the mean power constant throughout. In this paper, we show that this adaptive PSAM scheme can be further improved if power and rate are jointly adapted to the quality of the measurement at the receiver. We study the optimal power distribution scheme and find it to apply the following principle: more power is allocated to symbols corresponding to better estimates at the receiver, while maintaining the average energy constraint satisfied within a period. We find that this simple scheme is optimal and performs better than adapting to channel quality using schemes akin to 'water filling'. Our model is a Rayleigh fading channel where time-variance is described by a Gauss-Markov model (M. Medard, 2000). The transmitter periodically sends pilot tones to measure the channel at the receiver. We interleave different codes, while maintaining the power constant over a codebook, and the average power over codebooks satisfying the constraint. Performance is quantified in terms of achievable rates. Our scheme does not require any real time computation or adaptation at the transmitter, and so comes at no extra cost with respect to (M. Meadard et al., 2000). When considering causal and non causal estimation strategies at the receiver, considerable improvement was attained without any added complexity.

We analyze the error probability of peaky signaling on bandlimited multipath fading channels, the signaling strategy that achieves the capacity of such channels in the limit of infinite bandwidth under an average power constraint. We first derive an upper bound for general fading, then specialize to the case of Rayleigh fading, where we obtain upper and lower bounds that are exponentially tight and, therefore, yield the reliability function. These bounds constitute a strong coding theorem for the channel, as they not only delimit the range of achievable rates, but also give us a relationship among the error probability, data rate, bandwidth, peakiness, and fading parameters, such as the coherence time. They can be used to compare peaky signaling systems to other large bandwidth systems over fading channels, such as ultra-wideband radio and wideband code-division multiple access. We find that the error probability decreases slowly with the bandwidth W; under Rayleigh fading, the error probability varies roughly as W/sup -/spl alpha//, where /spl alpha/>0. With parameters typical of indoor wireless situations, we study the behavior of the upper and lower bounds on the error probability and the reliability function numerically.

We consider transmission over a discrete-time Rayleigh fading channel, in which successive symbols face independent fading, and where neither the transmitter nor the receiver has channel state information. Subject to an average power constraint, we study the capacity-achieving distribution of this channel and prove it to be discrete with a finite number of mass points, one of them located at the origin. We numerically compute the capacity and the corresponding optimal distribution as a function of the signal-to-noise ratio (SNR). The behavior of the channel at low SNR is studied and finally a comparison is drawn with the ideal additive white Gaussian noise channel.