We consider the estimation problem for jointly stable random variables. Under two specific dependency models: a linear transformation of two independent stable variables and a sub-Gaussian symmetric $α$-stable (S$α$S) vector, we show that the conditional mean estimator is linear in both cases. Moreover, we find dispersion optimal linear estimators. Interestingly, for the sub-Gaussian (S$α$S) vector, both estimators are identical generalizing the well-known Gaussian result of the conditional mean being the best linear minimum-mean square estimator.

We study the rate-distortion problem for both scalar and vector memoryless heavy-tailed α-stable sources (0 < α < 2). Using a recently defined notion of “strength” as a power measure, we derive the rate-distortion function for α-stable sources subject to a constraint on the strength of the error and show it to be logarithmic in the strength-to-distortion ratio. We show how our framework paves the way for finding optimal quantizers for α- stable sources and other general heavy-tailed ones. In addition, we study high-rate scalar quantizers and show that uniform ones are asymptotically optimal under the error-strength distortion measure. We compare uniform Gaussian and Cauchy quantizers and show that more representation points for the Cauchy source are required to guarantee the same quantization quality. Our findings generalize the well-known results of rate-distortion and quantization of Gaussian sources (α = 2) under a quadratic distortion measure.

A generalized notion of “relative entropy typicality” is introduced. The new definition accommodates non-i.i.d. scenarios and is parameterized by two families of parameters,$\left\{\delta^{[n]}\right\}_{n}$and$\left\{\epsilon^{[n]}\right\}_{n}$: the choice of$\left\{\delta^{[n]}\right\}_{n}$(the allowed “deviation” in the relative entropy typical set) can be optimized as a function of$\left\{\epsilon^{[n]}\right\}_{n}$, where ($1-\epsilon^{[n]}$) is the desired probability of the set. This generalized definition is shown to yield an extension of the Chernoff-Stein lemma, which characterizes the rate of decay of the type II error in hypothesis testing (under a fixed type I error constraint) as the KL divergence. In particular, the generalization accommodates both discrete and continuous random variables, non-i.i.d. settings, in addition to cases where the KL divergence grows non-linearly. Several example applications are discussed, including testing two correlated Gaussian distributions.

We propose a framework for the rate-distortion analysis that extends the Gaussian MSE setup for stable sources. Specifically, we study the rate-distortion function of$\alpha$-stable sources when the distortion constraint is specified using the recently defined notion of strength of the error variable. We also “generalize” the notion of$d$-tilted information under the new setup and use it to determine second-order terms for both fixed and variable-length codes.

The per-sample zero-dispersion channel model of the optical fiber is considered. It is shown that capacity is uniquely achieved by an input probability distribution that has continuous uniform phase and discrete amplitude that takes on finitely many values. This result holds when the channel is subject to general input cost constraints, that include a peak amplitude constraint and a joint average and peak amplitude constraint.

The per-sample zero-dispersion channel model of the optical fiber is considered. It is shown that capacity is uniquely achieved by an input probability distribution that has continuous uniform phase and discrete amplitude that takes on finitely many values. This result holds when the channel is subject to general input cost constraints, that include a peak amplitude constraint and a joint average and peak amplitude constraint.

We study the classical problem of characterizing the channel capacity and its achieving distribution in a generic fashion. We derive a simple relation between three parameters: the input-output function, the input cost function, and the noise probability density function, one which dictates the type of the optimal input. In layman terms, we prove that the support of the optimal input is bounded whenever the cost grows faster than a “cutoff” growth rate equal to the logarithm of the inverse of the noise probability density function evaluated at the input-output function. Furthermore, we prove a converse statement that says whenever the cost grows slower than the “cutoff” rate, the optimal input has necessarily an unbounded support. In addition, we show how the discreteness of the optimal input is guaranteed whenever the triplet satisfy some analyticity properties. We argue that a suitable cost function to be imposed on the channel input is one that grows similarly to the “cutoff” rate. Our results are valid for any cost function that is super-logarithmic. They summarize a large number of previous channel capacity results and give new ones for a wide range of communication channel models, such as Gaussian mixtures, generalized-Gaussians, and heavy-tailed noise models, that we state along with numerical computations.

Recent studies found that many channels are affected by additive noise that is impulsive in nature and is best explained by heavy-tailed symmetric alpha-stable distributions. Dealing with impulsive noise environments comes with an added complexity with respect to the standard Gaussian environment: the alpha-stable probability density functions do not possess closed-form expressions except in few special cases. Furthermore, they have an infinite second moment and the “nice” Hilbert space structure of the space of random variables having a finite second moment is lost along with its tools and methodologies. This is indeed the case in estimation theory, where classical tools to quantify the performance of an estimator are tightly related to the assumption of having finite variance variables. In alpha-stable environments, expressions, such as the mean square error and the Cramer-Rao bound, are hence problematic. In this paper, we tackle the parameter-estimation problem in the impulsive noise environments and develop novel tools that are tailored to the alpha-stable and heavy-tailed noise environments, tools that coincide with the standard ones adopted in the Gaussian setup, namely, a generalized “power” measure and a generalized Fisher information. We generalize known information inequalities commonly used in the Gaussian context: the de Bruijn identity, the Fisher information inequality, the isoperimetric inequality for entropies and the Cramer-Rao bound. Additionally, we derive upper bounds on the differential entropy of independent sums having a stable component. Intermediately, the new power measure is used to shed some light on the additive alpha-stable noise channel capacity in a setup that generalizes the linear average power constrained additive white Gaussian noise channel. Our theoretical findings are paralleled with numerical evaluations of various quantities and bounds using developed MATLAB packages.

We consider infinitesimal perturbations along symmetric stable variables and define a new information measure. We derive a generalized de Bruijn's identity, prove that the new measure satisfies a data processing inequality and a generalized Fisher information inequality which are used to establish an upper bound on the differential entropy of independent sums when one of the variables is stable.

Evaluating the channel capacity is one of many key problems in information theory. In this work, we derive rather-mild sufficient conditions under which the capacity of continuous channels is finite and achievable. These conditions are derived for generic, memoryless, and possibly nonlinear additive noise channels. The results are based on a novel sufficient condition that guarantees the convergence of differential entropies under point-wise convergence of probability density functions. Perhaps surprisingly, the finiteness of channel capacity holds for the majority of setups, including those where inputs and outputs have possibly infinite second-moments.

In this work, we consider a discrete-time memoryless communication channel where the input is subjected to an independent additive Cauchy noise. We find the input constraint under which a Cauchy input is capacity achieving. The constraint is logarithmic and depends on a scalar parameter k which we interpret as a power measure. We draw a parallelism between this setup and that of the Gaussian channel under the second moment constraint. In fact, a Cauchy input yields a Cauchy output over this channel and achieves a capacity value of “log(1 + SNR)”.

Many communication channels are reasonably modeled to be impaired by additive noise. Recent studies suggest that many of these channels are affected by additive noise that is best explained by alpha-stable statistics. We study in this work such channel models and we characterize the capacity-achieving input distribution for those channels under fractional order moment constraints. We prove that the optimal input is necessarily discrete with a compact support for all such channels. Interestingly, if the second moment is viewed as a measure of power, even when the channel input is allowed to have infinite second moment, the optimal one is found to have finite power.

This paper studies classes of generic deterministic, discrete time, memoryless, and “nonlinear” additive white Gaussian noise (AWGN) channels. Subject to multiple types of constraints such as the even-moment and compact-support constraints or a mixture, the optimal input is proved to be discrete with finite number of mass points in the vast majority of the cases. Only under the even-moment constraint and for special cases that emulate the average power constrained linear channel, capacity is found to be achieved by an absolutely continuous input. The results are extended to channels where the distortion is generally piecewise nonlinear where the discrete nature of the optimal input is conserved. These results are reached through the development of methodology and tools that are based on standard decompositions in a Hilbert space with the Hermite polynomials as a basis, and it is showcased how these bases are natural candidates for general information-theoretic studies of the capacity of channels affected by AWGN. Intermediately, novel results regarding the output rate of decay of Gaussian channels are derived. Namely, the output probability distribution of any channel subjected to additive Gaussian noise decays necessarily “slower” than the Gaussian itself. Finally, numerical computations are provided for some sample cases, optimal inputs are determined, and capacity curves are drawn. These results put into question the accuracy of adopting the widely used expression 1(1+ SNR) for computing capacities of Gaussian deterministic channels.

In communication theory, one of the best understood and commonly adopted channel model is the average-power con strained linear AWGN channel, the capacity of which is given by the expression 1/2 log (1 + SNR). But what if the channel is not linear? How bad is it to adopt a linear model for a non-linear channel? In this paper, we answer these questions by considering generic deterministic memoryless non-linear channel models. We study these models under an even-moment, a peak or a mixture input constraint. We prove that for the majority of the studied channels, the capacity achieving input distributions are of a discrete nature with a finite number of mass points. The linear model under the average power constraint and other "equivalent channels " being the only exceptions. We establish our results using Hermite bases in a standard Hilbert space decomposition of some relevant information quantities in channels affected by AWGN. We present numerical results for two sample sub-linear channels. We determine their optimal inputs and plot their capacity curves showing that adopting a linear model for slightly non-linear channels will have serious implications on achievable rates and their achieving distributions.