A well-known technique in assessing probabilities of rare events (used, e.g., in the sphere-packing bound), is that of finding a reference measure under which the event of interest has probability of order one and estimating the probability in question using the Kullback-Leibler divergence (KLD). A recent method has been proposed [2], that can be viewed as an extension of this idea in which the probability under the reference measure may itself be decaying exponentially, and the Rényi divergence (RD) is used instead. We demonstrate the usefulness of this approach in various information-theoretic settings. For channel coding, we provide a method for obtaining matched, mismatched and robust error exponent bounds, as well as new results in a variety of particular channel models. Other applications we address include rate-distortion coding and the problem of guessing.

A well-known technique in estimating the probabilities of rare events in general and in information theory in particular (used, for example, in the sphere-packing bound) is that of finding a reference probability measure under which the event of interest has the probability of order one and estimating the probability in question by means of the Kullback-Leibler divergence. A method has recently been proposed in [2] that can be viewed as an extension of this idea in which the probability under the reference measure may itself be decaying exponentially, and the Rényi divergence is used instead. The purpose of this paper is to demonstrate the usefulness of this approach in various information-theoretic settings. For the problem of channel coding, we provide a general methodology for obtaining matched, mismatched, and robust error exponent bounds, as well as new results in a variety of particular channel models. Other applications we address include rate-distortion coding and the problem of guessing.

Let$X$be a nonnegative random variable and let the conditional distribution of a random variable$Y$, given$X$, be Poisson$(\gamma\cdot X)$, for a parameter$\gamma\geq 0$. We identify a natural loss function such that: 1) the derivative of the mutual information between$X$and$Y$with respect to$\gamma$is equal to the minimum mean loss in estimating$X$based on$Y$, regardless of the distribution of$X$; 2) when$X\sim P$is estimated based on$Y$by a mismatched estimator that would have minimized the expected loss had$X\sim Q$, the integral over all values of$\gamma$of the excess mean loss is equal to the relative entropy between$P$and$Q$. For a continuous time setting where$X$is a nonnegative stochastic process and the conditional law of$Y$, given$X$, is that of a non-homogeneous Poisson process with intensity function$\gamma\cdot X$, under the same loss function: 1) the minimum mean loss in causal filtering when$\gamma=\gamma_{0}$is equal to the expected value of the minimum mean loss in noncausal filtering (smoothing) achieved with a channel whose parameter$\gamma$is uniformly distributed between 0 and$\gamma_{0}$. Bridging the two quantities is the mutual information between$X$and$Y$; 2) this relationship between the mean losses in causal and noncausal filtering holds also in the case where the filters employed are mismatched, i.e., optimized assuming a law on$X$which is not the true one. Bridging the two quantities in this case is the sum of the mutual information and the relative entropy between the true and the mismatched distribution of$Y$. Thus, relative entropy quantifies the excess estimation loss due to mismatch in this setting. These results are parallel to those recently found for the Gaussian channel: the I-MMSE relationship of Guo, the relative entropy and mismatched estimation relationship of Verdú, and the relationship between causal and noncasual mismatched estimation of Weissman.

Let X be a non-negative random variable and let the conditional distribution of a random variable Y, given X, be Poisson(γ·X), for a parameter γ ≥ 0. We identify a natural loss function such that: 1. The derivative of the mutual information between X and Y with respect to γ is equal to the minimum mean loss in estimating X based on Y, regardless of the distribution of X. 1 When X ~ P is estimated based on Y by a mismatched estimator that would have minimized the expected loss had X ~ Q, the integral over all values of γ of the excess mean loss is equal to the relative entropy between P and Q. For a continuous time setting where XT= {Xt, 0 ≤ t ≤ T} is a non-negative stochastic process and the conditional law of YT= {Yt, 0 ≤ t ≤ T}, given XT, is that of a non-homogeneous Poisson process with intensity function γ · XT, under the same loss function: 1. The minimum mean loss in causal filtering when γ = γ0is equal to the expected value of the minimum mean loss in non-causal filtering (smoothing) achieved with a channel whose parameter γ is uniformly distributed between 0 and γ0. Bridging the two quantities is the mutual information between XTand YT. 2. This relationship between the mean losses in causal and non-causal filtering holds also in the case where the filters employed are mismatched, i.e., optimized assuming a law on XTwhich is not the true one. Bridging the two quantities in this case is the sum of the mutual information and the relative entropy between the true and the mismatched distribution of YT. Thus, relative entropy quantifies the excess estimation loss due to mismatch in this setting. These results parallel those recently found for the Gaussian channel: the I-MMSE relationship of Guo Shamai and Verdii, the relative entropy and mismatched estimation relationship of Verdii, and the relationship between causal and non-casual mismatched estimation of Weissman.