This paper introduces scaled Bregman distances of probability distributions which admit nonuniform contributions of observed events. They are introduced in a general form covering not only the distances of discrete and continuous stochastic observations, but also the distances of random processes and signals. It is shown that the scaled Bregman distances extend not only the classical ones studied in the previous literature, but also the information divergence and the related wider class of convex divergences of probability measures. An information-processing theorem is established too, but only in the sense of invariance w.r.t. statistically sufficient transformations and not in the sense of universal monotonicity. Pathological situations where coding can increase the classical Bregman distance are illustrated by a concrete example. In addition to the classical areas of application of the Bregman distances and convex divergences such as recognition, classification, learning, and evaluation of proximity of various features and signals, the paper mentions a new application in 3-D exploratory data analysis. Explicit expressions for the scaled Bregman distances are obtained in general exponential families, with concrete applications in the binomial, Poisson, and Rayleigh families, and in the families of exponential processes such as the Poisson and diffusion processes including the classical examples of the Wiener process and geometric Brownian motion.

We compare twof-divergences and prove that their joint range is the convex hull of the joint range for distributions supported on only two points. The proofs use various methods from topology and convex analysis.

Many of the divergence measures used in statistics are of the f-divergence type. Often one is interested inequalities for one f-divergence in terms of another f-divergence. Such inequalities are for instance needed in order to calculate the relative efficiency of two f-divergences when used for testing goodness of fit but there are many other applications. In this paper we shall study the more general problem of determining the joint range of any pair of f-divergences. The results are useful in determining general conditions under which information divergence is a more efficient statistic for testing goodness of fit than another f-divergence, but will not be discussed in this short paper.

Recently it was shown that Shannon entropy is more Bahadur efficient than any Rényi entropy of order α ≫ 1. In this paper we shall show that relative Bahadur efficiency between any two Rényi entropies of orders α ∈ ]0; 1] is 1 when the relative Bahadur efficiency is defined according to [1]. Despite the fact that the relative Bahadur efficiency is 1 it is shown that in a certain sense Shannon entropy is more efficient than Rényi entropy for α ∈ ]0; 1]. This indicates that the definition of relative efficiency given in [1] does not fully capture the notion of efficiency.

This paper compares the power divergence statistics of orders with the information divergence statistic in the problem of testing the uniformity of a distribution. In this problem, the information divergence statistic is equivalent to the entropy statistic. Extending some previously established results about information diagrams, it is proved that the information divergence statistic in this problem is more efficient in the Bahadur sense than any power divergence statistic of order . This means that the entropy provides in this sense the most efficient way of characterizing the uniformity of a distribution.

This paper compares the power divergence statistics of orders alpha > 1 with the information divergence statistic in the problem of testing the uniformity of a distribution. In this problem the information divergence statistic is equivalent to the entropy statistic. Extending some previously established results about information diagrams, it is proved that in this problem the information divergence statistic is more efficient in the Bahadur sense than any power divergence statistic of order alpha > 1. This means that the entropy provides in a certain sense the most efficient way of characterizing the uniformity of a distribution.

We consider quantizations of observations represented by finite partitions of observation spaces. Partitions usually decrease the sensitivity of observations to their probability distributions. A sequence of quantizations is considered to be asymptotically sufficient for a statistical problem if the loss of sensitivity is asymptotically negligible. The sensitivity is measured by f-divergences of distributions or the closely related f-informations including the classical Shannon information. It is demonstrated that in some cases the maximization of f-divergences means the same as minimization of distortion of observations in the classical sense considered in mathematical statistics and information theory. The main result of the correspondence is a general sufficient condition for the asymptotic sufficiency of quantizations. Selected applications of this condition are studied leading to new simple criteria of asymptotic optimality for quantizations of vector-valued observations and observations on general Poisson processes

The paper deals with the f-divergences of Csiszar generalizing the discrimination information of Kullback, the total variation distance, the Hellinger divergence, and the Pearson divergence. All basic properties of f-divergences including relations to the decision errors are proved in a new manner replacing the classical Jensen inequality by a new generalized Taylor expansion of convex functions. Some new properties are proved too, e.g., relations to the statistical sufficiency and deficiency. The generalized Taylor expansion also shows very easily that all f-divergences are average statistical informations (differences between prior and posterior Bayes errors) mutually differing only in the weights imposed on various prior distributions. The statistical information introduced by De Groot and the classical information of Shannon are shown to be extremal cases corresponding to alpha=0 and alpha=1 in the class of the so-called Arimoto alpha-informations introduced in this paper for 0<alpha<1 by means of the Arimoto alpha-entropies. Some new examples of f-divergences are introduced as well, namely, the Shannon divergences and the Arimoto alpha-divergences leading for alphauarr1 to the Shannon divergences. Square roots of all these divergences are shown to be metrics satisfying the triangle inequality. The last section introduces statistical tests and estimators based on the minimal f-divergence with the empirical distribution achieved in the families of hypothetic distributions. For the Kullback divergence this leads to the classical likelihood ratio test and estimator

This correspondence considers the class P of discrete bivariate distributions p with marginals a(p), b(p) and the subclass Pa,b /spl sub/ P of distributions with given marginals a and b. For an arbitrary q /spl isin/ P we study the distributions p/spl tilde/ /spl isin/ P/sub a,b/ minimizing on P/sub a,b/ various /spl phi/-divergences D/sub /spl phi//(p, q) for convex /spl phi/ with finite /spl phi/(0).

Mutual asymptotic equivalence is established within three classes of information-theoretic divergences of discrete probability distributions, namely, f-divergences of Csiszar, f-divergences of Bregman, and f-divergences of Burbea-Rao. These equivalences are used to find asymptotic distributions of the corresponding divergence statistics for testing the goodness of fit when the hypothetic distribution is uniform. All results are based on standard expansion techniques and on a new relation between the Bregman and Burbea-Rao divergences formulated in Lemma 2 of the paper.

By the well-known information processing theorem, quantizations of observations reduce values of convex information functionals such as the information divergence, Fisher information, or Shannon information. This paper deals with the convergence of the reduced values of these functionals to their original unreduced values for various sequences P/sub n/ of partitions of the observation space. There is extensive literature dealing with this convergence when the partitions are nested in the sense P/sub n/ /spl sub/ P/sub n+1/ for all n. A systematic study for nonnested partitions P/sub n/, often considered in the literature, seems to be missing. This paper tries to partially fill this gap. It proves the convergence for the most common types of partitions. The results are formulated for generalized information divergences (Csiszar (1963, 1967, 1973) divergences), generalized Fisher information, and generalized Shannon information. Their applicability is illustrated on the Barron (1992) estimators of probability distributions.

We investigate a nonparametric estimator of the probability density introduced by Barron (1988, 1989). Earlier papers established its consistency in a strong sense, e.g., in the expected information divergence or expected chi-square divergence. This paper pays main attention to the expected chi-square divergence criterion. We give a new motivation of the Barron estimator by showing that a maximum-likelihood estimator (MLE) of a density from a family important in practice is consistent in expected information divergence but not in expected chi-square divergence. We also present new and practically applicable conditions of consistency in the expected chi-square divergence. Main attention is paid to optimization (in the sense of the mentioned criterion) of the two objects specifying the Barron estimator: the dominating probability density and the decomposition of the observation space into finitely many bins. Both problems are explicitly solved under certain regularity assumptions about the estimated density. A simulation study illustrates the results in exponential, Rayleigh, and Weibull families.

Analytical formulas for the entropy and the mutual information of multivariate continuous probability distributions are presented.

We demonstrate that it is possible to approximate the mutual information arbitrarily closely in probability by calculating the relative frequencies on appropriate partitions and achieving conditional independence on the rectangles of which the partitions are made. Empirical results, including a comparison with maximum-likelihood estimators, are presented.

By extending the information-theoretic arguments of previous papers dealing with the Barron-type density estimates, and their consistency in information divergence and chi-square divergence, the problem of consistency in Csiszar's /spl phi/-divergence is motivated for general convex functions /spl phi/. The problem of consistency in /spl phi/-divergence is solved for all /spl phi/ with /spl phi/(0)

The distances of discrete probability distributions are considered. Necessary and sufficient conditions for validity of the data processing theorem of information theory are established. These conditions are applied to the Burbea-Rao (1982) divergences and Bregman (1967) distances.

We derive an inequality for the minimum error probability which shows that in many situations this quantity converges to its supremum as the number of possible values of the input signal increases. The asymptotic behavior of this probability is related to the behavior of the equivocation. Implications for asymptotic rates of certain codes and for perfect secrecy of cryptosystems are mentioned.

Uncertainty is defined in a new manner, as a function of discrete probability distributions satisfying a simple and intuitively appealing weak monotonicity condition. It is shown that every uncertainty is Schur-concave and conversely, every Schur-concave function of distributions is an uncertainty. General properties of uncertainties are systematically studied. Many characteristics of distributions introduced previously in statistical physics, mathematical statistics, econometrics and information theory are shown to be particular examples of uncertainties. New examples are introduced, and traditional as well as some new methods for obtaining uncertainties are discussed. The information defined by decrease of uncertainty resulting from an observation is investigated and related to previous concepts of information. Further, statistical inference about uncertainties is investigated, based on independent observations of system states. In particular, asymptotic distributions of maximum likelihood estimates of uncertainties and uncertainty-related functions are derived, and asymptotically /spl alpha/-level Neyman-Pearson tests of hypotheses about these system characteristics are presented.

In analogy with the definition of Shannon information, M.H. De Groot (1962) defined statistical information as the difference between prior and posterior risk of a statistical decision problem. Relations are studied between the statistical information and the discrimination functions of information theory known as f-divergences. Using previous results, it is shown that every f-divergence I/sub f/(P,Q) is an average statistical information or decision problem with dichotomic parameter, 0-1 loss function, and corresponding observation distributions P and Q. The average is taken over a distribution on the parameter's prior probability. This distribution is uniquely determined by the function f. The main result is that every f-divergence is statistical information in some properly chosen statistical decision problem, and conversely, that every piece of statistical information is an f-divergence. This provides a new representation of discrimination functions figuring in signal detection, data compression, coding pattern classification, cluster analysis, etc.>

Minimization problems involving phi -entropy functionals (a generalization of Boltzmann-Shannon entropy) are studied over a given set A and a sequence of sets A/sub n/ and the properties of their optimal solutions x/sub phi /, x/sub n/. Under certain conditions on the objective functional and the sets A and A/sub n/, it is proven that as n increases to infinity, the optimal solution x/sub n/ converges in L/sub 1/ norm to the test phi -entropy estimate x/sub phi /.>

Composite hypotheses concerning stochastic system entropy, are considered. Asymptotically alpha -level tests for statistical testing of these hypotheses against their alternatives are derived. For the most interesting cases (extremely small and extremely large entropies) the critical regions are explicitely calculated. It is shown that these tests can be used for testing the probability of error of the Bayes predictor of the system states. The method is illustrated by testing the system stability and by testing optimality of a medical diagnosis expert system. The main attention is focused on Shannon entropies, but alternative entropies are also considered.>