An urn containing colored balls is sampled sequentially without replacement. New lower and upper bounds on the conditional and unconditional mutual information, and multi-information are presented. They estimate dependence between drawings in terms of the colored ball configuration. Asymptotics are worked out when the number of balls increases and the proportion of the balls of each color stabilizes. Inequalities by Stam and by Diaconis and Freedman are compared and improved. Distances between the sampling with and without replacement, and between the multinomial and multivariate hypergeometric distributions are discussed.

Integral functionals based on convex normal integrands are minimized over convex constraint sets. Generalized minimizers exist under a boundedness condition. Sequences of the minimization problems are studied when the constraint sets are nested. The corresponding sequences of generalized minimizers are related to the minimization over limit convex sets. Martingale theorems and moment problems are discussed.

The entropy region is constructed from vectors of random variables by collecting Shannon entropies of all subvectors. Its shape is studied here by means of polymatroidal constructions, notably by convolution. The closure of the region is decomposed into the direct sum of tight and modular parts, reducing the study to the tight part. The relative interior of the reduction belongs to the entropy region. Behavior of the decomposition under self-adhesivity is clarified. Results are specialized and extended to the region constructed from four tuples of random variables. This and computer experiments help to visualize approximations of a symmetrized part of the entropy region. The four-atom conjecture on the minimal Ingleton score is refuted.

Integral functionals based on convex normal integrands are minimized subject to finitely many moment constraints. The integrands are assumed to be strictly convex but not autonomous or differentiable. The effective domain of the value function is described by a modification of the concept of convex core. The minimization is viewed as a primal problem and studied together with a dual one in the framework of convex duality. Main results assume a dual constraint qualification but dispense with the primal constraint qualification. Minimizers and generalized minimizers are explicitly described whenever the primal value is finite. Existence of a generalized dual solution is established whenever the dual value is finite. A generalized Pythagorean identity is presented using Bregman distance and a correction term. Results are applied to minimization of Bregman distances.

The problem to maximize the information divergence from an exponential family is compared to the maximization of an entropy-like quantity over the boundary of a polytope. First-order conditions on directional derivatives define critical sets for the two problems. The bijection between the sets of global maximizers in the two problems found earlier is extended here to bijections between the sets of local maximizers and the critical sets. This is based on new inequalities relating the maximized quantities and a reformulation of the first order criticality conditions for the second problem.

The ergodic sequences consisting of vectors (ξn, ηn), n ≥ 1, over a finite alphabet A×B are colored with ⌊enα⌋ colors for Anand⌊enβ⌋ colors for Bn. Generic behavior of the colorings in terms of probabilities of monochromatic rectangles intersected with typical sets is examined. When n increases a big majority of pairs of colorings produces rectangles whose probabilities are bounded uniformly from above. Limiting rates of bounds are worked out in all regimes of the rates α and β of colorings. As a consequence, generic behavior of the colorings in terms of Shannon entropies of the partitions into rectangles is described.

The problem of minimizing convex integral functionals subject to moment-like constraints is treated in a general setting when the underlying convex function may be multivariate, perhaps not strictly convex or differentiable. The results are applied to the minimization of f-divergences simultaneously in both variables.

Maximization of the information divergence from any hierarchical log-linear model is studied. A new upper bound on the maximum is presented and its tightness analyzed. For the models given by the bases of a matroid, the latter is related to matroid representations by partitions or, equivalently, to ideal secret-sharing schemes. A new link between the divergence maximization, the maximum-likelihood principle, and secret sharing is established.

Minimization problems for entropy-like integrals and Bregman distances subject to a finite number of moment constraints are addressed in a general setting. Analogues of the authors’ previous results on information projections to families determined by linear constraints, and reverse information projections to exponential families, are established. No constraint qualification is assumed.

When finite, Shannon entropies of all sub vectors of a random vector are considered for the coordinates of an entropic point in Euclidean space. A linear combination of the coordinates gives rise to an unconstrained information inequality if it is nonnegative for all entropic points. With at least four variables no finite set of linear combinations generates all such inequalities. This is proved by constructing explicitly an infinite sequence of new linear information inequalities and a curve in a special geometric position to the halfspaces defined by the inequalities. The inequalities are constructed recurrently by adhesive pasting of restrictions of polymatroids and the curve ranges in the closure of a set of the entropic points.

For a standard full exponential family on Rd, or its canonically convex subfamily, the generalized maximum likelihood estimator is an extension of the mapping that assigns to the mean a ϵ Rd of a sample for which a maximizer v* of the corresponding likelihood function exists, the member of the family parameterized by v*. This extension assigns to each a ϵ Rd with the likelihood function bounded above, a member of the closure of the family in variation distance. Its detailed description, complete characterization of domain and range, and additional results are presented, in a general setting. In addition to basic convex analysis tools, the authors' prior results on convex cores of measures and closures of exponential families are used.

A new information inequality of non-Shannon type is proved for three discrete random variables under conditional independence constraints, using the framework of entropy functions and polymatroids. Tightness of the inequality is described via quasi-groups

The closure in variation distance of any subfamily with a convex set of canonical parameters of an exponential family has been described previously, in terms of the concept of extension of the full exponential family. This contribution describes the closure in reversed information divergence (rI-closure), via variation closures of auxiliary subfamilies. Also, variation convergence and rI-convergence in the extension are characterized.

A closure of an exponential family in reversed information divergence may be neither reverse information-closed nor log-convex.

The goal of this paper is to complete results available about I-projections, reverse I-projections, and their generalized versions, with focus on linear and exponential families. Pythagorean-like identities and inequalities are revisited and generalized, and generalized maximum-likelihood (ML) estimates for exponential families are introduced. The main tool is a new concept of extension of exponential families, based on our earlier results on convex cores of measures.

A model of finite Radon transforms composed of Radon projections is presented. The model generalizes to finite group projections in the classical Radon transform theory. The Radon projector averages a function on a group over cosets of a subgroup. Reconstruction formulae that were formally similar to the convolved backprojection ones are derived, and an iterative reconstruction technique is found to converge after a finite number of steps. Applying these results to the group Z/sub 2//sup P/, new computationally favorable image representations have been obtained. A numerical study of the transform coding aspects is attached.>