Minimization problems with respect to a one-parameter family of generalized relative entropies are studied. These relative entropies, which we term relative α-entropies (denoted Iα), arise as redundancies under mismatched compression when cumulants of compressed lengths are considered instead of expected compressed lengths. These parametric relative entropies are a generalization of the usual relative entropy (Kullback-Leibler divergence). Just like relative entropy, these relative α-entropies behave like squared Euclidean distance and satisfy the Pythagorean property. Minimizers of these relative α-entropies on closed and convex sets are shown to exist. Such minimizations generalize the maximum Rényi or Tsallis entropy principle. The minimizing probability distribution (termed forward Iα-projection) for a linear family is shown to obey a power-law. Other results in connection with statistical inference, namely subspace transitivity and iterated projections, are also established. In a companion paper, a related minimization problem of interest in robust statistics that leads to a reverse Iα-projection is studied.

In part I of this two-part work, certain minimization problems based on a parametric family of relative entropies (denoted ℐα) were studied. Such minimizers were called forward ℐα-projections. Here, a complementary class of minimization problems leading to the so-called reverse ℐα-projections are studied. Reverse ℐα-projections, particularly on log-convex or power-law families, are of interest in robust estimation problems (α > 1) and in constrained compression settings (αα-projection into a forward ℐα-projection. The transformed problem is a simpler quasi-convex minimization subject to linear constraints.

This paper extends some geometric properties of a one-parameter family of relative entropies. These arise as redundancies when cumulants of compressed lengths are considered instead of expected compressed lengths. These parametric relative entropies are a generalization of the Kullback-Leibler divergence. They satisfy the Pythagorean property and behave like squared distances. This property, which was known for finite alphabet spaces, is now extended for general measure spaces. Existence of projections onto convex and certain closed sets is also established. Our results may have applications in the Rényi entropy maximization rule of statistical physics.