Tsallis and Rényi entropies, which are monotone transformations of each other, are deformations of the celebrated Shannon entropy. Maximization of these deformed entropies, under suitable constraints, leads to the$q$-exponential family which has applications in non-extensive statistical physics, information theory and statistics. In previous information-geometric studies, the$q$-exponential family was analyzed using classical convex duality and Bregman divergence. In this paper, we show that a generalized$\lambda $-duality, where$\lambda = 1 - q$is to be interpreted as the constant information-geometric curvature, leads to a generalized exponential family which is essentially equivalent to the$q$-exponential family and has deep connections with Rényi entropy and optimal transport. Using this generalized convex duality and its associated logarithmic divergence, we show that our$\lambda $-exponential family satisfies properties that parallel and generalize those of the exponential family. Under our framework, the Rényi entropy and divergence arise naturally, and we give a new proof of the Tsallis/Rényi entropy maximizing property of the$q$-exponential family. We also introduce a$\lambda $-mixture family which may be regarded as the dual of the$\lambda $-exponential family, and connect it with other mixture-type families. Finally, we discuss a duality between the$\lambda $-exponential family and the$\lambda $-logarithmic divergence, and study its statistical consequences.

Reproducing kernel Hilbert space (RKHS) methods have become powerful tools in machine learning. However, their kernels, which measure similarity of inputs, are required to be symmetric, constraining certain applications in practice. Furthermore, the celebrated representer theorem only applies to regularizers induced by the norm of an RKHS. To remove these limitations, we introduce the notion of reproducing kernel Banach spaces (RKBS) for pairs of reflexive Banach spaces of functions by making use of semi-inner-products and the duality mapping. As applications, we develop the framework of RKBS standard learning schemes including minimal norm interpolation, regularization network, and support vector machines. In particular, existence, uniqueness and representer theorems are established.

We explicitly analyze the trajectories of learning near singularities in hierarchical networks, such as multilayer perceptrons and radial basis function networks, which include permutation symmetry of hidden nodes, and show their general properties. Such symmetry induces singularities in their parameter space, where the Fisher information matrix degenerates and odd learning behaviors, especially the existence of plateaus in gradient descent learning, arise due to the geometric structure of singularity. We plot dynamic vector fields to demonstrate the universal trajectories of learning near singularities. The singularity induces two types of plateaus, the on-singularity plateau and the near-singularity plateau, depending on the stability of the singularity and the initial parameters of learning. The results presented in this letter are universally applicable to a wide class of hierarchical models. Detailed stability analysis of the dynamics of learning in radial basis function networks and multilayer perceptrons will be presented in separate work.

From a smooth, strictly convex function phi: Rn --> R, a parametric family of divergence function Dphi(alpha) may be introduced: [ equation: see text] for x, y epsilon int dom (Phi) subset Rn, and for alpha in R, with Dphi(+/-1) defined through taking the limit of alpha. Each member is shown to induce an alpha-independent Riemannian metric, as well as a pair of dual alpha-connections, which are generally nonflat, except for alpha = +/-1. In the latter case, Dphi(+/-1) reduces to the (nonparametric) Bregman divergence, which is representable using phi and its convex conjugate phi* and becomes the canonical divergence for dually flat spaces (Amari, 1982, 1985; Amari & Nagaoka, 2000). This formulation based on convex analysis naturally extends the informationgeometric interpretation of divergence functions (Eguchi, 1983) to allow the distinction between two different kinds of duality: referential duality (alpha <--> -alpha) and representational duality (phi <--> phi*). When applied to (not necessarily normalized) probability densities, the concept of conjugated representations of densities is introduced, so that +/-alpha-connections defined on probability densities embody both referential and representational duality and are hence themselves bidual. When restricted to a finite-dimensional affine submanifold, the natural parameters of a certain representation of densities and the expectation parameters under its conjugate representation form biorthogonal coordinates. The alpha representation (indexed by beta now, beta epsilon [-1, 1]) is shown to be the only measure-invariant representation. The resulting two-parameter family of divergence functionals D(alpha,beta), (alpha, beta) epsilon [-1, 1] x [-1, 1] induces identical Fisher information but bidual alpha-connection pairs; it reduces in form to Amari's alpha-divergence family when alpha = +/-1 or when beta = 1, but to the family of Jensen difference (Rao, 1987) when beta = -1.