We investigate the properties of the entropy of a probability measure along the heat flow and more precisely we seek for closed algebraic representations of its derivatives. Provided that the measure admits moments of any order, it has been proved in Guo et al. (2010) that this functional is smooth, and in Ledoux (2016) that its derivatives at zero can be expressed into multivariate polynomials evaluated in the moments (or cumulants) of the underlying measure. Moreover, these algebraic expressions are derived through$\Gamma $-calculus techniques which provide implicit recursive formulas for these polynomials. Our main contribution consists in a fine combinatorial analysis of these inductive relations and for the first time to derive closed formulas for the leading coefficients of these polynomials expressions. Building upon these explicit formulas we revisit the so-called “MMSE conjecture” from Ledoux (2016) which asserts that two distributions on the real line with the same entropy along the heat flow must coincide up to translation and symmetry. Our approach enables us to provide new conditions on the source distributions ensuring that the MMSE conjecture holds and to refine several criteria proved in Ledoux (2016). As illustrating examples, our findings cover the cases of uniform and Rademacher distributions, for which previous results in the literature were inapplicable.

In this paper, we introduce new Stein identities for gamma target distribution as well as a new non-linear channel specifically designed for gamma inputs. From these two ingredients, we derive an explicit and simple formula for the derivative of the input-output mutual information of this non-linear channel with respect to the channel quality parameter. This relation is reminiscent of the well-known link between the derivative of the input-output mutual information of additive Gaussian noise channel with respect to the signal-to-noise ratio and the minimum mean-square error. The proof relies on a rescaled version of De Bruijn identity for gamma target distribution together with a stochastic representation for the gamma-specific Fisher information. Finally, we are able to derive precise bounds and asymptotics for the input-output mutual information of the non-linear channel with gamma inputs.

We introduce a new formalism for computing expectations of functionals of arbitrary random vectors, by using generalised integration by parts formulae. In doing so we extend recent representation formulae for the score function introduced in [19] and also provide a new proof of a central identity first discovered in [7]. We derive a representation for the standardised Fisher information of sums of i.i.d. random vectors which we use to provide rates of convergence in information theoretic central limit theorems (both in Fisher information distance and in relative entropy) and a Stein bound for Fisher information distance.

Pinsker's inequality states that the relative entropy between two random variables X and Y dominates the square of the total variation distance between X and Y. In this paper, we introduce generalized Fisher information distances and prove that these also dominate the square of the total variation distance. To this end, we introduce a general discrete Stein operator for which we prove a useful covariance identity. We illustrate our approach with several examples. Whenever competitor inequalities are available in the literature, the constants in ours are at least as good, and, in several cases, better.