The problem of measuring conditional dependence between two random phenomena arises when a third one (a confounder) has a potential influence on the amount of information between them. A typical issue in this challenging problem is the inversion of ill-conditioned autocorrelation matrices. This letter presents a novel measure of conditional dependence based on the use ofincompleteunbiased statistics of degree two, which allows to re-interpret independence as uncorrelatedness on a finite-dimensional feature space. This formulation enables to prune data according to observations of the confounder itself, thus avoiding matrix inversions altogether. The proposed approach is articulated as an extension of the Hilbert-Schmidt independence criterion, which becomes expressible through kernels that operate on 4-tuples of data.

It is shown that R´enyi's entropy of a Gaussian mixture with entropic index α ∊ (1,∞] is upper-bounded by the cluster with minimum variance. This basic idea leads to a clean worst-case formulation of the minimum error entropy principle in the context of linear multi-sensor fusion by using a largely contaminated Gaussian distribution to model sensor errors with outliers. The obtained entropic best linear unbiased estimator leads to an operational interpretation in terms of a precision/reliability trade-off, it resonates closely with model order selection methods, and it provides a possible information theoretic root to sparsity-promoting regularization.

This article explores a plug-in estimator of second-order Tsallis entropy based on Kernel Density Estimation KDE and its implicit regularization process. First, it is shown that the expected value of the estimator corresponds to the entropy of an (AWGN) model. Then, we prove various relevant properties of the Tsallis entropy: it is monotonically non-decreasing under random variables addition, its derivative with respect to the Gaussian noise power is monotonically non-increasing, and it is concave in the additive noise power. From these, we derive an information metric that provides an alternative to the strategy of regularization.

This paper aims to estimate the information between two random phenomena by using consolidated second-order statistics tools. The squared-loss mutual information, a surrogate of the Shannon mutual information, is chosen due to its property of being expressed as a second-order moment. We first review the rationale for i.i.d. discrete sources, which involves mapping the data onto the simplex space, and we highlight the links with other well-known related concepts in the literature based on local approximations of information-theoretic measures. Then, the problem is translated to analog sources by mapping the data onto the characteristic space, focusing on the adaptability between the discrete and the analog case and its limitations. The proposed approach gains interpretability and scalability for its use on large data sets, providing a unified rationale for the free regularization parameters. Moreover, the structure of the proposed mapping allows resorting to Szegö’s theorem to reduce the complexity for high dimensional mappings, exhibiting a strong duality with spectral analysis. The performance of the developed estimators is analyzed using Gaussian mixtures.

We study the problem of estimating the overall mutual information in M independent parallel discrete-time memory-less Gaussian channels from N independent data sample pairs per channel (inputs and outputs). We focus on the case where the number of active channels L is sparse in comparison with the total number of channels (L ≪ M), for which the direct application of the maximum likelihood principle is problematic due to overfitting, especially for moderate to small N. For this regime, we show that the bias of the mutual information estimate is reduced by resorting to the minimum description length (MDL) principle. As a result, simple pre-processing based on a per-channel threshold on the empirical squared correlation coefficient is required with a fixed threshold that monotonically decreases with N as 1 - N-1/N, for N ≥ 4. The resulting improvement is shown in terms of the estimated information bias.

Squared-loss mutual information (SMI) is a surrogate of Shannon mutual information that is more advantageous for estimation. On the other hand, the coherence matrix of a pair of random vectors, a power-normalized version of the sample cross-covariance matrix, is a well-known second-order statistic found in the core of fundamental signal processing problems, such as canonical correlation analysis (CCA). This paper shows that SMI can be estimated from a pair of independent and identically distributed (i.i.d.) samples as a squared Frobenius norm of a coherence matrix estimated after mapping the data onto some fixed feature space. Moreover, low computation complexity is achieved through the fast Fourier transform (FFT) by exploiting the Toeplitz structure of the involved autocorrelation matrices in that space. The performance of the method is analyzed via computer simulations using Gaussian mixture models.

This paper shows that de Bruijn identity, which relates entropy with Fisher information, can be obtained as a particular case of an immediate generalization of Price's theorem, which is a tool used in the analysis of nonlinear memoryless systems with Gaussian inputs. It is shown that, while the general Price's theorem follows since the density of the perturbation satisfies the heat equation, the particular case of de Bruijn identity follows since the score function is zero-mean, which is the well-known condition that provides the insightful Cramér- Rao bound expression based on the negative second derivative of the log-likelihood function. The unified framework uses the characteristic function as a main tool and becomes a more intuitive alternative to the classical technical proof obtained by integrating by parts. Second-order Tsallis entropy is also briefly explored under this general framework.