Partial information decompositions (PIDs), which quantify information interactions between three or more variables in terms of uniqueness, redundancy and synergy, are gaining traction in many application domains. However, our understanding of the operational interpretations of PIDs is still incomplete for many popular PID definitions. In this paper, we discuss the operational interpretations of unique information through the lens of two well-known PID definitions. We reexamine an interpretation from statistical decision theory showing how unique information upper bounds the risk in a decision problem. We then explore a new connection between the two PIDs, which allows us to develop an informal but appealing interpretation, and generalize the PID definitions using a common Lagrangian formulation. Finally, we provide a new PID definition that is able to capture the information that is unique. We also show that it has a straightforward interpretation and examine its properties.The full version of this paper is available online [1].

Recent advances in neuroscientific experimental techniques have enabled us to simultaneously record the activity of thousands of neurons across multiple brain regions. This has led to a growing need for computational tools capable of analyzing how task-relevant information is represented and communicated between several brain regions. Partial information decompositions (PIDs) have emerged as one such tool, quantifying how much unique, redundant and synergistic information two or more brain regions carry about a task-relevant message. However, computing PIDs is computationally challenging in practice, and statistical issues such as the bias and variance of estimates remain largely unexplored. In this paper, we propose a new method for efficiently computing and estimating a PID definition on multivariate Gaussian distributions. We show empirically that our method satisfies an intuitive additivity property, and recovers the ground truth in a battery of canonical examples, even at high dimensionality. We also propose and evaluate, for the first time, a method to correct the bias in PID estimates at finite sample sizes. Finally, we demonstrate that our Gaussian PID effectively characterizes inter-areal interactions in the mouse brain, revealing higher redundancy between visual areas when a stimulus is behaviorally relevant.

Bivariate partial information decompositions (PIDs) characterize how the information in a "message" random variable is decomposed between two "constituent" random variables in terms of unique, redundant and synergistic information components. These components are a function of the joint distribution of the three variables, and are typically defined using an optimization over the space of all possible joint distributions. This makes it computationally challenging to compute PIDs in practice and restricts their use to low-dimensional random vectors. To ease this burden, we consider the case of jointly Gaussian random vectors in this paper. This case was previously examined by Barrett [1], who showed that certain operationally well-motivated PIDs reduce to a closed form expression for scalar messages. Here, we show that Barrett’s result does not extend to vector messages in general, and characterize the set of multivariate Gaussian distributions that reduce to closed-form. Then, for all other multivariate Gaussian distributions, we propose a convex optimization framework for approximately computing a specific PID definition based on the statistical concept of deficiency. Using simplifying assumptions specific to the Gaussian case, we provide an efficient algorithm to approximately compute the bivariate PID for multivariate Gaussian variables with tens or even hundreds of dimensions. We also theoretically and empirically justify the goodness of this approximation.

Motivated by neuroscientific and clinical applications, we empirically examine whether observational measures of information flow can suggest interventions. We do so by performing experiments on artificial neural networks in the context of fairness in machine learning, where the goal is to induce fairness in the system through interventions. Using our recently developed M-information flow framework, we measure the flow of information about the true label (responsible for accuracy, and hence desirable), and separately, the flow of information about a protected attribute (responsible for bias, and hence undesirable) on the edges of a trained neural network. We then compare the flow magnitudes against the effect of intervening on those edges by pruning. We show that pruning edges that carry larger information flows about the protected attribute reduces bias at the output to a greater extent. This demonstrates that M-information flow can meaningfully suggest targets for interventions, answering the title's question in the affirmative. We also evaluate bias-accuracy tradeoffs for different intervention strategies, to analyze how one might use estimates of desirable and undesirable information flows (here, accuracy and bias flows) to inform interventions that preserve the former while reducing the latter.

With the growing use of machine learning algorithms in highly consequential domains, the quantification and removal of disparity in decision making with respect to protected attributes, such as gender, race, etc., is becoming increasingly important. While quantifying disparity is essential, sometimes the needs of a business (e.g., hiring) may require the use of certain features that are critical in a way that any disparity that can be explained by them might need to be exempted. For instance, in hiring a software engineer for a safety-critical application, a coding-test score may be a critical feature that is weighed strongly in the decision even if it introduces disparity, whereas other features, such as name, zip code, or reference letters may be used to improve decision-making, but only to the extent that they do not add disparity. In this work, we propose a novel information-theoretic decomposition of the total disparity (a quantification inspired from counterfactual fairness) into two components: a non-exempt component which quantifies the part of the disparity that cannot be accounted for by the critical features, and an exempt component which quantifies the remaining disparity. This decomposition is important: it allows one to check if the disparity arose purely due to the critical features (inspired from the business necessity defense of disparate impact law) and also enables selective removal of the non-exempt component of disparity if desired. We arrive at this decomposition through canonical examples that lead to a set of desirable properties (axioms) that any measure of non-exempt disparity should satisfy. We then demonstrate that our proposed counterfactual measure of non-exempt disparity satisfies all of them. Our quantification bridges ideas of causality, Simpson's paradox, and a body of work from information theory called Partial Information Decomposition (PID). We also obtain an impossibility result showing that no observational measure of non-exempt disparity can satisfy all of the desired properties, which leads us to relax our goals and examine alternative observational measures that satisfy only some of these properties. We perform case studies to show how one can audit existing models as well as train new models while reducing non-exempt disparity.

The needs of a business (e.g., hiring) may require the use of certain features that are critical in a way that any discrimination arising due to them should be exempted. In this work, we propose a novel information-theoretic decomposition of the total discrimination (in a counterfactual sense) into a non-exempt component, which quantifies the part of the discrimination that cannot be accounted for by the critical features, and an exempt component, which quantifies the remaining discrimination. Our decomposition enables selective removal of the non-exempt component if desired. We arrive at this decomposition through examples and counterexamples that enable us to first obtain a set of desirable properties that any measure of non-exempt discrimination should satisfy. We then demonstrate that our proposed quantification of non-exempt discrimination satisfies all of them. This decomposition leverages a body of work from information theory called Partial Information Decomposition (PID). We also obtain an impossibility result showing that no observational measure of non-exempt discrimination can satisfy all of the desired properties, which leads us to relax our goals and examine alternative observational measures that satisfy only some of these properties. We then perform a case study using one observational measure to show how one might train a model allowing for exemption of discrimination due to critical features.

Recently, we developed a systematic framework for defining and inferring flows of information about a specific message in neural circuits [2], [3]. We defined a computational model of a neural circuit consisting of computational nodes and transmissions being sent between these nodes over time. We then gave a formal definition of information flow pertaining to a specific message, which was capable of identifying paths along which information flowed in such a system. However, this definition also had some non-intuitive properties, such as the existence of "orphans"-nodes from which information flowed out, even though no information flowed in. In part, these non-intuitive properties arose because we restricted our attention to measures that were functions of transmissions at a single time instant, and measures that were observational rather than counterfactual. In this paper, we consider alternative definitions, including one that is a function of transmissions at multiple time instants, one that is counterfactual, and a new observational definition. We show that a definition of information flow based on counterfactual causal influence (CCI) guarantees the existence of information paths while also having no orphans. We also prove that no observational definition of information flow that satisfies the information path property can match CCI in every instance. Furthermore, each of the definitions we examine (including CCI) is shown to have examples in which the information flow can take a non-intuitive path. Nevertheless, we believe our framework remains more amenable to drawing clear interpretations than classical tools used in neuroscience, such as Granger Causality.

We develop a theoretical framework for defining and identifying flows of information in computational systems. Here, a computational system is assumed to be a directed graph, with “clocked” nodes that send transmissions to each other along the edges of the graph at discrete points in time. We are interested in a definition that captures the dynamic flow of information about a specific message, and which guarantees an unbroken “information path” between appropriately defined inputs and outputs in the directed graph. Prior measures, including those based on Granger Causality and Directed Information, fail to provide clear assumptions and guarantees about when they correctly reflect information flow about a message. We take a systematic approach-iterating through candidate definitions and counterexamples-to arrive at a definition for information flow that is based on conditional mutual information, and which satisfies desirable properties, including the existence of information paths. Finally, we describe how information flow might be detected in a noiseless setting, and provide an algorithm to identify information paths on the time-unrolled graph of a computational system.

We develop a theoretical framework for defining information flow in neural circuits, within the context of "eventrelated" experimental paradigms in neuroscience. Here, a neural circuit is modeled as a directed graph, with "clocked" nodes that send transmissions to each other along the edges of the graph at discrete points in time. We are interested in a definition that captures the flow of "stimulus"-related information, and which guarantees a continuous information path between appropriately defined inputs and outputs in the directed graph. Prior measures, including those based on Granger Causality and Directed Information, fail to provide clear assumptions and guarantees about when they correctly reflect stimulus-related information flow, due to the absence of a theoretical foundation with a mathematical definition. We take a methodical approach- iterating through candidate definitions and counterexamples- to arrive at a definition for information flow that is based on conditional mutual information, and which satisfies desirable properties, including the existence of information paths.

We propose and analyze the problems of \textit{community goodness-of-fit and two-sample testing} for stochastic block models (SBM), where changes arise due to modification in community memberships of nodes. Motivated by practical applications, we consider the challenging sparse regime, where expected node degrees are constant, and the inter-community mean degree ($b$) scales proportionally to intra-community mean degree ($a$). Prior work has sharply characterized partial or full community recovery in terms of a ``signal-to-noise ratio'' ($\mathrm{SNR}$) based on $a$ and $b$. For both problems, we propose computationally-efficient tests that can succeed far beyond the regime where recovery of community membership is even possible. Overall, for large changes, $s \gg \sqrt{n}$, we need only $\mathrm{SNR}= O(1)$ whereas a na\"ive test based on community recovery with $O(s)$ errors requires $\mathrm{SNR}= \Theta(\log n)$. Conversely, in the small change regime, $s \ll \sqrt{n}$, via an information theoretic lower bound, we show that, surprisingly, no algorithm can do better than the na\"ive algorithm that first estimates the community up to $O(s)$ errors and then detects changes. We validate these phenomena numerically on SBMs and on real-world datasets as well as Markov Random Fields where we only observe node data rather than the existence of links.

The “source localization” problem is one in which we estimate the location of a point source observed through a diffusive medium using an array of sensors. We obtain lower bounds on the minimax risk (mean squared-error in location) in estimating the location of the source, which apply to all estimators, for certain classes of diffusive media, when using a uniformly distributed sensor array. We show that for sensors of a fixed size, the lower bound decays to zero with increasing numbers of sensors. We also analyze a more physical sensor model to understand the effect of shrinking the size of sensors as their number increases to infinity, wherein the bound saturates for large sensor numbers. In this scenario, it is seen that there is greater benefit to increasing the number of sensors as the signal-to-noise ratio increases. Our bounds are the first to give a scaling for the minimax risk in terms of the number of sensors used.

This paper uses an information-theoretic lens to examine the use and implementation of electroencephalography (EEG) systems for neural imaging. There is a widespread belief in the clinical and neuroscientific community that ultra-high-density EEG imaging of the brain (beyond the typically used few hundred or fewer electrodes) will not yield higher spatial resolution. Theoretically, this belief rests on a spatial Nyquist rate analysis for human head models. This analysis in turn relies on the understanding that high spatial frequencies (that carry high-resolution information) decay as they travel from the brain to the scalp surface. Interestingly, the belief continues to be held despite being recently challenged experimentally, in part because of the spatial Nyquist results. In this work, we question the spatial Nyquist rate analysis, mathematically as well as conceptually. Mathematically, we correct and generalize the Nyquist rate analysis, and then use it to obtain improved estimates for spatial Nyquist rates. Conceptually, we observe that the Nyquist rate analysis provides a limit on the following problem: what is the minimum number of sensors required to reconstruct the scalp EEG to within a specified (small) mean-squared error? However, this problem is fundamentally different from the imaging problem of minimizing error in reconstructing the neural source inside the brain, which optimizes a different objective, and requires inclusion of noise in the analysis. To that end, we utilize the transfer function to obtain an information-theoretic fundamental limit on the achievable accuracy in localizing single-dipole sources. The technique relies on computing the distortion-rate function for a single dipole source and evaluating it at an upper bound on mutual information across the brain-to-scalp channel, and can be extended to more general sources as well. Finally, we observe that the main obstacle in understanding the required number of EEG sensors is the lack of ultra-high-density systems that can test these limits, and information theory can prove helpful in this context as well. One engineering difficulty is that the required circuit area and energy for sampling EEG signals can be too large to enable these systems to be safe, compact, and portable. Toward reducing this area and energy, we observe that the decay of high spatial frequencies in EEG, while being a detriment to reconstruction accuracy, also leads to large spatial correlations in the recorded EEG signals. Interestingly, these correlations can be harnessed using a novel information-theoretic "hierarchical referencing" technique that can reduce circuit energy and area to enable high-resolution high-density implementations.