The Shannon entropy behaves in many ways like a signed measure. Previous work has explored this connection by defining a signed measure on abstract sets, representing the information that different variables contain. This yields many measure-theoretic counterparts to information quantities such as the mutual information (set intersection), the joint entropy (set union), and the conditional entropy (set difference), but an explicit construction of the elements of these sets remains ambiguous. Here we provide a concrete characterisation of these abstract sets and a corresponding signed measure by extending Yeung’s I-measure to all possible outcomes in an outcome space Ω, and in doing so we demonstrate a much finer decomposition with intuitive properties that we call the logarithmic decomposition (LD). We show that this signed measure space has the useful property that its logarithmic atoms can be characterised as having negative or positive entropy (depending only on their structure), while being consistent with classical information quantities. Using this logarithmic decomposition, we re-examine the Gács-Körner common information, the functional common information, and minimally sufficient statistics from this new perspective and characterise them in terms of our logarithmic atoms—a property we call logarithmic decomposability. We present possible extensions of this construction to continuous probability distributions, before finally applying our new decomposition to the Dyadic and Triadic systems of James and Crutchfield as a motivating example. We show that, in contrast to the I-measure alone, our classically derived decomposition qualitatively distinguishes between them.

The Shannon entropy of a random variable X has much behaviour analogous to a signed measure. Previous work has concretized this connection by defining a signed measure µ on an abstract information space $\tilde X$, which is taken to represent the information that X contains. This construction is sufficient to derive many measure-theoretical counterparts to information quantities such as the mutual information $I(X;Y) = \mu (\tilde X \cap \tilde Y)$, the joint entropy $H(X,Y) = \mu (\tilde X \cup \tilde Y)$, and the conditional entropy $H(X|Y) = \mu (\tilde X\backslash \tilde Y)$. We demonstrate that there exists a much finer decomposition with intuitive properties which we call the logarithmic decomposition (LD). We show that this signed measure space has the useful property that its logarithmic atoms are easily characterised with negative or positive entropy, while also being coherent with Yeung’s I-measure [14]. We present the usability of our approach by re-examining the Gács-Körner common information from this new geometric perspective and characterising it in terms of our logarithmic atoms. We then highlight that our geometric refinement can account for an entire class of information quantities, which we call logarithmically decomposable quantities.