This paper introduces two Gaussian graphical models defined on complete bipartite graphs. We show that the determinants of the precision matrices associated with the models are equal up to scale, where the scale factor only depends on model parameters. In this context, we will introduce a notion of “equivalence” between the two Gaussian graphical models. This equivalence has two key applications: first, it can significantly reduce the complexity of computing the exact value of the determinant, and second, it enables the derivation of closed-form expressions for the determinants in certain special cases.

For a specific class of sparse Gaussian graphical models, we provide a closed-form solution for the determinant of the covariance matrix. In our framework, the graphical interaction model (i.e., the covariance selection model) is equal to replacement product of ${\mathcal{K}_n}$ and ${\mathcal{K}_{n - 1}}$, where ${\mathcal{K}_n}$ is the complete graph with n vertices. Our analysis is based on taking the Fourier transform of the local factors of the model, which can be viewed as an application of the Normal Factor Graph Duality Theorem and holographic algorithms. The closed-form expression is obtained by applying the Matrix Determinant Lemma on the transformed graphical model. In this context, we will also define a notion of equivalence between two Gaussian graphical models.

We present local mappings that relate the marginal probabilities of a global probability mass function represented by its primal normal factor graph to the corresponding marginal probabilities in its dual normal factor graph. The mapping is based on the Fourier transform of the local factors of the models. Details of the mapping are provided for the Ising model, where it is proved that the local extrema of the fixed points are attained at the phase transition of the two-dimensional nearest-neighbor Ising model. The results are further extended to the Potts model, to the clock model, and to Gaussian Markov random fields. By employing the mapping, we can transform simultaneously all the estimated marginal probabilities from the dual domain to the primal domain (and vice versa), which is advantageous if estimating the marginals can be carried out more efficiently in the dual domain. An example of particular significance is the ferromagnetic Ising model in a positive external magnetic field. For this model, there exists a rapidly mixing Markov chain (called the subgraphs--world process) to generate configurations in the dual normal factor graph of the model. Our numerical experiments illustrate that the proposed procedure can provide more accurate estimates of marginal probabilities of a global probability mass function in various settings.

The dual normal factor graph and the factor graph duality theorem have been considered for discrete graphical models. In this paper, we show an application of the factor graph duality theorem to continuous graphical models. Specifically, we propose a method to solve exactly the Gaussian graphical models defined on the ladder graph if certain conditions on the local covariance matrices are satisfied. Unlike the conventional approaches, the efficiency of the method depends on the position of the zeros in the local covariance matrices. The method and details of the dualization are illustrated on two toy examples.

We prove that the marginal densities of a global probability mass function in aprimal normal factor graph and the corresponding marginal densities in the dual normal factor graph are related via local mappings. The mapping depends on the Fourier transform of the local factors of the models. Details of the mapping, including its fixed points, are derived for the Ising model, and then extended to the Potts model. By employing the mapping, we can transform simultaneously all the estimated marginal densities from one domain to the other, which is advantageous if estimating the marginals can be carried out more efficiently in the dual domain.An example of particular significance is the ferromagnetic Ising model in a positive external field, for which there is a rapidly mixing Markov chain (called the subgraphs-world process) to generate configurations in the dual normal factor graph of the model. Our numerical experiments illustrate that the proposed procedure can provide more accurate estimates of marginal densities in various settings.

Normal factor graph duality offers new possibilities for Monte Carlo algorithms in graphical models. Specifically, we consider the problem of estimating the partition function of the ferromagnetic Ising and Potts models by Monte Carlo methods, which are known to work well at high temperatures but to fail at low temperatures. We propose Monte Carlo methods (uniform sampling and importance sampling) in the dual normal factor graph and demonstrate that they behave differently: they work particularly well at low temperatures. By comparing the relative error in estimating the partition function, we show that the proposed importance sampling algorithm significantly outperforms the state-of-the-art deterministic and Monte Carlo methods. For the ferromagnetic Ising model in an external field, we show the equivalence between the valid configurations in the dual normal factor graph and the terms that appear in the high-temperature series expansion of the partition function. Following this result, we discuss connections with Jerrum-Sinclair's polynomial randomized approximation scheme (the subgraphs-world process) for evaluating the partition function of ferromagnetic Ising models.

We propose Monte Carlo methods to estimate the partition function of the two-dimensional Ising model in the presence of an external magnetic field. The estimation is done in the dual of the Forney factor graph representing the model. The proposed methods can efficiently compute an estimate of the partition function in a wide range of model parameters. As an example, we consider models that are in a strong external field.

The partition function of a factor graph and the partition function of the dual factor graph are related to each other by the normal factor graph duality theorem. We apply this result to the classical problem of computing the partition function of the Ising model. In the one-dimensional case, we thus obtain an alternative derivation of the (well-known) analytical solution. In the two-dimensional case, we find that Monte Carlo methods are much more efficient on the dual graph than on the original graph, especially at low temperature.

The paper proposes Monte Carlo algorithms for the computation of the information rate of 2-D source/channel models. The focus of the paper is on binary-input channels with constraints on the allowed input configurations. The problem of numerically computing the information rate, and even the noiseless capacity, of such channels has so far remained largely unsolved. Both problems can be reduced to computing a Monte Carlo estimate of a partition function. The proposed algorithms use tree-based Gibbs sampling and multilayer (multitemperature) importance sampling. The viability of the proposed algorithms is demonstrated by simulation results.

The partition function of a factor graph can sometimes be accurately estimated by Monte Carlo methods. In this paper, such methods are extended to factor graphs with negative and complex factors.

The performance of the generalized belief propagation algorithm to compute the noiseless capacity and mutual information rates of finite-size two-dimensional and three-dimensional run-length limited constraints is investigated. In both cases, the problem is reduced to estimating the partition function of graphical models with cycles. The partition function is then estimated using the region-based free energy approximation technique. For each constraint, a method is proposed to choose the basic regions and to construct the region graph which provides the graphical framework to run the generalized belief propagation algorithm. Simulation results for the noiseless capacity of different constraints as a function of the size of the channel are reported. In the cases that tight lower and upper bounds on the Shannon capacity exist, convergence to the Shannon capacity is discussed. For noisy constrained channels, simulation results are reported for mutual information rates as a function of signal-to-noise ratio.

The problem of computing the information rate of noisy two-dimensional constrained source/channel models has been an unsolved problem. In this paper, we propose two Monte Carlo methods for this problem. The first method, which is exact in expectation, combines tree-based Gibbs sampling with importance sampling. The second method uses generalized belief propagation and is shown to yield a good approximation of the information rate.

The performance of the generalized belief propagation algorithm for computing the noiseless capacity of finite-sized two-dimensional and three-dimensional run-length limited constraints is investigated. For each constraint, a method is proposed to choose a set of clusters. Simulation results for different sizes of channels with different constraints are reported. Convergence to the Shannon capacity is also discussed.

Monte Carlo methods are considered to compute, first, the partition function of graphical models, and second, the information rate of source/channel models, in both cases for factor graphs with cycles. The convergence of two basic Monte Carlo methods is improved by sampling only a cycle breaking subset of the variables and using exact sum-product computations for the remaining variables. The methods are demonstrated by their application to a two-dimensional Ising model and to a two-dimensional intersymbol interference channel.

Computing the partition function and the marginals of a global probability distribution are two important issues in any probabilistic inference problem. In a previous work, we presented sub-tree based upper and lower bounds on the partition function of a given probabilistic inference problem. Using the entropies of the sub-trees we proved an inequality that compares the lower bounds obtained from different sub-trees. In this paper we investigate the properties of one specific lower bound, namely the lower bound computed by the minimum entropy sub-tree. We also investigate the relationship between the minimum entropy sub-tree and the sub-tree that gives the best lower bound.

A probabilistic inference problem is concerned with computing the partition function and the marginals of a global probability distribution. We investigate upper and lower bounds on the partition function of a given inference problem. Our bounds depend on the partition function of any sub-junction tree of a given junction graph representing the inference problem. In a theorem we compare such bounds using the entropies of the sub-junction trees. Finally we propose a greedy algorithm that computes low-complexity bounds on the partition function. Simulation results on two-dimensional grids are reported

Recent work has related the belief propagation algorithm for probabilistic inference problem to some approximations to the free energy function in statistical physics. In this paper we investigate some properties of one such approximation, called the Bethe-Kikuchi approximation. We also derive low-complexity upper and lower bounds on the partition function, i.e. the global normalization constant, for a given inference problem

This paper extends the construction and analysis of Raptor codes originally designed in A. Shokrollahi (2004) for the erasure channel to general symmetric channels. We explicitly calculate the asymptotic fraction of output nodes of degree one and two for capacity-achieving Raptor codes, and discuss techniques to optimize the output degree distribution.