We present Probabilistic Orthogonal Matching Pursuit (PrOMP), a novel probabilistic approach that builds upon orthogonal matching pursuit (OMP) for sparse representations of data. Like OMP, PrOMP is a greedy algorithm for regression that iteratively selects columns of a matrix according to a score. This score is based on a rarely employed feature of the EM algorithm and thus optimizes a marginal probability distribution. While OMP uses correlation as the score, in our probabilistic approach we define the scores to be the value of the resulting marginal likelihood—if adding a new signal does not improve this term, the algorithm automatically terminates. Our theoretical analysis also builds on the previous theory for OMP. We demonstrate the algorithm with a focus on a sparse dictionary learning and signal representation task using Bayesian nonparametrics. We first consider the nonparametric Beta Process Factor Analysis (BPFA) model. In addition, we present a new model based on BPFA that we call Beta Process Subspace Analysis (BPSA) which learns a set of subspaces and their respective dimensionalities from data.

Bayesian nonparametric methods provide a convenient and well-founded framework for constructing spatio-temporally evolving ensemble models. They not only provide a flexible way to weight different underlying models in an ensemble according to the strengths of each model on different regions of the input space, but also allow for quantification of the ensemble’s uncertainty. However, computational costs can pose a challenge to kernel-based ensemble methods when spatio-temporal resolution is high, such as environmental models. We propose a Bayesian Nonparametric Ensemble (BNE) method for spatio-temporal ensemble learning based on the Gaussian process. Our ensemble relies on theoretically well-founded linearized approximations to the Gaussian process to adaptively weight the underlying models in a way that is scalable and amenable to stochastic learning methods. We investigate both the random Fourier feature and Nystrom approaches in this setting. We demonstrate the practicality and usefulness of the approximate model on the problem of air pollution prediction across the contiguous USA over a 6 year period.

We present a Bayesian tensor factorization model for inferring latent group structures from dynamic pairwise interaction patterns. For decades, political scientists have collected and analyzed records of the form "country i took action a toward country j at time t" - known as dyadic events - in order to form and test theories of international relations. We represent these event data as a tensor of counts and develop Bayesian Poisson tensor factorization to infer a low-dimensional, interpretable representation of their salient patterns. We demonstrate that our model's predictive performance is better than that of standard non-negative tensor factorization methods. We also provide a comparison of our variational updates to their maximum likelihood counterparts. In doing so, we identify a better way to form point estimates of the latent factors than that typically used in Bayesian Poisson matrix factorization. Finally, we showcase our model as an exploratory analysis tool for political scientists. We show that the inferred latent factor matrices capture interpretable multilateral relations that both conform to and inform our knowledge of international a airs.

We present the collaborative Kalman filter (CKF), a dynamic model for collaborative filtering and related factorization models. Using the matrix factorization approach to collaborative filtering, the CKF accounts for time evolution by modeling each low-dimensional latent embedding as a multidimensional Brownian motion. Each observation is a random variable whose distribution is parameterized by the dot product of the relevant Brownian motions at that moment in time. This is naturally interpreted as a Kalman filter with multiple interacting state space vectors. We also present a method for learning a dynamically evolving drift parameter for each location by modeling it as a geometric Brownian motion. We handle posterior intractability via a mean-field variational approximation, which also preserves tractability for downstream calculations in a manner similar to the Kalman filter. We evaluate the model on several large datasets, providing quantitative evaluation on the 10 million Movie lens and 100 million Netflix datasets and qualitative evaluation on a set of 39 million stock returns divided across roughly 6,500 companies from the years 1962-2014.