Real-valued time series are ubiquitous in the sciences and engineering. In this work, a general, hierarchical Bayesian modelling framework is developed for building mixture models for times series. This development is based, in part, on the use of context trees, and it includes a collection of effective algorithmic tools for learning and inference. A discrete context (or ‘state’) is extracted for each sample, consisting of a discretised version of some of the most recent observations preceding it. The set of all relevant contexts are represented as a discrete context tree. At the bottom level, a different real-valued time series model is associated with each context-state, i.e., with each leaf of the tree. This defines a very general framework that can be used in conjunction with any existing model class to build flexible and interpretable mixture models. Extending the idea of context-tree weighting leads to algorithms that allow for efficient, exact Bayesian inference in this setting. The utility of the general framework is illustrated in detail when autoregressive (AR) models are used at the bottom level, resulting in a nonlinear AR mixture model. The associated methods are found to outperform several state-of-the-art techniques on simulated and real-world experiments.

Estimating the entropy rate of discrete time series is a challenging problem with important applications in numerous areas including neuroscience, genomics, image processing and natural language processing. A number of approaches have been developed for this task, typically based either on universal data compression algorithms, or on statistical estimators of the underlying process distribution. In this work, we propose a fully-Bayesian approach for entropy estimation. Building on the recently introduced Bayesian Context Trees (BCT) framework for modelling discrete time series as variable-memory Markov chains, we show that it is possible to sample directly from the induced posterior on the entropy rate. This can be used to estimate the entire posterior distribution, providing much richer information than point estimates. We develop theoretical results for the posterior distribution of the entropy rate, including proofs of consistency and asymptotic normality. The practical utility of the method is illustrated on both simulated and real-world data, where it is found to outperform state-of-the-art alternatives.

The Context-Tree Weighting (CTW) algorithm and the accompanying collection of ideas and techniques have a long history of statistical applications in discrete time series analysis. CTW was recently revisited from a principled Bayesian statistics point of view, and a general modelling framework called Bayesian Context Trees (BCT) was introduced and found to be very effective in numerous core statistical tasks. In this work, a novel representation of the induced BCT posterior distribution on model space is derived in terms of a simple branching process, and several consequences of this are explored in theory and in practice. First, it is shown that it leads to a simple variable-dimensional Monte Carlo sampler for the joint posterior on models and parameters, which is found to be more efficient than earlier MCMC samplers for the same tasks. Then the branching process representation is used to establish the asymptotic consistency of the BCT posterior, including the derivation of an almost-sure convergence rate.

Change-point detection for discrete time series is an important task with numerous applications. We develop a new hierarchical Bayesian framework for modelling inhomogeneous discrete time series with change-points. The distributions of different segments are modelled as variable-memory Markov chains, defining piece-wise homogeneous variable-memory chains. Building on the recently introduced Bayesian Context Trees framework, it is shown that the Context-Tree Weighting algorithm can be employed to compute the prior predictive likelihood of each segment, with all models and parameters integrated out. This is then used to develop a new class of effective Markov chain Monte Carlo algorithms for the posterior of the number and locations of change-points. These not only identify the most likely change-points, but also provide access to their entire posterior distribution. Estimates of the actual models in each segment can be obtained at negligible cost. Results on both synthetic and real-world data sets indicate that the proposed methodology performs better or as well as state-of-the-art techniques.

We revisit the statistical foundation of the celebrated context tree weighting (CTW) algorithm, and we develop a Bayesian modelling framework for the class of higher-order, variable-memory Markov chains, along with an associated collection of methodological tools for exact inference for discrete time series. In addition to deterministic algorithms that learn the a posteriori most likely models and compute their posterior probabilities, we introduce a family of variable-dimension Markov chain Monte Carlo samplers, facilitating further exploration of the posterior. The performance of the proposed methods in model selection, Markov order estimation and prediction is illustrated through simulation experiments and real-world applications.