In this paper, we summarize some recent results in Li et al. (2012), which can be used to extend an important PAC-Bayesian approach, namely the Gibbs posterior, to study the nonadditive ranking risk. The methodology is based on assumption-free risk bounds and nonasymptotic oracle inequalities, which leads to nearly optimal convergence rates and optimal model selection to balance the approximation errors and the stochastic errors.

This letter considers Bayesian binary classification where data are assumed to consist of multiple time series (panel data) with binary class labels (binary choice). The observed data can be represented as {yit, xit}T,t=1i = 1, … , n. Here yit∈ {0, 1} represents binary choices, and xitrepresents the exogenous variables. We consider prediction of yitby its own lags, as well as by the exogenous components. The prediction will be based on a Bayesian treatment using a Gibbs posterior that is constructed directly from the empirical error of classification. Therefore, this approach is less sensitive to the misspecification of the probability model compared to the usual likelihood-based posterior, which is confirmed by Monte Carlo simulations. We also study the effects of various choices of n and T both numerically (by simulations) and theoretically (by considering two alternative asymptotic situations: large n and large T). We find that increasing T helps to reduce the prediction error more effectively compared to increasing n. We also illustrate the method in a real data application on the brand choice of yogurt purchases.

We consider a class of nonlinear models based on mixtures of local autoregressive time series. At any given time point, we have a certain number of linear models, denoted as experts, where the vector of covariates may include lags of the dependent variable. Additionally, we assume the existence of a latent multinomial variable, whose distribution depends on the same covariates as the experts, that determines which linear process is observed. This structure, denoted as mixture-of-experts (ME), is considerably flexible in modeling the conditional mean function, as shown by Jiang and Tanner. In this paper, we present a formal treatment of conditions to guarantee the asymptotic normality of the maximum likelihood estimator (MLE), under stationarity and nonstationarity, and under correct model specification and model misspecification. The performance of common model selection criteria in selecting the number of experts is explored via Monte Carlo simulations. Finally, we present applications to simulated and real data sets, to illustrate the ability of the proposed structure to model not only the conditional mean, but also the whole conditional density.

We consider a novel class of non-linear models based on mixtures of local generalized linear time series. In our models, at any given time point, we have a certain number of generalized linear models (GLM), denoted as experts, where the vector of covariates may include functions of lags of the dependent variable. Additionally, we have a latent variable, whose distribution depends on the same covariates as the experts, that determines which GLM is observed. This structure is considerably flexible, as was shown by Jiang and Tanner in a series of papers for mixtures of GLM with independent observations. Carvalho and Tanner (2002) show that the maximum likelihood estimator is consistent and asymptotically normal for correctly specified as well as misspecified models, under the appropriate regularity conditions. In this paper, we discuss the use of the Wald test for hypothesis testing and illustrate the theory with an example using financial time series.

Previous researchers developed new learning architectures for sequential data by extending conventional hidden Markov models through the use of distributed state representations. Although exact inference and parameter estimation in these architectures is computationally intractable, Ghahramani and Jordan (1997) showed that approximate inference and parameter estimation in one such architecture, factorial hidden Markov models (FHMMs), is feasible in certain circumstances. However, the learning algorithm proposed by these investigators, based on variational techniques, is difficult to understand and implement and is limited to the study of real-valued data sets. This chapter proposes an alternative method for approximate inference and parameter estimation in FHMMs based on the perspective that FHMMs are a generalization of a well-known class of statistical models known as generalized additive models (GAMs; Hastie & Tibshirani, 1990). Using existing statistical techniques for GAMs as a guide, we have developed the generalized backfitting algorithm. This algorithm computes customized error signals for each hidden Markov chain of an FHMM and then trains each chain one at a time using conventional techniques from the hidden Markov models literature. Relative to previous perspectives on FHMMs, we believe that the viewpoint taken here has a number of advantages. First, it places FHMMs on firm statistical foundations by relating them to a class of models that are well studied in the statistics community, yet it generalizes this class of models in an interesting way. Second, it leads to an understanding of how FHMMs can be applied to many different types of time-series data, including Bernoulli and multinomial data, not just data that are real valued. Finally, it leads to an effective learning procedure for FHMMs that is easier to understand and easier to implement than existing learning procedures. Simulation results suggest that FHMMs trained with the generalized backfitting algorithm are a practical and powerful tool for analyzing sequential data.

In the class of hierarchical mixtures-of-experts (HME) models, "experts" in the exponential family with generalized linear mean functions of the form /spl psi/(/spl alpha/+x/sup T//spl beta/) are mixed, according to a set of local weights called the "gating functions" depending on the predictor x. Here /spl psi/(/spl middot/) is the inverse link function. We provide regularity conditions on the experts and on the gating functions under which the maximum-likelihood method in the large sample limit produces a consistent and asymptotically normal estimator of the mean response. The regularity conditions are validated for Poisson, gamma, normal, and binomial experts.

We investigate a class of hierarchical mixtures-of-experts (HME) models where generalized linear models with nonlinear mean functions of the form psi (alpha + xT beta) are mixed. Here psi (.) is the inverse link function. It is shown that mixtures of such mean functions can approximate a class of smooth functions of the form psi (h(x)), where h(.) epsilon W2;K infinity (a Sobolev class over [0, 1]s), as the number of experts m in the network increases. An upper bound of the approximation rate is given as O(m-2/s) in Lp norm. This rate can be achieved within the family of HME structures with no more than s-layers, where s is the dimension of the predictor x.

There does not exist a statistical model that shows good performance on all tasks. Consequently, the model selection problem is unavoidable; investigators must decide which model is best at summarizing the data for each task of interest. This article presents an approach to the model selection problem in hierarchical mixtures-of-experts architectures. These architectures combine aspects of generalized linear models with those of finite mixture models in order to perform tasks via a recursive "divide-and-conquer" strategy. Markov chain Monte Carlo methodology is used to estimate the distribution of the architectures' parameters. One part of our approach to model selection attempts to estimate the worth of each component of an architecture so that relatively unused components can be pruned from the architecture's structure. A second part of this approach uses a Bayesian hypothesis testing procedure in order to differentiate inputs that carry useful information from nuisance inputs. Simulation results suggest that the approach presented here adheres to the dictum of Occam's razor; simple architectures that are adequate for summarizing the data are favored over more complex structures. Copyright 1997 Elsevier Science Ltd. All Rights Reserved.