Selecting a "best subset" of input variables is a critical issue in forecasting. This is especially true when the number of available input series is large, and an exhaustive search through all combinations of variables is computationally infeasible. Inclusion of irrelevant variables not only doesn't help prediction, but can reduce forecasting accuracy through added noise or systematic bias. We demonstrate a technique called "sensitivity-based pruning" (SBP) that removes irrelevant input variables from a nonlinear forecasting or regression model. The technique makes use of a saliency measure computed for each input variable and uses estimates of prediction risk for determining the number of input variables to prune. We present preliminary results of the SBP technique applied to neural network predictors of a key business cycle measure, the US Index of Industrial Production.

We show that there are strong relationships between approaches to optmization and learning based on statistical physics or mixtures of experts. In particular, the EM algorithm can be interpreted as converging either to a local maximum of the mixtures model or to a saddle point solution to the statistical physics system. An advantage of the statistical physics approach is that it naturally gives rise to a heuristic continuation method, deterministic annealing, for finding good solutions.