Nonlinear multilayer principal component analysis (NMPCA) is well-known as an improved version of principal component analysis (PCA) using a five layer bottleneck neural network. NMPCA enables us to extract nonlinear hidden structure from high dimensional data, however, it has been difficult for users to understand obtained results, because trained results of NMPCA have many different locally optimal parameters depending on initial parameters. There has been no method how to find a few essential structures from many differently trained networks. This paper proposes a new interpretation method of NMPCA by extracting a few essential structures from many differently trained and locally optimal parameters. In the proposed method, firstly the weight parameters are made to be sparsely represented by LASSO training and appropriately ordered using the generalized factor loadings, then classified into a few hierarchical clusters, so that users can understand the extracted results. Its effectiveness is shown by both artificial and real world problems.

Almost all learning machines used in computational intelligence are not regular but singular statistical models, because they are nonidentifiable and their Fisher information matrices are singular. In singular learning machines, neither the Bayes a posteriori distribution converges to the normal distribution nor the maximum likelihood estimator satisfies the asymptotic normality, resulting that it has been difficult to estimate generalization performances. In this paper, we establish a formula of equations of states which holds among Bayes and Gibbs generalization and training errors, and show that two generalization errors can be estimated from two training errors. The equations of states proved in this paper hold for any true distribution, any learning machine, and a priori distribution, and any singularities, hence they define widely applicable information criteria.

Many singular learning machines such as neural networks, normal mixtures, Bayesian networks, and hidden Markov models belong to singular learning machines and are widely used in practical information systems. In these learning machines, it is well known that Bayesian learning provides better generalization performance than maximum-likelihood estimation. However, it needs huge computational cost to sample from a Bayesian posterior distribution of a singular learning machine by a conventional Markov chain Monte Carlo (MCMC) method, such as the metropolis algorithm, because of singularities. Recently, the exchange Monte Carlo (MC) method, which is well known as an improved algorithm of MCMC method, has been proposed to apply to Bayesian neural network learning in the literature. In this paper, we propose the idea that the exchange MC method has a better effect on Bayesian learning in singular learning machines than that in regular learning machines, and show its effectiveness by comparing the numerical stochastic complexity with the theoretical one.

In supervised learning, we commonly assume that training and test data are sampled from the same distribution. However, this assumption can be violated in practice and then standard machine learning techniques perform poorly. This paper focuses on revealing and improving the performance of Bayesian estimation when the training and test distributions are different. We formally analyze the asymptotic Bayesian generalization error and establish its upper bound under a very general setting. Our important finding is that lower order terms---which can be ignored in the absence of the distribution change---play an important role under the distribution change. We also propose a novel variant of stochastic complexity which can be used for choosing an appropriate model and hyper-parameters under a particular distribution change.

It is well known that in unidentifiable models, the Bayes estimation provides much better generalization performance than the maximum likelihood (ML) estimation. However, its accurate approximation by Markov chain Monte Carlo methods requires huge computational costs. As an alternative, a tractable approximation method, called the variational Bayes (VB) approach, has recently been proposed and has been attracting attention. Its advantage over the expectation maximization (EM) algorithm, often used for realizing the ML estimation, has been experimentally shown in many applications; nevertheless, it has not yet been theoretically shown. In this letter, through analysis of the simplest unidentifiable models, we theoretically show some properties of the VB approach. We first prove that in three-layer linear neural networks, the VB approach is asymptotically equivalent to a positive-part James-Stein type shrinkage estimation. Then we theoretically clarify its free energy, generalization error, and training error. Comparing them with those of the ML estimation and the Bayes estimation, we discuss the advantage of the VB approach. We also show that unlike in the Bayes estimation, the free energy and the generalization error are less simply related with each other and that in typical cases, the VB free energy well approximates the Bayes one, while the VB generalization error significantly differs from the Bayes one.

A lot of singular learning machines such as neural networks, normal mixtures, Bayesian networks and hidden Markov models are widely used in practical information systems. In these learning machines, it was clarified that the Bayesian learning provides the better generalization performance than the maximum likelihood method. However, it needs huge computational costs to realize the Bayesian posterior distribution by the conventional Monte Carlo method. In this paper, we propose that the exchange Monte Carlo method is appropriate for the Bayesian learning of singular learning machines, and experimentally show that it attains the better posterior distribution than the conventional Monte Carlo method.

Bayesian learning has been widely used and proved to be effective in many data modeling problems. However, computations involved in it require huge costs and generally cannot be performed exactly. The variational Bayesian approach, proposed as an approximation of Bayesian learning, has provided computational tractability and good generalization performance in many applications. The properties and capabilities of variational Bayesian learning itself have not been clarified yet. It is still unknown how good approximation the variational Bayesian approach can achieve. In this paper, we discuss variational Bayesian learning of Gaussian mixture models and derive upper and lower bounds of variational stochastic complexities. The variational stochastic complexity, which corresponds to the minimum variational free energy and a lower bound of the Bayesian evidence, not only becomes important in addressing the model selection problem, but also enables us to discuss the accuracy of the variational Bayesian approach as an approximation of true Bayesian learning.

Variational Bayesian learning was proposed as the approximation method of Bayesian learning. Inspite of efficiency and experimental good performance, their mathematical property has not yet been clarified. In this paper we analyze variational Bayesian hidden Markov models which include the true one thus the models are non-identifiable. We derive their asymptotic stochastic complexity. It is shown that, in some prior condition, the stochastic complexity is much smaller than those of identifiable models.

The Variational Bayesian framework has been widely used to approximate the Bayesian learning. In various applications, it has provided computational tractability and good generalization performance. In this paper, we discuss the Variational Bayesian learning of the mixture of exponential families and provide some additional theoretical support by deriving the asymptotic form of the stochastic complexity. The stochastic complexity, which corresponds to the minimum free energy and a lower bound of the marginal likelihood, is a key quantity for model selection. It also enables us to discuss the effect of hyperparameters and the accuracy of the Variational Bayesian approach as an approximation of the true Bayesian learning.

It is well known that Boltzmann machines are nonregular statistical models. The set of their parameters for a small size model is an analytic set with singularities in the space of a large size one. The mathematical foundation of their learning is not yet constructed because of these singularities, though they are applied to information engineering. Recently we established a method to calculate the Bayes generalization errors using an algebraic geometric method even if the models are nonregular. This paper clarifies that the upper bounds of generalization errors in Boltzmann machines are smaller than those in regular statistical models.

Hierarchical learning machines such as layered neural networks have singularities in their parameter spaces. At singularities, the Fisher information matrix becomes degenerate, with the result that the conventional learning theory of regular statistical models does not hold. Recently, it was proved that if the parameter of the true distribution is contained in the singularities of the learning machine, the generalization error in Bayes estimation is asymptotically equal toλ/n, where 2λ is smaller than the dimension of the parameter andn is the number of training samples. However, the constantλ strongly depends on the local geometrical structure of singularities; hence, the generalization error is not yet clarified when the true distribution is almost but not completely contained in the singularities. In this article, in order to analyze such cases, we study the Bayes generalization error under the condition that the Kullback distance of the true distribution from the distribution represented by singularities is in proportion to 1/n and show two results. First, if the dimension of the parameter from inputs to hidden units is not larger than three, then there exists a region of true parameters such that the generalization error is larger than that of the corresponding regular model. Second, if the dimension from inputs to hidden units is larger than three, then for arbitrary true distribution, the generalization error is smaller than that of the corresponding regular model.

A lot of learning machines with hidden variables used in infor- mation science have singularities in their parameter spaces. At singularities, the Fisher information matrix becomes degenerate, resulting that the learning theory of regular statistical models does not hold. Recently, it was proven that, if the true parameter is contained in singularities, then the coe(cid:14)cient of the Bayes gen- eralization error is equal to the pole of the zeta function of the Kullback information. In this paper, under the condition that the true parameter is almost but not contained in singularities, we show two results. (1) If the dimension of the parameter from in- puts to hidden units is not larger than three, then there exits a region of true parameters where the generalization error is larger than those of regular models, however, if otherwise, then for any true parameter, the generalization error is smaller than those of regular models. (2) The symmetry of the generalization error and the training error does not hold in singular models in general.

Prospective authors are requested to submit new, unpublished manuscripts for inclusion in the upcoming event described in this call for papers.

This article clarifies the relation between the learning curve and the algebraic geometrical structure of an unidentifiable learning machine such as a multilayer neural network whose true parameter set is an analytic set with singular points. By using a concept in algebraic analysis, we rigorously prove that the Bayesian stochastic complexity or the free energy is asymptotically equal to lambda(1) log n - (m(1) - 1) log log n + constant, where n is the number of training samples and lambda(1) and m(1) are the rational number and the natural number, which are determined as the birational invariant values of the singularities in the parameter space. Also we show an algorithm to calculate lambda(1) and m(1) based on the resolution of singularities in algebraic geometry. In regular statistical models, 2lambda(1) is equal to the number of parameters and m(1) = 1, whereas in nonregular models, such as multilayer networks, 2lambda(1) is not larger than the number of parameters and m(1) > or = 1. Since the increase of the stochastic complexity is equal to the learning curve or the generalization error, the nonidentifiable learning machines are better models than the regular ones if Bayesian ensemble learning is applied.

This paper proves that the Bayesian stochastic complexity of a layered neural network is asymptotically smaller than that of a regular statistical model if it contains the true distribution. We consider a case when a three-layer perceptron with M input units, H hidden units and N output units is trained to estimate the true distribution represented by the model with H(0) hidden units and prove that the stochastic complexity is asymptotically smaller than (1/2) {H(0) (M+N)+R} log n where n is the number of training samples and R is a function of H-H(0), M, and N that is far smaller than the number of redundant parameters. Since the generalization error of Bayesian estimation is equal to the increase of stochastic complexity, it is smaller than (1/2n) {H(0) (M+N)+R} if it has an asymptotic expansion. Based on the results, the difference between layered neural networks and regular statistical models is discussed from the statistical point of view.

Algebraic geometry is essential to learning theory. In hierarchical learning machines such as layered neural networks and gaussian mixtures, the asymptotic normality does not hold, since Fisher in(cid:173) formation matrices are singular. In this paper , the rigorous asymp(cid:173) totic form of the stochastic complexity is clarified based on resolu(cid:173) tion of singularities and two different problems are studied. (1) If the prior is positive, then the stochastic complexity is far smaller than BIO, resulting in the smaller generalization error than regular statistical models, even when the true distribution is not contained in the parametric model. nate free and equal to zero at singularities, is employed then the stochastic complexity has the same form as BIO. It is useful for model selection, but not for generalization. (2) If Jeffreys' prior, which is coordi(cid:173)

Proposes three ways of designing artificial neural networks based on a unified framework and uses them to develop new models. First, the authors show that artificial neural networks can be understood as probability density functions with parameters. Second, the authors propose three design methods for new models: a method for estimating the occurrence probability of the inputs, a method for estimating the variance of the outputs, and a method for estimating the simultaneous probability of inputs and outputs. Third, the authors design three new models using the proposed methods: a neural network with occurrence probability estimation, a neural network with output variance estimation, and a probability competition neural network. The authors' experimental results show that the proposed neural networks have important abilities in information processing; they can tell how often a given input occurs, how widely the outputs are distributed, and from what kinds of inputs a given output is inferred.

By combining ultrasonic imaging with neural networks, the authors have developed a 3-D object recognition system for use in robotic vision. Ultrasonic imaging is used to calculate the initial 3-D images of the objects. These images are then passed to neural networks that identify object categories, estimate object locations, and improve 3-D images. The authors explain the 3-D ultrasonic imaging method, propose three neural network structures for 3-D image analysis, and demonstrate the practicability of the system through experimental results, which show that quick and accurate recognition can be achieved using only a small set of transducers.>