The dynamics of supervised learning play a main role in deep learning, which takes place in the parameter space of a multilayer perceptron (MLP). We review the history of supervised stochastic gradient learning, focusing on its singular structure and natural gradient. The parameter space includes singular regions in which parameters are not identifiable. One of our results is a full exploration of the dynamical behaviors of stochastic gradient learning in an elementary singular network. The bad news is its pathological nature, in which part of the singular region becomes an attractor and another part a repulser at the same time, forming a Milnor attractor. A learning trajectory is attracted by the attractor region, staying in it for a long time, before it escapes the singular region through the repulser region. This is typical of plateau phenomena in learning. We demonstrate the strange topology of a singular region by introducing blow-down coordinates, which are useful for analyzing the natural gradient dynamics. We confirm that the natural gradient dynamics are free of critical slowdown. The second main result is the good news: the interactions of elementary singular networks eliminate the attractor part and the Milnor-type attractors disappear. This explains why large-scale networks do not suffer from serious critical slowdowns due to singularities. We finally show that the unit-wise natural gradient is effective for learning in spite of its low computational cost.

The importance of researches in the old days is the retrieval of similar objects. Content-Based Image Retrieval (CBIR) system uses low-level features such as texture, color, and shape to extract the features, but when it comes to food images it is hard to get satisfactory accurate results. Recognition of food images has recently become very important and challenging due to people's health care, religious or cultural reasons. In this paper, we propose a system that recognizes small food images consist of 10 categories by using bag of features (BoF) based on SURF detection features. In addition, we achieved up to 78% accuracy rate, and try to improve the feature detection by using color features at the same time with the SURF feature detection. This experiment shows that more accurate rate of results will be obtained than the existing methods.

Due to food culture, religion, allergy and food intolerance we have to find a good system to help us recognize our food. In this paper, we propose methods to recognize food and to show the ingredients using a bag-of-features (BoF) based on SURF detection features. We also propose bag of SURF features and bag-of HOG Features at the same time with the SURF feature detection to recognize the food items. In the experiment, we have achieved up to 72% of accuracy rate on a small food image dataset of 10 categories. Our experiments show that the proposed representation is significantly accurate at identifying food in the existing methods. Moreover, the enhancement of the visual dataset with more images will improve the accuracy rates, especially for the classes with high diversity.

Pedestrian detection is used in video surveillance systems and driver assistance systems. The purpose is to build automated vision systems for detecting pedestrians as shown in figure 1. We use Histograms of Oriented Gradients (HOG), which are one of the well-known features for object recognition. HOG features are calculated by taking orientation histograms of edge intensity in a local region [1]. In this paper we select the interesting point in the image by using FAST features detector and extracted HOG features around these strongest corners and use them as an input vector of linear Support Vector Machine (SVM) to classify the given input into pedestrian/non-pedestrian. By using FAST detector we reduce the number of features less than half without lowering the performance.

Reinforcement learning is one of research fields in artificial intelligence. The learning method usually assumes a discrete state in computer simulations. However, we must treat a continuous value in a realistic situation. In this paper, we investigate various techniques of the tile cording scheme which is a representative technique to handle continuous states. We check the performance of single tiling, multiple tiling, time-shift method and the proposed method in the issue of space search.

Face recognition is used to identify humans by their face image. Recently it becomes most common application in information security. Bag of features has been successfully applied in face recognition. In our research we use SURF features and try to improve it by using block-based bag of feature models. In this method we partition the image into multiple blocks and we extract SURF features densely on each block. We compare the performance of the original bag of feature model with Grid/Detector method and bag of block-based feature model.

We explicitly analyze the trajectories of learning near singularities in hierarchical networks, such as multilayer perceptrons and radial basis function networks, which include permutation symmetry of hidden nodes, and show their general properties. Such symmetry induces singularities in their parameter space, where the Fisher information matrix degenerates and odd learning behaviors, especially the existence of plateaus in gradient descent learning, arise due to the geometric structure of singularity. We plot dynamic vector fields to demonstrate the universal trajectories of learning near singularities. The singularity induces two types of plateaus, the on-singularity plateau and the near-singularity plateau, depending on the stability of the singularity and the initial parameters of learning. The results presented in this letter are universally applicable to a wide class of hierarchical models. Detailed stability analysis of the dynamics of learning in radial basis function networks and multilayer perceptrons will be presented in separate work.

The dynamical behavior of learning is known to be very slow for the multilayer perceptron, being often trapped in the "plateau." It has been recently understood that this is due to the singularity in the parameter space of perceptrons, in which trajectories of learning are drawn. The space is Riemannian from the point of view of information geometry and contains singular regions where the Riemannian metric or the Fisher information matrix degenerates. This paper analyzes the dynamics of learning in a neighborhood of the singular regions when the true teacher machine lies at the singularity. We give explicit asymptotic analytical solutions (trajectories) both for the standard gradient (SGD) and natural gradient (NGD) methods. It is clearly shown, in the case of the SGD method, that the plateau phenomenon appears in a neighborhood of the critical regions, where the dynamical behavior is extremely slow. The analysis of the NGD method is much more difficult, because the inverse of the Fisher information matrix diverges. We conquer the difficulty by introducing the "blow-down" technique used in algebraic geometry. The NGD method works efficiently, and the state converges directly to the true parameters very quickly while it staggers in the case of the SGD method. The analytical results are compared with computer simulations, showing good agreement. The effects of singularities on learning are thus qualitatively clarified for both standard and NGD methods.

The parameter spaces of hierarchical systems such as multilayer perceptrons include singularities due to the symmetry and degeneration of hidden units. A parameter space forms a geometrical manifold, called the neuromanifold in the case of neural networks. Such a model is identified with a statistical model, and a Riemannian metric is given by the Fisher information matrix. However, the matrix degenerates at singularities. Such a singular structure is ubiquitous not only in multilayer perceptrons but also in the gaussian mixture probability densities, ARMA time-series model, and many other cases. The standard statistical paradigm of the Cramér-Rao theorem does not hold, and the singularity gives rise to strange behaviors in parameter estimation, hypothesis testing, Bayesian inference, model selection, and in particular, the dynamics of learning from examples. Prevailing theories so far have not paid much attention to the problem caused by singularity, relying only on ordinary statistical theories developed for regular (nonsingular) models. Only recently have researchers remarked on the effects of singularity, and theories are now being developed. This article gives an overview of the phenomena caused by the singularities of statistical manifolds related to multilayer perceptrons and gaussian mixtures. We demonstrate our recent results on these problems. Simple toy models are also used to show explicit solutions. We explain that the maximum likelihood estimator is no longer subject to the gaussian distribution even asymptotically, because the Fisher information matrix degenerates, that the model selection criteria such as AIC, BIC, and MDL fail to hold in these models, that a smooth Bayesian prior becomes singular in such models, and that the trajectories of dynamics of learning are strongly affected by the singularity, causing plateaus or slow manifolds in the parameter space. The natural gradient method is shown to perform well because it takes the singular geometrical structure into account. The generalization error and the training error are studied in some examples.

Singularities are ubiquitous in the parameter space of hierarchical models such as multilayer perceptrons. At singularities, the Fisher information matrix degenerates, and the Cramer-Rao paradigm does no more hold, implying that the classical model selection the(cid:173) ory such as AIC and MDL cannot be applied. It is important to study the relation between the generalization error and the training error at singularities. The present paper demonstrates a method of analyzing these errors both for the maximum likelihood estima(cid:173) tor and the Bayesian predictive distribution in terms of Gaussian random fields, by using simple models.