This paper proposes a time-series data embedding technique that preserves curvature and orientation, with a focus on visualizing temporal manifold-valued data. Manifold-valued data provide pair-wise local distances on which the proposed method is built. First, we introduce a simpler form of our method, conformal folding embedding (CFE), as an interpretable straightforward algorithm to perfectly preserve the angles between adjacent velocity vectors, maintaining local geometric structure while forgoing global structure. Then we introduce the general formulation, dubbed local distance correlation embedding (LDCE), that maximizes distance correlation between input manifoldvalued data and the embedded ones while preserving both global distance structure and local geometric structure. Although the two algorithms are different in formulation, we show their theoretical connection by proving that CFE is a special case of LDCE. We empirically showcase the effectiveness of LDCE in preserving curvature/orientation by visualizing simulated data. The method is also applied to analyze the temporal information encoded at a population level in the inferior temporal cortex of monkeys.

This letter proposes an extension of principal component analysis for gaussian process (GP) posteriors, denoted by GP-PCA. Since GP-PCA estimates a low-dimensional space of GP posteriors, it can be used for metalearning, a framework for improving the performance of target tasks by estimating a structure of a set of tasks. The issue is how to define a structure of a set of GPs with an infinite-dimensional parameter, such as coordinate system and a divergence. In this study, we reduce the infiniteness of GP to the finite-dimensional case under the information geometrical framework by considering a space of GP posteriors that have the same prior. In addition, we propose an approximation method of GP-PCA based on variational inference and demonstrate the effectiveness of GP-PCA as meta-learning through experiments.

The full-span log-linear (FSLL) model introduced in this letter is considered an nth order Boltzmann machine, where n is the number of all variables in the target system. Let X=(X0,…,Xn-1) be finite discrete random variables that can take |X|=|X0|…|Xn-1| different values. The FSLL model has |X|-1 parameters and can represent arbitrary positive distributions of X. The FSLL model is a highest-order Boltzmann machine; nevertheless, we can compute the dual parameter of the model distribution, which plays important roles in exponential families in O(|X|log|X|) time. Furthermore, using properties of the dual parameters of the FSLL model, we can construct an efficient learning algorithm. The FSLL model is limited to small probabilistic models up to |X|≈225; however, in this problem domain, the FSLL model flexibly fits various true distributions underlying the training data without any hyperparameter tuning. The experiments showed that the FSLL successfully learned six training data sets such that |X|=220 within 1 minute with a laptop PC.

A large number of neurons form cell assemblies that process information in the brain. Recent developments in measurement technology, one of which is calcium imaging, have made it possible to study cell assemblies. In this study, we aim to extract cell assemblies from calcium imaging data. We propose a clustering approach based on non-negative matrix factorization (NMF). The proposed approach first obtains a similarity matrix between neurons by NMF and then performs spectral clustering on it. The application of NMF entails the problem of model selection. The number of bases in NMF affects the result considerably, and a suitable selection method is yet to be established. We attempt to resolve this problem by model averaging with a newly defined estimator based on NMF. Experiments on simulated data suggest that the proposed approach is superior to conventional correlation-based clustering methods over a wide range of sampling rates. We also analyzed calcium imaging data of sleeping/waking mice and the results suggest that the size of the cell assembly depends on the degree and spatial extent of slow wave generation in the cerebral cortex.

Gaze prediction is a significant approach for processing a large amount of incoming visual information of videos. Recent gaze prediction algorithms often employ sparse models with the assumption that every superpixel in the video frames can be represented as linear combinations of a few salient superpixels. However, they are not actuated enough because of the insufficient knowledge that video signals contain a non-negative request. Hence, we develop a novel gaze prediction based on an inverse sparse coding framework with a determinant sparse measure. By introducing this sparse measure, the solutions are non-negative and sparser than conventional sparse constraints. However, the proposed optimization problem becomes nonconvex, which is difficult to solve. To efficiently address the corresponding nonconvex optimization problem, we propose a novel algorithm based on the difference in convex function programming, which can yield the global solutions. Experimental results indicate the improved accuracy of the proposed approach compared with state-of-the-art algorithms.

The Fisher information matrix (FIM) plays an essential role in statistics and machine learning as a Riemannian metric tensor or a component of the Hessian matrix of loss functions. Focusing on the FIM and its variants in deep neural networks (DNNs), we reveal their characteristic scale dependence on the network width, depth, and sample size when the network has random weights and is sufficiently wide. This study covers two widely used FIMs for regression with linear output and for classification with softmax output. Both FIMs asymptotically show pathological eigenvalue spectra in the sense that a small number of eigenvalues become large outliers depending on the width or sample size, while the others are much smaller. It implies that the local shape of the parameter space or loss landscape is very sharp in a few specific directions while almost flat in the other directions. In particular, the softmax output disperses the outliers and makes a tail of the eigenvalue density spread from the bulk. We also show that pathological spectra appear in other variants of FIMs: one is the neural tangent kernel; another is a metric for the input signal and feature space that arises from feedforward signal propagation. Thus, we provide a unified perspective on the FIM and its variants that will lead to more quantitative understanding of learning in large-scale DNNs.

The Fisher information matrix (FIM) is a fundamental quantity to represent the characteristics of a stochastic model, including deep neural networks (DNNs). The present study reveals novel statistics of FIM that are universal among a wide class of DNNs. To this end, we use random weights and large width limits, which enables us to utilize mean field theories. We investigate the asymptotic statistics of the FIM’s eigenvalues and reveal that most of them are close to zero while the maximum eigenvalue takes a huge value. Because the landscape of the parameter space is defined by the FIM, it is locally flat in most dimensions, but strongly distorted in others. Moreover, we demonstrate the potential usage of the derived statistics in learning strategies. First, small eigenvalues that induce flatness can be connected to a norm-based capacity measure of generalization ability. Second, the maximum eigenvalue that induces the distortion enables us to quantitatively estimate an appropriately sized learning rate for gradient methods to converge.

Normalization methods play an important role in enhancing the performance of deep learning while their theoretical understandings have been limited. To theoretically elucidate the effectiveness of normalization, we quantify the geometry of the parameter space determined by the Fisher information matrix (FIM), which also corresponds to the local shape of the loss landscape under certain conditions. We analyze deep neural networks with random initialization, which is known to suffer from a pathologically sharp shape of the landscape when the network becomes sufficiently wide. We reveal that batch normalization in the last layer contributes to drastically decreasing such pathological sharpness if the width and sample number satisfy a specific condition. In contrast, it is hard for batch normalization in the middle hidden layers to alleviate pathological sharpness in many settings. We also found that layer normalization cannot alleviate pathological sharpness either. Thus, we can conclude that batch normalization in the last layer significantly contributes to decreasing the sharpness induced by the FIM.

The goal of fairness-aware classification is to categorize data while taking into account potential issues of fairness, discrimination, neutrality, and/or independence. For example, when applying data mining technologies to university admissions, admission criteria must be non-discriminatory and fair with regard to sensitive features, such as gender or race. In this context, such fairness can be formalized as statistical independence between classification results and sensitive features. The main purpose of this paper is to analyze this formal fairness in order to achieve better trade-offs between fairness and prediction accuracy, which is important for applying fairness-aware classifiers in practical use. We focus on a fairness-aware classifier, Calders and Verwer’s two-naive-Bayes ( CV2NB ) method, which has been shown to be superior to other classifiers in terms of fairness. We hypothesize that this superiority is due to the difference in types of independence. That is, because CV2NB achieves actual independence, rather than satisfying model-based independence like the other classifiers, it can account for model bias and a deterministic decision rule. We empirically validate this hypothesis by modifying two fairness-aware classifiers, a prejudice remover method and a reject option-based classification ( ROC ) method, so as to satisfy actual independence. The fairness of these two modified methods was drastically improved, showing the importance of maintaining actual independence, rather than model-based independence. We additionally extend an approach adopted in the ROC method so as to make it applicable to classifiers other than those with generative models, such as SVMs.

We propose a method for intrinsic dimension estimation. By fitting the power of distance from an inspection point and the number of samples included inside a ball with a radius equal to the distance, to a regression model, we estimate the goodness of fit. Then, by using the maximum likelihood method, we estimate the local intrinsic dimension around the inspection point. The proposed method is shown to be comparable to conventional methods in global intrinsic dimension estimation experiments. Furthermore, we experimentally show that the proposed method outperforms a conventional local dimension estimation method.

This study considers the common situation in data analysis when there are few observations of the distribution of interest or the target distribution, while abundant observations are available from auxiliary distributions. In this situation, it is natural to compensate for the lack of data from the target distribution by using data sets from these auxiliary distributions-in other words, approximating the target distribution in a subspace spanned by a set of auxiliary distributions. Mixture modeling is one of the simplest ways to integrate information from the target and auxiliary distributions in order to express the target distribution as accurately as possible. There are two typical mixtures in the context of information geometry: the [Formula: see text]- and [Formula: see text]-mixtures. The [Formula: see text]-mixture is applied in a variety of research fields because of the presence of the well-known expectation-maximazation algorithm for parameter estimation, whereas the [Formula: see text]-mixture is rarely used because of its difficulty of estimation, particularly for nonparametric models. The [Formula: see text]-mixture, however, is a well-tempered distribution that satisfies the principle of maximum entropy. To model a target distribution with scarce observations accurately, this letter proposes a novel framework for a nonparametric modeling of the [Formula: see text]-mixture and a geometrically inspired estimation algorithm. As numerical examples of the proposed framework, a transfer learning setup is considered. The experimental results show that this framework works well for three types of synthetic data sets, as well as an EEG real-world data set.

A graph is a mathematical representation of a set of variables where some pairs of the variables are connected by edges. Common examples of graphs are railroads, the Internet, and neural networks. It is both theoretically and practically important to estimate the intensity of direct connections between variables. In this study, a problem of estimating the intrinsic graph structure from observed data is considered. The observed data in this study are a matrix with elements representing dependency between nodes in the graph. The dependency represents more than direct connections because it includes influences of various paths. For example, each element of the observed matrix represents a co-occurrence of events at two nodes or a correlation of variables corresponding to two nodes. In this setting, spurious correlations make the estimation of direct connection difficult. To alleviate this difficulty, a digraph Laplacian is used for characterizing a graph. A generative model of this observed matrix is proposed, and a parameter estimation algorithm for the model is also introduced. The notable advantage of the proposed method is its ability to deal with directed graphs, while conventional graph structure estimation methods such as covariance selections are applicable only to undirected graphs. The algorithm is experimentally shown to be able to identify the intrinsic graph structure.

In the independent component analysis, polynomial functions of higher order statistics are often used as cost functions. However, such cost functions usually have many local minima, hence gradient-type and fixed-point-type algorithms tend to be trapped into a nonglobal local minimum. Recently, the polynomial optimization method that guarantees global convergence has been developed, where the optimization problem is relaxed as a semidefinite programming problem. In this paper, we apply the polynomial optimization method to the independent component analysis, and show the global convergence property. From some empirical studies, we further give a conjecture that the algorithm has polynomial time computational complexity.

For self-calibration of general radially symmetric distortion (RSD) of omni directional cameras such as fish-eye lenses, calibration parameters are usually estimated so that curved lines, which are supposed to be straight in the real-world, are mapped to straight lines in the calibrated image, which is assumed to be taken by an ideal pin-hole camera. In this paper, a method of calibrating RSD is introduced base on the notion of principal component analysis (PCA). In the proposed method, the distortion function, which maps a distorted image to an ideal pin-hole camera image, is assumed to be a linear combination of a certain class of basis functions, and an algorithm for solving its coefficients by using line patterns is given. Then a method of selecting good basis functions is proposed, which aims to realize appropriate calibration in practice. Experimental results for synthetic data and real images are presented to emonstrate the performance of our calibration method.

A recommender system has to collect users' preference data. To collect such data, rating or scoring methods that use rating scales, such as good-fair-poor or a five-point-scale, have been employed. We replaced such collection methods with a ranking method, in which objects are sorted according to the degree of a user's preference. We developed a technique to convert the rankings to scores based on order statistics theory. This technique successfully improved the accuracy of ranking recommended items. However, we targeted only memory-based recommendation algorithms. To test whether or not the use of ranking methods and our conversion technique are effective for wide variety of recommenders, we apply our conversion technique to model-based algorithms.

The aim of transfer learning is to improve prediction accuracy on a target task by exploiting the training examples for tasks that are related to the target one. Transfer learning has received more attention in recent years, because this technique is considered to be helpful in reducing the cost of labeling. In this paper, we propose a very simple approach to transfer learning: TrBagg, which is the extension of bagging. TrBagg is composed of two stages: Many weak classifiers are first generated as in standard bagging, and these classifiers are then filtered based on their usefulness for the target task. This simplicity makes it easy to work reasonably well without severe tuning of learning parameters. Further, our algorithm equips an algorithmic scheme to avoid negative transfer. We applied TrBagg to personalized tag prediction tasks for social bookmarks. Our approach has several convenient characteristics for this task such as adaptation to multiple tasks with low computational cost.

Exponential principal component analysis (e-PCA) has been proposed to reduce the dimension of the parameters of probability distributions using Kullback information as a distance between two distributions. It also provides a framework for dealing with various data types such as binary and integer for which the Gaussian assumption on the data distribution is inappropriate. In this paper, we introduce a latent variable model for the e-PCA. Assuming the discrete distribution on the latent variable leads to mixture models with constraint on their parameters. This provides a framework for clustering on the lower dimensional subspace of exponential family distributions. We derive a learning algorithm for those mixture models based on the variational Bayes (VB) method. Although intractable integration is required to implement the algorithm for a subspace, an approximation technique using Laplace's method allows us to carry out clustering on an arbitrary subspace. Combined with the estimation of the subspace, the resulting algorithm performs simultaneous dimensionality reduction and clustering. Numerical experiments on synthetic and real data demonstrate its effectiveness for extracting the structures of data as a visualization technique and its high generalization ability as a density estimation model.

We address the problem of estimating the structure of networks described as a system of differential equations. In each experiment, the network's steady state is measured as an output depending on a controllable input. Due to the high cost of experiments, it is crucial to actively design the inputs for accurate estimation. Although standard active learning methods are designed to minimize the entropy of parameter distributions, it is very unstable to estimate the entropy of network structure. Therefore, we propose the two step algorithm as follows: first, the most uncertain link is chosen, and then the input is designed so as to minimize the variance of system equation parameter instead of network structure. Our method is tested in simulation experiments of gene networks following Yeung et al., PNAS (2002). We show that our algorithm gives stable and computationally effective solution.

Usability is a vital quality factor of electronic environments. Increasing complexity of Web based platforms and mining large data volumes constantly rise demands on effective usability analysis tools. The article presents a novel formalism allowing examination of usability characteristics with the spectrum of metrics. The approach was applied to the usability analysis of a large corporate intranet. Majority of the users were knowledge workers. Important usability features have been revealed. The knowledge workers efficiently utilized only a minor portion of available intranet resources and exhibited strong pattern formation tendency. The frequently repeating browsing patterns were generally easily executable

We investigate the use of the Riemannian optimization method over the flag manifold in subspace ICA problems such as independent subspace analysis (ISA) and complex ICA. In the ISA experiment, we use the Riemannian approach over the flag manifold together with an MCMC method to overcome the problem of local minima of the ISA cost function. Experiments demonstrate the effectiveness of both Riemannian methods - simple geodesic gradient descent and hybrid geodesic gradient descent, compared with the ordinary gradient method.

Ordered lists of objects are widely used as representational forms. Such ordered objects include Web search results and best-seller lists. Techniques for processing such ordinal data are being developed, particularly methods for a supervised ordering task: i.e., learning functions used to sort objects from sample orders. In this article, we propose two dimension reduction methods specifically designed to improve prediction performance in a supervised ordering task.

Ordered lists of objects are widely used as representational forms. Such ordered objects include Web search results or bestseller lists. In spite of their importance, methods of processing orders have received little attention. However, research concerning orders has become common; in particular, researchers have developed various methods for the task of supervised ordering to acquire functions for object sorting from example orders. Here, we give a unified view of these methods and our new one, and empirically survey their merits and demerits.

Filling-in techniques are important, since missing values frequently appear in real data. Such techniques have been established for categorical or numerical values. Though lists of ordered objects are widely used as representational forms (e.g., Web search results, best-seller lists), filling-in techniques for orders have received little attention. We therefore propose a simple but effective technique to fill-in missing objects in orders. We built this technique into our collaborative filtering system.

We propose a method for extracting a low dimensional structure from a set of parameters of probability distributions. By an information geometrical interpretation, we show that there exist two kinds of possible flat structures for fitting (e-PCA and m-PCA). We derive alternating procedures to find the low dimensional structures. Each alternating procedure can be written in a nonlinear equation. It can be solved analytically in some special cases. Otherwise, we need to apply gradient type methods that we also derive. Since the overall algorithm may converge to a local optimum, we propose a method to find a good initial solution by using metric information.

This letter analyzes the Fisher kernel from a statistical point of view. The Fisher kernel is a particularly interesting method for constructing a model of the posterior probability that makes intelligent use of unlabeled data (i.e., of the underlying data density). It is important to analyze and ultimately understand the statistical properties of the Fisher kernel. To this end, we first establish sufficient conditions that the constructed posterior model is realizable (i.e., it contains the true distribution). Realizability immediately leads to consistency results. Subsequently, we focus on an asymptotic analysis of the generalization error, which elucidates the learning curves of the Fisher kernel and how unlabeled data contribute to learning. We also point out that the squared or log loss is theoretically preferable-because both yield consistent estimators-to other losses such as the exponential loss, when a linear classifier is used together with the Fisher kernel. Therefore, this letter underlines that the Fisher kernel should be viewed not as a heuristics but as a powerful statistical tool with well-controlled statistical properties.

We advocate a new learning task that deals with orders of items, and we call this the learning from order examples (LOE) task. The aim of the task is to acquire the rule that is used for estimating the proper order of a given unordered item set. The rule is acquired from training examples that are ordered item sets. We present several solution methods for this task, and evaluate the performance and the characteristics of these methods based on the experimental results of tests using both artificial data and realistic data.

We show the existence of critical points as lines for the likelihood func- tion of mixture-type models. They are given by embedding of a critical point for models with less components. A sufficient condition that the critical line gives local maxima or saddle points is also derived. Based on this fact, a component-split method is proposed for a mixture of Gaus- sian components, and its effectiveness is verified through experiments.

In this paper, we construct a learning method of stochastic perceptron based on semiparametric inference, and show that this method produces large margin solutions. In semiparametric inference, the parameters are divided into structural parameters which are to be estimated and nuisance parameters in which we do not have any interest. Here, the weight vector of perceptron is defined as structural parameters and the steepness of transfer function is defined as a nuisance parameter. Usually, rough estimate is substituted to nuisance parameters and only structural parameters are estimated. To compensate the estimation error caused by rough estimate, an additional term is added to the derivative of likelihood. We will show that this additional term is related to the regularization term which causes large margin solutions. This work suggests that the success of large margin classifiers can be attributed to semiparametric inference.

A human continuously directs his eyes toward interesting points to recognize an object. Such eye movement or visual attention plays an important role in human vision. To simulate such eye movements, the paper proposes a virtual active vision head with binocular fish-eye lenses. Moving objects are detected within the wide angle of view obtained through the fish-eye lens by comparing the input image with the background image estimated by an adaptive M-estimator. The location of the moving object is used as a visual cue to achieve the saccadic eye movement. The proposed virtual active vision system was implemented on a PC cluster. Two PCs are used to capture images through the left and right fish-eye lenses and a PC with four processors is used for image processing. Real time gaze control is achieved on the proposed virtual binocular active vision system.

Theories of learning and generalization hold that the generalization bias, defined as the difference between the training error and the generalization error, increases on average with the number of adaptive parameters. This article, however, shows that this general tendency is violated for a gaussian mixture model. For temperatures just below the first symmetry breaking point, the effective number of adaptive parameters increases and the generalization bias decreases. We compute the dependence of the neural information criterion on temperature around the symmetry breaking. Our results are confirmed by numerical cross-validation experiments.

We propose MICA (multimodal independent component analysis) that extends ICA (independent component analysis) to the case that there is a pair of information sources. MICA extracts statistically dependent pairs of features from the sources, where the components of feature vector extracted from each source are independent. Therefore, the cost function is constructed to optimize this degree of pairwise dependence as well as optimizing the cost function of ICA. We approximate the cost function by two dimensional Gram-Charlier expansion and propose a gradient descent algorithm derived by Amari's natural gradient The relation between MICA and traditional CCA (canonical correlation analysis) is similar to the relation between ICA and PCA (principal component analysis).

A design of mobile robot for robust life-long navigation in office environment is proposed and evaluated. The key idea is combining probabilistic map and dialog with humans for reducing the location uncertainty. Bayesian inference with the map represented by probabilistic automata is used in order to reduce the number of queries and to evaluate the success rate of planned paths. We experimentally implemented the design using a simple Bayesian network with continuous nodes and demonstrated its effectiveness in a real environment.

A new lower bound on the capacity of three-layer neural networks is given by a constructive method which uses the sparse encoding scheme. The capacity is closely related to the learning ability from examples, giving an estimate of the number of examples to be required for learning a function. It is shown that the capacitymof a neural network withhhidden elements andninput lines satisfies (in the sense of order)nh/log (nh2)mnhlogh