Phase retrieval refers to the problem of recovering a high-dimensional vector ${\mathbf{x}} \in {\mathbb{C}^N}$ from the magnitude of its linear transform z = Ax, observed through a noisy channel. To improve the ill-posed nature of the inverse problem, it is a common practice to observe the magnitude of linear measurements z(1)= A(1)x,…, z(L)= A(L)x using multiple sensing matrices A(1),…, A(L), with ptychographic imaging being a remarkable example of such strategies. Inspired by existing algorithms for ptychographic reconstruction, we introduce stochasticity to Vector Approximate Message Passing (VAMP), a computationally efficient algorithm applicable to a wide range of Bayesian inverse problems. By testing our approach in the setup of phase retrieval, we show the superior convergence speed of the proposed algorithm.

We estimate the three Herschel-Bulkley parameters (yield stress σ Y , power-law index n , and consistency parameter η ) for shear-dependent fluid-like materials possibly with large-scale inclusions, for which rheometers may fail to provide a useful measurement. We perform experiments using the unknown material for dam-break (or column collapse) setups and capture video footage. We then use simulations to optimize for the material parameters. For better match up with the simple shear flow encountered in a rheometer, we modify the flow rule for the elasto-viscoplastic Herschel-Bulkley model. Analyzing the loss landscape for optimization, we realize a similarity relation ; material parameters far away within this relation would result in matched simulations, making it hard to distinguish the parameters. We found that by exploiting the setup dependency of the similarity relation, we can improve on the estimation using multiple setups, which we propose by analyzing the Hessian of the similarity relation. We validate the efficacy of our method by comparing the estimations to the measurements from a rheometer (for materials without large-scale inclusions) and show applications to materials with large-scale inclusions, including various salad or pasta sauces, and congee.

Adaptation and invariance to multiple environments are both crucial abilities for intelligent systems. Model-agnostic meta-learning (MAML) is a meta-learning algorithm to enable such adaptability, and invariant risk minimization (IRM) is a problem setting to achieve the invariant representation across multiple environments. We can formulate both methods as optimization problems with the environment-dependent constraint and this constraint is known to hamper optimization. Therefore, understanding the effect of the constraint on the optimization is important. In this paper, we provide a conceptual insight on how the constraint affects the optimization of MAML and IRM by analyzing the trainability of the gradient descent on the loss with the gradient norm penalty, which is easier to study but is related to both MAML and IRM. We conduct numerical experiments with practical datasets and architectures for MAML and IRM and validate that the analysis of the gradient norm penalty loss captures well the empirical relationship between the constraint and the trainability of MAML and IRM.

The plateau phenomenon, wherein the loss value stops decreasing during the process of learning, has been reported by various researchers. The phenomenon is actively inspected in the 1990s and found to be due to the fundamental hierarchical structure of neural network models. Then the phenomenon has been thought as inevitable. However, the phenomenon seldom occurs in the context of recent deep learning. There is a gap between theory and reality. In this paper, using statistical mechanical formulation, we clarified the relationship between the plateau phenomenon and the statistical property of the data learned. It is shown that the data whose covariance has small and dispersed eigenvalues tend to make the plateau phenomenon inconspicuous.

The materials around us usually exist as mixtures of constituents, each constituent with possibly a different elasto-viscoplastic property. How can we describe the material property of such a mixture is the core question of this paper. We propose a nonlinear blending model that can capture intriguing flowing behaviors that can differ from that of the individual constituents (Fig. 1). We used a laboratory device, rheometer , to measure the flowing properties of various fluid-like foods, and found that an elastic Herschel-Bulkley model has nice agreements with the measured data even for the mixtures of these foods. We then constructed a blending model such that it qualitatively agrees with the measurements and is closed in the parameter space of the elastic Herschel-Bulkley model. We provide validations through comparisons between the measured and estimated properties using our model, and comparisons between simulated examples and captured footages. We show the utility of our model for producing interesting behaviors of various mixtures.

The dynamics of supervised learning play a main role in deep learning, which takes place in the parameter space of a multilayer perceptron (MLP). We review the history of supervised stochastic gradient learning, focusing on its singular structure and natural gradient. The parameter space includes singular regions in which parameters are not identifiable. One of our results is a full exploration of the dynamical behaviors of stochastic gradient learning in an elementary singular network. The bad news is its pathological nature, in which part of the singular region becomes an attractor and another part a repulser at the same time, forming a Milnor attractor. A learning trajectory is attracted by the attractor region, staying in it for a long time, before it escapes the singular region through the repulser region. This is typical of plateau phenomena in learning. We demonstrate the strange topology of a singular region by introducing blow-down coordinates, which are useful for analyzing the natural gradient dynamics. We confirm that the natural gradient dynamics are free of critical slowdown. The second main result is the good news: the interactions of elementary singular networks eliminate the attractor part and the Milnor-type attractors disappear. This explains why large-scale networks do not suffer from serious critical slowdowns due to singularities. We finally show that the unit-wise natural gradient is effective for learning in spite of its low computational cost.

Machine learning has been often used for drug discovery in recent years. Inductive logic programming (ILP), that can express common features of each data in a qualitative, is one of the machine learning methods. The advantage of ILP is that the classification model is clear compared with other machine learning methods such as Support Vector Machine (SVM) because ILP classifies inhibitors using generated rules. ILP is allowed to learn structure of the compounds so that we can draw the common structure of the ligands from the generated rules. In this method, the data of ligands and decoys are collected from “A Database of Useful Decoys: Enhanced” (DUD-E). ILP provides classification model called a rule by learning these data. This study applies the ILP algorithm to the virtual screening of inhibitors of carbonic anhydrase II (CAH2). We demonstrate its performance by classifying ligands and decoys which aren't included in DUD-E. Our results show that ILP has the performance equivalent to SVM which is known for its high classification performance. In addition, this paper shows that ILP can derive rules explaining structural features of CAH2 ligands. Several of these rules are consistent with known property of CAH2 ligand. This paper demonstrate that ILP has high classification performance and clear classification model. Our method is useful for generating rules for ligand design.

We investigate the detection dynamics of a soft parallel interference canceler (soft-PIC), which includes a hard-PIC as a special case, for code-division multiple-access (CDMA) multiuser detection, applied to a randomly spread, fully synchronous base-band uncoded CDMA channel model with additive white Gaussian noise under perfect power control in the large-system limit. We analyze the detection dynamics of some iterative detectors, namely soft-PIC, the Onsager-reaction-canceling parallel interference canceler (ORC-PIC) and the belief-propagation-based detector (BP-based detector), by the generating functional analysis (GFA). The GFA allows us to study the asymptotic behavior of the dynamics in the infinitely large system without assuming the independence of messages. We study the detection dynamics and the stationary estimates of an iterative algorithm. We also show the decoupling principle in iterative multiuser detection algorithms in the large-system limit. For a generic iterative multiuser detection algorithm with binary input, it is shown that the multiuser channel is equivalent to a bank of independent single-user additive non-Gaussian channels, whose signal-to-noise ratio degrades due to both the multiple-access interference and the Onsager reaction, at each stage of the algorithm. If an algorithm cancels the Onsager reaction, the equivalent single-user channels coincide with an additive white Gaussian noise channel. We also discuss ORC-PIC and the BP-based detector.

The computational modelling of the primary auditory cortex (A1) has been less fruitful than that of the primary visual cortex (V1) due to the less organized properties of A1. Greater disorder has recently been demonstrated for the tonotopy of A1 that has traditionally been considered to be as ordered as the retinotopy of V1. This disorder appears to be incongruous, given the uniformity of the neocortex; however, we hypothesized that both A1 and V1 would adopt an efficient coding strategy and that the disorder in A1 reflects natural sound statistics. To provide a computational model of the tonotopic disorder in A1, we used a model that was originally proposed for the smooth V1 map. In contrast to natural images, natural sounds exhibit distant correlations, which were learned and reflected in the disordered map. The auditory model predicted harmonic relationships among neighbouring A1 cells; furthermore, the same mechanism used to model V1 complex cells reproduced nonlinear responses similar to the pitch selectivity. These results contribute to the understanding of the sensory cortices of different modalities in a novel and integrated manner.

The neural substrates of decision making have been intensively studied using experimental and computational approaches. Alternative-choice tasks accompanying reinforcement have often been employed in investigations into decision making. Choice behavior has been empirically found in many experiments to follow Herrnstein's matching law. A number of theoretical studies have been done on explaining the mechanisms responsible for matching behavior. Various learning rules have been proved in these studies to achieve matching behavior as a steady state of learning processes. The models in the studies have consisted of a few parameters. However, a large number of neurons and synapses are expected to participate in decision making in the brain. We investigated learning behavior in simple but large-scale decision-making networks. We considered the covariance learning rule, which has been demonstrated to achieve matching behavior as a steady state (Loewenstein & Seung, 2006 ). We analyzed model behavior in a thermodynamic limit where the number of plastic synapses went to infinity. By means of techniques of the statistical mechanics, we can derive deterministic differential equations in this limit for the order parameters, which allow an exact calculation of the evolution of choice behavior. As a result, we found that matching behavior cannot be a steady state of learning when the fluctuations in input from individual sensory neurons are so large that they affect the net input to value-encoding neurons. This situation naturally arises when the synaptic strength is sufficiently strong and the excitatory input and the inhibitory input to the value-encoding neurons are balanced. The deviation from matching behavior is caused by increasing variance in the input potential due to the diffusion of synaptic efficacies. This effect causes an undermatching phenomenon, which has been often observed in behavioral experiments.

Many redundancies play functional roles in motor control and motor learning. For example, kinematic and muscle redundancies contribute to stabilizing posture and impedance control, respectively. Another redundancy is the number of neurons themselves; there are overwhelmingly more neurons than muscles, and many combinations of neural activation can generate identical muscle activity. The functional roles of this neuronal redundancy remains unknown. Analysis of a redundant neural network model makes it possible to investigate these functional roles while varying the number of model neurons and holding constant the number of output units. Our analysis reveals that learning speed reaches its maximum value if and only if the model includes sufficient neuronal redundancy. This analytical result does not depend on whether the distribution of the preferred direction is uniform or a skewed bimodal, both of which have been reported in neurophysiological studies. Neuronal redundancy maximizes learning speed, even if the neural network model includes recurrent connections, a nonlinear activation function, or nonlinear muscle units. Furthermore, our results do not rely on the shape of the generalization function. The results of this study suggest that one of the functional roles of neuronal redundancy is to maximize learning speed.

We propose an algorithm for simultaneously estimating state transitions among neural states and nonstationary firing rates using a switching state-space model (SSSM). This algorithm enables us to detect state transitions on the basis of not only discontinuous changes in mean firing rates but also discontinuous changes in the temporal profiles of firing rates (e.g., temporal correlation). We construct estimation and learning algorithms for a nongaussian SSSM, whose nongaussian property is caused by binary spike events. Local variational methods can transform the binary observation process into a quadratic form. The transformed observation process enables us to construct a variational Bayes algorithm that can determine the number of neural states based on automatic relevance determination. Additionally, our algorithm can estimate model parameters from single-trial data using a priori knowledge about state transitions and firing rates. Synthetic data analysis reveals that our algorithm has higher performance for estimating nonstationary firing rates than previous methods. The analysis also confirms that our algorithm can detect state transitions on the basis of discontinuous changes in temporal correlation, which are transitions that previous hidden Markov models could not detect. We also analyze neural data recorded from the medial temporal area. The statistically detected neural states probably coincide with transient and sustained states that have been detected heuristically. Estimated parameters suggest that our algorithm detects the state transitions on the basis of discontinuous changes in the temporal correlation of firing rates. These results suggest that our algorithm is advantageous in real-data analysis.

When animals repeatedly choose actions from multiple alternatives, they can allocate their choices stochastically depending on past actions and outcomes. It is commonly assumed that this ability is achieved by modifications in synaptic weights related to decision making. Choice behavior has been empirically found to follow Herrnstein’s matching law. Loewenstein & Seung (2006) demonstrated that matching behavior is a steady state of learning in neural networks if the synaptic weights change proportionally to the covariance between reward and neural activities. However, their proof did not take into account the change in entire synaptic distributions. In this study, we show that matching behavior is not necessarily a steady state of the covariance-based learning rule when the synaptic strength is sufficiently strong so that the fluctuations in input from individual sensory neurons influence the net input to output neurons. This is caused by the increasing variance in the input potential due to the diffusion of synaptic weights. This effect causes an undermatching phenomenon, which has been observed in many behavioral experiments. We suggest that the synaptic diffusion effects provide a robust neural mechanism for stochastic choice behavior.

We propose an algorithm for simultaneously estimating state transitions among neural states, the number of neural states, and nonstationary firing rates using a switching state space model (SSSM). This model enables us to detect state transitions based not only on the discontinuous changes of mean firing rates but also on discontinuous changes in temporal profiles of firing rates, e.g., temporal correlation. We derive a variational Bayes algorithm for a non-Gaussian SSSM whose non-Gaussian property is caused by binary spike events. Synthetic data analysis reveals the high performance of our algorithm in estimating state transitions, the number of neural states, and nonstationary firing rates compared to previous methods. We also analyze neural data recorded from the medial temporal area. The statistically detected neural states probably coincide with transient and sustained states, which have been detected heuristically. Estimated parameters suggest that our algorithm detects the state transition based on discontinuous change in the temporal correlation of firing rates, which transitions previous methods cannot detect. This result suggests the advantage of our algorithm in real-data analysis.

Neural activity is nonstationary and varies across time. Hidden Markov models (HMMs) have been used to track the state transition among quasi-stationary discrete neural states. Within this context, an independent Poisson model has been used for the output distribution of HMMs; hence, the model is incapable of tracking the change in correlation without modulating the firing rate. To achieve this, we applied a multivariate Poisson distribution with correlation terms for the output distribution of HMMs. We formulated a variational Bayes (VB) inference for the model. The VB could automatically determine the appropriate number of hidden states and correlation types while avoiding the overlearning problem. We developed an efficient algorithm for computing posteriors using the recursive relationship of a multivariate Poisson distribution. We demonstrated the performance of our method on synthetic data and real spike trains recorded from a songbird.

Spatiotemporal context in sensory stimulus has profound effects on neural responses and perception, and it sometimes affects task difficulty. Recently reported experimental data suggest that human detection sensitivity to motion in a target stimulus can be enhanced by adding a slow surrounding motion in an orthogonal direction, even though the illusory motion component caused by the surround is not relevant to the task. It is not computationally clear how the task-irrelevant component of motion modulates the subject's sensitivity to motion detection. In this study, we investigated the effects of encoding biases on detection performance by modeling the stochastic neural population activities. We modeled two types of modulation on the population activity profiles caused by a contextual stimulus: one type is identical to the activity evoked by a physical change in the stimulus, and the other is expressed more simply in terms of response gain modulation. For both encoding schemes, the motion detection performance of the ideal observer is enhanced by a task-irrelevant, additive motion component, replicating the properties observed for real subjects. The success of these models suggests that human detection sensitivity can be characterized by a noisy neural encoding that limits the resolution of information transmission in the cortical visual processing pathway. On the other hand, analyses of the neuronal contributions to the task predict that the effective cell populations differ between the two encoding schemes, posing a question concerning the decoding schemes that the nervous system used during illusory states.

This paper proposes a novel region-based coupled Markov random field (MRF) model for coarse image region segmentation on silicon platforms. Coupled MRF models are classified into boundary- and region-based models, in which hidden variables are referred to as a line process and a label process, respectively. These hidden variables are crucial for detecting discontinuities in motion, intensity, color, and depth in visual scenes. For a coarse image region segmentation task, we address a region-based coupled MRF model with hidden phase variables. It is shown that the region-based coupled MRF model has an advantage over the resistive-fuse network, which is a boundary-based coupled MRF model, in dealing with the hidden variables explicitly. These models work complementarily for a coarse image region segmentation task. For real-time region segmentation operation, we have designed a merged analog/digital CMOS circuit implementing both functions of the boundary- and region-based coupled MRF models using a pulse modulation approach.

Exponential principal component analysis (e-PCA) has been proposed to reduce the dimension of the parameters of probability distributions using Kullback information as a distance between two distributions. It also provides a framework for dealing with various data types such as binary and integer for which the Gaussian assumption on the data distribution is inappropriate. In this paper, we introduce a latent variable model for the e-PCA. Assuming the discrete distribution on the latent variable leads to mixture models with constraint on their parameters. This provides a framework for clustering on the lower dimensional subspace of exponential family distributions. We derive a learning algorithm for those mixture models based on the variational Bayes (VB) method. Although intractable integration is required to implement the algorithm for a subspace, an approximation technique using Laplace's method allows us to carry out clustering on an arbitrary subspace. Combined with the estimation of the subspace, the resulting algorithm performs simultaneous dimensionality reduction and clustering. Numerical experiments on synthetic and real data demonstrate its effectiveness for extracting the structures of data as a visualization technique and its high generalization ability as a density estimation model.

A contour tree is a powerful tool for delineating the topological evolution of isosurfaces of a single-valued function, and thus has been frequently used as a means of extracting features from volumes and their time-varying behaviors. Several sophisticated algorithms have been proposed for constructing contour trees while they often complicate the software implementation especially for higher-dimensional cases such as time-varying volumes. This paper presents a simple yet effective approach to plotting in 3D space, approximate contour trees from a set of scattered samples embedded in the high-dimensional space. Our main idea is to take advantage of manifold learning so that we can elongate the distribution of high-dimensional data samples to embed it into a low-dimensional space while respecting its local proximity of sample points. The contribution of this paper lies in the introduction of new distance metrics to manifold learning, which allows us to reformulate existing algorithms as a variant of currently available dimensionality reduction scheme. Efficient reduction of data sizes together with segmentation capability is also developed to equip our approach with a coarse-to-fine analysis even for large-scale datasets. Examples are provided to demonstrate that our proposed scheme can successfully traverse the features of volumes and their temporal behaviors through the constructed contour trees.

Research field of my laboratory includes statistical mechanics especially for random magnetic systems, brain and information sciences. These three research areas share common mathematical structure. Their common feature is that they commonly treat very large freedom systems, that is, magnetism which originates from cooperative phenomena of O(1023) spins, brain functions carried by the large number of neuron activities, digital information represented by the large number of bit sequences. The statistical mechanics plays an important role to treat such large freedom systems because it can deduce macroscopic and low dimensional descriptions like the Boyle-Charle's law from microscopic and high dimensional descriptions of Newtonian or quantum mechanics. Since the statistical mechanics is considered as a systematic tool for the dimension reduction, it can be applicable to the brain and information systems. Since the dimension reduction is a common feature between the visualization and the statistical mechanics, the visualization also plays an important role to treat these systems. The visualization offers effective and essential low dimensional visual representations which help us to intuitively understand these systems. In this talk, I will present two examples regarding the visualization of high dimensional data in the brain and information sciences where the principal component analysis (PCA) and the mixture of Gaussian analysis (MGA) are employed. Sugase et al. performed single unit recordings in the inferior temporal (IT) cortex in the monkey's brain which is one of crucial areas regarding the pattern recognition. They presented 38 images to the monkey.

A lossy data compression scheme for uniformly biased Boolean messages is investigated via statistical mechanics techniques. The present paper utilize tree-like committee machine (committee tree) and tree-like parity machine (parity tree) whose transfer functions are non-monotonic, completing the study of the lossy compression scheme using perceptron-based decoder. The scheme performance at the infinite code length limit is analyzed using the replica method. Both committee and parity treelike networks are shown to saturate the Shannon bound.

Neural activity is non-stationary and varies across time. Hidden Markov Models (HMMs) have been used to track the state transition among quasi-stationary discrete neural states. Within this context, independent Poisson models have been used for the output distribution of HMMs; hence, the model is incapable of tracking the change in correlation without modulating the firing rate. To achieve this, we applied a multivariate Poisson distribution with correlation terms for the output distribution of HMMs. We formulated a Variational Bayes (VB) inference for the model. The VB could automatically determine the appropriate number of hidden states and correlation types while avoiding the overlearning problem. We developed an efficient algorithm for computing posteriors using the recursive relationship of a multivariate Poisson distribution. We demonstrated the performance of our method on synthetic data and a real spike train recorded from a songbird.

``How is information decoded in the brain?'' is one of the most difficult and important questions in neuroscience. Whether neural correlation is important or not in decoding neural activities is of special interest. We have developed a general framework for investigating how far the decoding process in the brain can be simplified. First, we hierarchically construct simplified probabilistic models of neural responses that ignore more than $K$th-order correlations by using a maximum entropy principle. Then, we compute how much information is lost when information is decoded using the simplified models, i.e., ``mismatched decoders''. We introduce an information theoretically correct quantity for evaluating the information obtained by mismatched decoders. We applied our proposed framework to spike data for vertebrate retina. We used 100-ms natural movies as stimuli and computed the information contained in neural activities about these movies. We found that the information loss is negligibly small in population activities of ganglion cells even if all orders of correlation are ignored in decoding. We also found that if we assume stationarity for long durations in the information analysis of dynamically changing stimuli like natural movies, pseudo correlations seem to carry a large portion of the information.

We propose a theoretical framework to investigate statistical performance of inverse digital-halftoning problems. In the context of the maximizer of the posterior marginal (MPM) estimate corresponding to the Markov random fields (MRFs) model in which each pixel takes discrete values such as 1, ..., Q, we formulate the problem of inverse digital-halftoning in which digital images are generated by the threshold constant and the so-called Bayers' matrices. To construct the Gibbs sampler for the MRFs, we carry out Markov chain Monte Carlo (MCMC) simulations and investigate hyper-parameter dependence of the performance in terms of the mean-square error. By using the statistical-mechanical analysis, we also investigate averaged case performance of the inverse-halftoning for the corresponding analytically tractable class of the MRFs models. Both equilibrium and dynamical properties of the MPM estimation of the original grayscale images are revealed.

We investigate detection dynamics of a soft parallel interference canceller (soft-PIC), which includes a hard PIC as a special case, for CDMA multiuser detection, applied to a randomly spread, fully synchronous base-band uncoded CDMA channel model with additive white Gaussian noise (AWGN) under perfect power control in the large-system limit. We analyze the detection dynamics of the soft-PIC and the belief-propagation-based detector by the generating functional analysis (GFA). The GFA allows us to study the dynamics of infinitely large disordered system in an exact way.

Repetitions of precise spike patterns observed both in vivo and in vitro have been reported for more than a decade. Studies on the spike volley (a pulse packet) propagating through a homogeneous feedforward network have demonstrated its capability of generating spike patterns with millisecond fidelity. This model is called the synfire chain and suggests a possible mechanism for generating repeated spike patterns (RSPs). The propagation speed of the pulse packet determines the temporal property of RSPs. However, the relationship between propagation speed and network structure is not well understood. We studied a feedforward network with Mexican-hat connectivity by using the leaky integrate-and-fire neuron model and analyzed the network dynamics with the Fokker-Planck equation. We examined the effect of the spatial pattern of pulse packets on RSPs in the network with multistability. Pulse packets can take spatially uniform or localized shapes in a multistable regime, and they propagate with different speeds. These distinct pulse packets generate RSPs with different timescales, but the order of spikes and the ratios between interspike intervals are preserved. This result indicates that the RSPs can be transformed into the same template pattern through the expanding or contracting operation of the timescale.

With spatially organized neural networks, we examined how bias and noise inputs with spatial structure result in different network states such as bumps, localized oscillations, global oscillations, and localized synchronous firing that may be relevant to, for example, orientation selectivity. To this end, we used networks of McCulloch-Pitts neurons, which allow theoretical predictions, and verified the obtained results with numerical simulations. Spatial inputs, no matter whether they are bias inputs or shared noise inputs, affect only firing activities with resonant spatial frequency. The component of noise that is independent for different neurons increases the linearity of the neural system and gives rise to less spatial mode mixing and less bistability of population activities.

Inspired by recent studies regarding dendritic computation, we constructed a recurrent neural network model incorporating dendritic lateral inhibition. Our model consists of an input layer and a neuron layer that includes excitatory cells and an inhibitory cell; this inhibitory cell is activated by the pooled activities of all the excitatory cells, and it in turn inhibits each dendritic branch of the excitatory cells that receive excitations from the input layer. Dendritic nonlinear operation consisting of branch-specifically rectified inhibition and saturation is described by imposing nonlinear transfer functions before summation over the branches. In this model with sufficiently strong recurrent excitation, on transiently presenting a stimulus that has a high correlation with feed- forward connections of one of the excitatory cells, the corresponding cell becomes highly active, and the activity is sustained after the stimulus is turned off, whereas all the other excitatory cells continue to have low activities. But on transiently presenting a stimulus that does not have high correlations with feedforward connections of any of the excitatory cells, all the excitatory cells continue to have low activities. Interestingly, such stimulus-selective sustained response is preserved for a wide range of stimulus intensity. We derive an analytical formulation of the model in the limit where individual excitatory cells have an infinite number of dendritic branches and prove the existence of an equilibrium point corresponding to such a balanced low-level activity state as observed in the simulations, whose stability depends solely on the signal-to-noise ratio of the stimulus. We propose this model as a model of stimulus selectivity equipped with self-sustainability and intensity-invariance simultaneously, which was difficult in the conventional competitive neural networks with a similar degree of complexity in their network architecture. We discuss the biological relevance of the model in a general framework of computational neuroscience.

We analyze the generalization performance of a student in a model composed of linear perceptrons: a true teacher, ensemble teachers, and the student. Calculating the generalization error of the student analytically using statistical mechanics in the framework of online learning, we prove that when the learning rate satisfies ν1, the properties are completely reversed. If the variety of the ensemble teachers is rich enough, the direction cosine between the true teacher and the student becomes unity in the limit of ν → 0 and K →∞.

We investigate a detection dynamics of a generalized parallel interference canceller (PIC) for CDMA multiuser detection, applied to a randomly spread, fully synchronous base-band uncoded CDMA channel model with additive white Gaussian noise (AWGN) under perfect power control in the large-system limit. To the analysis, we apply the generating functional analysis (GFA), which allows us to study the dynamics of infinitely large disordered system in an exact way. Using the analysis of our generalized PIC, we also show a detection dynamics of Tanaka-Okada PIC (T. Tanaka and M. Okada, 1990) by GFA. The predictions of GFA exhibit good consistency with the computer simulation result

We considered a gamma distribution of interspike intervals as a statistical model for neuronal spike generation. A gamma distribution is a natural extension of the Poisson process taking the effect of a refractory period into account. The model is specified by two parameters: a time-dependent firing rate and a shape parameter that characterizes spiking irregularities of individual neurons. Because the environment changes over time, observed data are generated from a model with a time-dependent firing rate, which is an unknown function. A statistical model with an unknown function is called a semiparametric model and is generally very difficult to solve. We used a novel method of estimating functions in information geometry to estimate the shape parameter without estimating the unknown function. We obtained an optimal estimating function analytically for the shape parameter independent of the functional form of the firing rate. This estimation is efficient without Fisher information loss and better than maximum likelihood estimation. We suggest a measure of spiking irregularity based on the estimating function, which may be useful for characterizing individual neurons in changing environments.

We considered a gamma distribution of interspike intervals as a statisti- cal model for neuronal spike generation. The model parameters consist of a time-dependent firing rate and a shape parameter that characterizes spiking irregularities of individual neurons. Because the environment changes with time, observed data are generated from the time-dependent firing rate, which is an unknown function. A statistical model with an unknown function is called a semiparametric model, which is one of the unsolved problem in statistics and is generally very difficult to solve. We used a novel method of estimating functions in information geometry to estimate the shape parameter without estimating the unknown function. We analytically obtained an optimal estimating function for the shape parameter independent of the functional form of the firing rate. This estimation is efficient without Fisher information loss and better than maximum likelihood estimation.

We report on deterministic and stochastic evolutions of firing states through a feedforward neural network with Mexican-hat-type connectivity. The prevalence of columnar structures in a cortex implies spatially localized connectivity between neural pools. Although feedforward neural network models with homogeneous connectivity have been intensively studied within the context of the synfire chain, the effect of local connectivity has not yet been studied so thoroughly. When a neuron fires independently, the dynamics of macroscopic state variables (a firing rate and spatial eccentricity of a firing pattern) is deterministic from the law of large numbers. Possible stable firing states, which are derived from deterministic evolution equations, are uniform, localized, and nonfiring. The multistability of these three states is obtained where the excitatory and inhibitory interactions among neurons are balanced. When the presynapse-dependent variance in connection efficacies is incorporated into the network, the variance generates common noise. Then the evolution of the macroscopic state variables becomes stochastic, and neurons begin to fire in a correlated manner due to the common noise. The correlation structure that is generated by common noise exhibits a nontrivial bimodal distribution. The development of a firing state through neural layers does not converge to a certain fixed point but keeps on fluctuating.

We present a theory to analyze the performance of the parallel interference canceller (PIC) for code-division multiple-access (CDMA) multiuser detection, applied to a randomly spread, fully synchronous baseband uncoded CDMA channel model with additive white Gaussian noise under perfect power control in the large-system limit. We reformulate PIC as an approximation to the belief propagation algorithm for the detection problem. We then apply the density evolution framework to analyze its detection dynamics. It turns out that density evolution for PIC is essentially the same as statistical neurodynamics, a theory to describe dynamics of a certain type of neural network model. Adopting this correspondence, we develop the density evolution framework for PIC using statistical neurodynamics. The resulting formulas, however, are only approximately correct for describing detection dynamics of PIC even in the large-system limit, because we ignore the Onsager reaction terms in the derivation. We then propose a modified PIC algorithm, in which we subtract the Onsager reaction terms algorithmically, for which the density evolution formulas give a correct description of the detection dynamics in the large-system limit.

Within the framework of on-line learning, we study the generalization error of an ensemble learning machine learning from a linear teacher perceptron. The generalization error achieved by an ensemble of linear perceptrons having homogeneous initial weight vectors is precisely calculated at the thermodynamic limit of a large number of input elements and shows rich behavior. Our main findings are as follows. The generalization error using an infinite number of linear student perceptrons is equal to only half that of a single linear perceptron, and converges with that of the infinite case with O(1/K) for a finite number of K linear perceptrons.

Ensemble learning of K nonlinear perceptrons, which determine their outputs by sign functions, is discussed within the framework of online learning and statistical mechanics. This paper shows that ensemble generalization error can be calculated by using two order parameters, that is, the similarity between a teacher and a student, and the similarity among students. The differential equations that describe the dynamical behaviors of these parameters are derived analytically in the cases of Hebbian, perceptron and AdaTron learning. These three rules show different characteristics in their affinity for ensemble learning, that is "maintaining variety among students". Results show that AdaTron learning is superior to the other two rules.

In this paper we apply a modification scheme of the parallel interference cancellation (PIC), recently proposed by Tanaka and Okada based on techniques of the statistical neurodynamics, to the linear PIC (LPIC) with a tentative soft decision function f(x)=ax, and show that the modification enhances its convergence property.

Repeated spike patterns have often been taken as evidence for the synfire chain, a phenomenon that a stable spike synchrony propagates through a feedforward network. Inter-spike intervals which represent a repeated spike pattern are influenced by the propagation speed of a spike packet. However, the relation between the propagation speed and network struc- ture is not well understood. While it is apparent that the propagation speed depends on the excitatory synapse strength, it might also be related to spike patterns. We analyze a feedforward network with Mexican-Hat- type connectivity (FMH) using the Fokker-Planck equation. We show that both a uniform and a localized spike packet are stable in the FMH in a certain parameter region. We also demonstrate that the propagation speed depends on the distinct firing patterns in the same network.

We modeled the inhibitory effects of transcranial magnetic stimulation (TMS) on a neural population. TMS is a noninvasive technique, with high temporal resolution, that can stimulate the brain via a brief magnetic pulse from a coil placed on the scalp. Because of these advantages, TMS is extensively used as a powerful tool in experimental studies of motor, perception, and other functions in humans. However, the mechanisms by which TMS interferes with neural activities, especially in terms of theoretical aspects, are totally unknown. In this study, we focused on the temporal properties of TMS-induced perceptual suppression, and we computationally analyzed the response of a simple network model of a sensory feature detector system to a TMS-like perturbation. The perturbation caused the mean activity to transiently increase and then decrease for a long period, accompanied by a loss in the degree of activity localization. When the afferent input consisted of a dual phase, with a strong transient component and a weak sustained component, there was a critical latency period of the perturbation during which the network activity was completely suppressed and converged to the resting state. The range of the suppressive period increased with decreasing afferent input intensity and reached more than 10 times the time constant of the neuron. These results agree well with typical experimental data for occipital TMS and support the conclusion that dynamical interaction in a neural population plays an important role in TMS-induced perceptual suppression.

A novel analytical method based on information geometry was recently proposed, and this method may provide useful insights into the statistical interactions within neural groups. The link between informationgeometric measures and the structure of neural interactions has not yet been elucidated, however, because of the ill-posed nature of the problem. Here, possible neural architectures underlying information-geometric measures are investigated using an isolated pair and an isolated triplet of model neurons. By assuming the existence of equilibrium states, we derive analytically the relationship between the information-geometric parameters and these simple neural architectures. For symmetric networks, the first- and second-order information-geometric parameters represent, respectively, the external input and the underlying connections between the neurons provided that the number of neurons used in the parameter estimation in the log-linear model and the number of neurons in the network are the same. For asymmetric networks, however, these parameters are dependent on both the intrinsic connections and the external inputs to each neuron. In addition, we derive the relation between the information-geometric parameter corresponding to the two-neuron interaction and a conventional cross-correlation measure. We also show that the information-geometric parameters vary depending on the number of neurons assumed for parameter estimation in the log-linear model. This finding suggests a need to examine the information-geometric method carefully. A possible criterion for choosing an appropriate orthogonal coordinate is also discussed. This article points out the importance of a model-based approach and sheds light on the possible neural structure underlying the application of information geometry to neural network analysis.

It is known that storage capacity per synapse increases by synaptic pruning in the case of a correlation-type associative memory model. However, the storage capacity of the entire network then decreases. To overcome this difficulty, we propose decreasing the connectivity while keeping the total number of synapses constant by introducing delayed synapses. In this paper, a discrete synchronous-type model with both delayed synapses and their prunings is discussed as a concrete example of the proposal. First, we explain the Yanai-Kim theory by employing statistical neurodynamics. This theory involves macrodynamical equations for the dynamics of a network with serial delay elements. Next, considering the translational symmetry of the explained equations, we rederive macroscopic steady-state equations of the model by using the discrete Fourier transformation. The storage capacities are analyzed quantitatively. Furthermore, two types of synaptic prunings are treated analytically: random pruning and systematic pruning. As a result, it becomes clear that in both prunings, the storage capacity increases as the length of delay increases and the connectivity of the synapses decreases when the total number of synapses is constant. Moreover, an interesting fact becomes clear: the storage capacity asymptotically approaches 2/pi due to random pruning. In contrast, the storage capacity diverges in proportion to the logarithm of the length of delay by systematic pruning and the proportion constant is 4/pi. These results theoretically support the significance of pruning following an overgrowth of synapses in the brain and may suggest that the brain prefers to store dynamic attractors such as sequences and limit cycles rather than equilibrium states.

This paper proposes neural mechanisms of transcranial magnetic stim- ulation (TMS). TMS can stimulate the brain non-invasively through a brief magnetic pulse delivered by a coil placed on the scalp, interfering with specific cortical functions with a high temporal resolution. Due to these advantages, TMS has been a popular experimental tool in various neuroscience fields. However, the neural mechanisms underlying TMS- induced interference are still unknown; a theoretical basis for TMS has not been developed. This paper provides computational evidence that in- hibitory interactions in a neural population, not an isolated single neuron, play a critical role in yielding the neural interference induced by TMS.

We applied statistical mechanics to an inverse problem of linear mapping to investigate the physics of optimal lossy compressions. We used the replica symmetry breaking technique with a toy model to demonstrate Shannon’s result. The rate distortion function, which is widely known as the theoretical limit of the compression with a fidelity criterion, is derived. Numerical study shows that sparse constructions of the model provide suboptimal compressions.

Recent biological experimental findings have shown that synaptic plasticity depends on the relative timing of the pre- and postsynaptic spikes. This determines whether long-term potentiation (LTP) or long-term depression (LTD) is induced. This synaptic plasticity has been called temporally asymmetric Hebbian plasticity (TAH). Many authors have numerically demonstrated that neural networks are capable of storing spatiotemporal patterns. However, the mathematical mechanism of the storage of spatiotemporal patterns is still unknown, and the effect of LTD is particularly unknown. In this article, we employ a simple neural network model and show that interference between LTP and LTD disappears in a sparse coding scheme. On the other hand, the covariance learning rule is known to be indispensable for the storage of sparse patterns. We also show that TAH has the same qualitative effect as the covariance rule when spatiotemporal patterns are embedded in the network.

Recent biological experimental (cid:12)ndings have shown that the synap- tic plasticity depends on the relative timing of the pre- and post- synaptic spikes which determines whether Long Term Potentiation (LTP) occurs or Long Term Depression (LTD) does. The synaptic plasticity has been called \Temporally Asymmetric Hebbian plas- ticity (TAH)". Many authors have numerically shown that spatio- temporal patterns can be stored in neural networks. However, the mathematical mechanism for storage of the spatio-temporal pat- terns is still unknown, especially the e(cid:11)ects of LTD. In this paper, we employ a simple neural network model and show that inter- ference of LTP and LTD disappears in a sparse coding scheme. On the other hand, it is known that the covariance learning is in- dispensable for storing sparse patterns. We also show that TAH qualitatively has the same e(cid:11)ect as the covariance learning when spatio-temporal patterns are embedded in the network.

The dynamics of selective recall in an associative memory model are analyzed in the scenario of one-to-many association. One-to-many association is one of the most important characteristics of our memory system because a homophone, for example, associates with more than one word and each word can have several meanings. The present model, which can deal with one-to-many association, consists of a heteroassociative network and an autoassociative network. In the heteroassociative network, a mixture of associative items in one-to-many association is recalled by a key item. In the autoassociative network, the selective recall of one of the associative items is examined by providing a seed of a target item either to the heteroassociative network (Model 1) or to the autoassociative network (Model 2). We show by both simulation studies and theoretical analysis that the critical similarity of Model 2 is not sensitive to the change in the dimension ratio of key vectors to associative vectors, and it has smaller critical similarity (correlation between the seed and the target item) than Model 1 for a large initial overlap. On the other hand, we show that Model 1 has smaller critical similarity for a small initial overlap. We also show that unreachable equilibrium states exist in the proposed model. There is a critical loading rate alphar where the reachable equilibrium states are disappeared. Above the critical loading rate alphar, which is smaller than the storage capacity alphac, all equilibrium states are stable, but cannot be reached.