The present paper aims to propose a new learning and interpretation method called “impartial competitive learning”, meaning that all participants in a competition should be winners. Due to its importance, the impartiality is forced to be realised even by increasing the corresponding cost in terms of the strength of weights. For the first approximation, three types of impartial competition can be developed: componential, computational, and collective competition. In the componential competition, every weight should have an equal chance on average to win the competition. In the computational competition, all computational procedures should have an equal chance to be applied sequentially in learning. In collective computing for interpretation, all network configurations, obtained by learning, have an equal chance to participate in a process of interpretation, representing one of the most idealised forms of impartiality. The method was applied to a well-known second-language-learning data set. The intuitive conclusion, stressed in the specific science, could not be extracted by the conventional natural language processing methods, because they can deal only with word frequency. The present method tried to extract a main feature beyond the word frequency by competing connection weights and computational procedures impartially, followed by collective and impartial competition for interpretation.

The present paper aims to extend the potential learning method to overcome the problem of collective interpretation, which aims to interpret multi-layered neural networks by compressing them into the simplest ones. In the process of compression, positive, negative, and complicated weights have had unfavourable effects for interpretation. To deal with the problems of collective interpretation, the potential learning is extended only to use positive weights. In addition, to obtain more appropriate weights for interpretation, the number of candidate weights for higher potentialities is first increased as much as possible. Then, from among many candidates, more appropriate weights are selected as more important ones. This extended potentiality learning is expected to produce more stable and more simple representations for easy interpretation. The extended method was applied to three datasets, namely, an artificial dataset, a real eye-tracking dataset, and a student evaluation dataset. In all cases, it was observed that the selectivity of connection weights could be increased. Correspondingly, the majority of connection weights became positive, and the collective weights were quite similar to the regression coefficients of the logistic regression analysis. Finally, for the third dataset (student evaluations), the extended method could extract more explicit input-output relations, compared with the logistic regression analysis, while improving generalisation performance.

The present paper aims to show that neural networks should be constructed so as to increase cooperative potentiality. When the potentiality becomes higher, connections tend to be mutually reinforced and enhanced each other to realize cooperative states in neural networks. Because the cooperative potentiality increase is accompanied by increase in the strength of connection weights, we try to select a small number of connection weights with stronger cooperative potentiality. The cooperative potentiality can be increased by increasing the selectivity of connection weights, and the excessive potentiality acquisition can be reduced. The method was applied to two data sets. In both sets, it was found that the cooperative potentiality could be increased by increasing the selectivity. Also, high potentiality means higher generalization performance to a certain point. By selectivity maximization, a smaller number of neurons could be extracted, whose interpretation made it possible to understand the main mechanism of neural networks.

The present paper tries to show that the greedy layer-wise supervised learning becomes effective enough to improve generalization and interpretation by the help of potential learning. It has been observed that unsupervised pre-training has a shortcoming of vanishing information as is the case of simple multi-layered network training. When the layer becomes higher, valuable information becomes smaller. Information through many different layers tends to diminish considerably and naturally from the information-theoretic point of view. For this, we use the layer-wise supervised training to prevent information from diminishing. The supervised learning has been said to be not good for pre-training for multi-layered neural networks. However, we have found that the new potential learning can be effectively used to extract valuable information through supervised pre-training. With the help of important components extracted by the potential learning, the supervised pre-training becomes effective for training multi-layered neural networks. We applied the method to two data sets, namely, an artificial and banknote data sets. In both cases, the potential learning proved to be effective in increasing generalization performance. In addition, we could show a possibility that final representation by this method could be clearly understood.

The present paper aims to present a new computational method called “repeated assimilation” to enhance the potential learning. The potential learning has been developed to simplify the information maximization methods. The method is based on indirect information maximization and the positive firing of neurons with the independent operation of error minimization and potentiality control. The method is simple enough to be applied to many problems. However, it was observed that the information content was not sufficiently increased by the method in particular for the complex problems. To solve this problem, we here propose a computational method in which potentiality application procedures are successively applied until information reaches the sufficiently large values. The method was applied to the well-known Australian credit data set from the machine learning database. In the experiment, it was found that the number of epochs was sufficient small to reach the final state of learning. Information, in particular, information content in hidden neurons increased to sufficiently large values. Then, it was observed that generalization errors decreased in proportion to increase in information. Finally, compared with other conventional methods, it was confirmed that the best generalization performance in any measures could be obtained by the present method.

The present paper aims to develop a new type of information-theoretic method in which the method is simplified as much as possible by decomposing the learning procedures and gradual information control is used for training multi-layered neural networks. The information-theoretic methods have been successfully applied to the training of neural networks with two main problems, namely, the use of strong inhibition and complicated learning procedures. The strong inhibition on neurons may degrade the performance of neural networks. In addition, complicated learning procedures make it hard to apply the methods to the large-scaled data. Thus, the present paper tries to propose a new information-theoretic method without strong inhibition and to simplify the learning procedures by decomposing them into independent components. In addition, information is gradually increased for the higher layers to transfer smoothly information from the lower to the upper layers. The method was applied to the cardiotocography data set. The experimental results showed that the method could increase collectively information content by decreasing information for each neuron. This information change was found to be related to improved generalization.

The present paper proposes a new type of information-theoretic method to enhance the potentiality of input neurons for improving the class structure of the self-organizing maps (SOM). The SOM has received much attention in neural networks, because it can be used to visualize input patterns, in particular, to clarify class structure. However, it has been observed that the good performance of visualization is limited to relatively simple data sets. To visualize more complex data sets, it is needed to develop a method to extract main characteristics of input patterns more explicitly. For this, several information-theoretic methods have been developed with some problems. One of the main problems is that the method needs much heavy computation to obtain the main features, because the computational procedures to obtain information content should be repeated many times. To simplify the procedures, a new measure called “potentiality” of input neurons is proposed. The potentiality is based on the variance of connection weights for input neurons and it can be computed without the complex computation of information content. The method was applied to the artificial and symmetric data set and the biodegradation data from the machine learning database. Experimental results showed that the method could be used to enhance a smaller number of input neurons. Those neurons were effective in intensifying class boundaries for clearer class structures. The present results show the effectiveness of the new measure of the potentiality for improved visualization and class structure.

The paper proposes a new information-theoretic method to improve the generalization performance of multi-layered neural networks, called "self-organized mutual information maximization learning". In the method, the self-organizing map (SOM) is successively applied to give the knowledge to the subsequent multi-layered neural networks. In this process, the mutual information between input patterns and competitive neurons is forced to increase by changing the spread parameter. Though several methods to increase information have been proposed in multi-layered neural networks, the present paper is the first to confirm that mutual information play important roles in learning in multi-layered neural networks and how to compute the mutual information. The method was applied to the extended Senate data. In the experiments, it is examined whether mutual information is actually increased by the present method, because mutual information can be seemingly increased by changing the spread parameter. Experimental results shows that even if the parameter responsible for changing mutual information was fixed, mutual information could be increased. This means that neural networks can be organized so as to store information content on input patterns by the present method. In addition, it could be observed that generalization performance was much improved by this increase in mutual information.

The present paper proposes a new type of method based on the self-organizing maps to enhance the potentiality of input neurons which can be applied to the extraction important features and improved generalization performance. The importance of input neurons plays important roles in the self-organizing maps. However, little attempts have been made to determine the importance of input neurons, because it has been difficult to measure the importance of neurons in unsupervised learning such as the SOM. Though some information-theoretic methods have been developed to estimate the importance, they need heavy computation to reach the final state. In this context, a new and very simple method is proposed to estimate the importance of input neurons and its performance is experimentally evaluated. The new method is based on the concept of "potentiality". The potentiality means the variance of input neurons toward competitive neurons. When the potentiality becomes higher, the variance of input neurons becomes larger. The self-organizing maps with this potentiality was applied to the two well-known data sets. In both cases, the smaller number of important neurons could be extracted and generalization performance in the supervised mode was much improved compared with that by the conventional methods. However, the map quality in terms of quantization and topographic errors may be degraded. This implies that in actual application, it is needed to compromise between improved generalization and the quality of maps. Though some problems should be solved, the present method of potentiality is simple and strong enough for extracting the importance of input neurons.

We have previously proposed a new type of information-theoretic method where a neuron is evaluated by itself (self-evaluation) and by its surrounding neurons (outer-evaluation). If contradiction between different types of evaluation exists, it is reduced as much as possible. In the present paper, we try to separate self- and outer-evaluation more explicitly and introduce the importance of neurons. First, we separate self- and outer-evaluation to enhance the characteristics shared by the two types of evaluation. Second, we introduce the importance of neurons in evaluation. By using a limited number of important neurons in evaluation, we expect the main characteristics in input patterns to emerge. We applied this contradiction resolution to two types of data, namely, the Senate data and the Euro-yen exchange rates. In both data sets, experimental results confirmed that improved prediction performance was obtained. Prediction performance was better than that obtained by the conventional self-organising map (SOM) and radial basis function networks. In addition, final representations obtained by contradiction resolution were easier to interpret than those given by the conventional SOM. Experimental results confirmed that improved interpretation and visualisation were accompanied by improved prediction performance.

In this paper, we propose a new information-theoretic method to train multi-layered neural networks. The method is composed of unsupervised and supervised phase. In the unsupervised phase, the information-theoretic SOM is used to produce knowledge or SOM knowledge in terms of connection weights. In the supervised phase, connection weights obtained in the unsupervised phase are given as the initial connection weights. We applied the method to the segmentation data in machine learning database. The information-theoretic SOM produced connection weights with explicit class boundaries even for the higher layers. By these connection weights, networks reached their lower level of classification errors very rapidly. In addition, the classification error was lower by the choice of the appropriate number of layers.

In this paper, we improve the information enhancement method to extract important input variables. The information enhancement method has been developed to detect important components in neural systems. Our previous method focused on the detection of the important components; therefore, it has not fully incorporated the information contained in the components into learning processes. We here embed the information enhancement into the processes of self-organizing. The embedding is realized simply by adding the importance of input neurons to the self-organizing maps (SOM). We applied the method to an artificial data set and a real data set taken in the field of the second language learning (L2 learning). In the artificial data set, we could show that the symmetric properties of the data could be explicitly extracted. In the L2 learning data set, only one important variable could be detected. The detected input variable represented a grammatical item termed “the inanimate subject.” This variable was found to be the one which differentiated Japanese high school students from university students in the level of L2 grammatical competence. It has been claimed that students have difficulty in producing sentences with inanimate subjects in the field of L2 learning. Thus, the finding by the method well corresponds to the knowledge of L2 learning.

In this paper, we improve contradiction resolution method. In contradiction resolution, a neuron is self-evaluated to fire without considering other neurons. On the other hand, a neuron is outer-evaluated by considering all neighboring neurons. We improve contradiction resolution by separating the results by self-evaluation from those by outer-evaluation and by limiting the number of winning neurons. The explicit separation is used to enhance contradiction between self and outer-evaluation. The reduction of the number of winning neurons is to focus on a limited number of neurons for extracting main characteristics of input patterns. We applied contradiction resolution to the Senate data. Experimental results confirmed that improved prediction was accompanied by improved visualization and interpretation performance.

In this paper, we propose a new type of information-theoretic method called "dependent input neuron selection" to improve contradiction resolution. Contradiction resolution has been previously introduced to realize self-organizing maps by supposing two types of evaluation, namely, self and outer-evaluation. In self-evaluation, a neuron's firing rate is determined by itself, while in outer-evaluation, the firing rate is determined by other neurons. Outer-evaluation corresponds to cooperation between neurons in the self-organizing maps. Dependent input neuron selection aims to use a small number of input neurons which are forced to respond to different input patterns. Our method was applied to the prediction of dollar-yen exchange rates. Experimental results confirmed that prediction performance was improved by choosing the appropriate number of winning input neurons. The improved performance can be attributed to the fact that connection weights were condensed into several groups and winning input neurons tended to respond to different time lags.

In this paper, we propose a new type of neural learning method where two types of neurons interact with other. The two types of neurons are individually and collectively treated neurons. Though there are many types of interaction between two neurons, we suppose for simplification that individually treated neurons should be similar to collectively treated neurons as much as possible. This model can be applied to the self-organizing maps whose performance can be enhanced by the introduction of interaction of neurons. Then, we applied the method to the breast tissue and protein classification problem of the machine learning database. The experimental results showed that much clearer class boundaries could be produced, though quantization and topographic errors were slightly higher than those by the conventional SOM.

In this paper, we propose a new type of information-theoretic method called “contradiction resolution.” Neurons are supposed to be evaluated by two different ways. First, the neurons must evaluate themselves for themselves, namely, self-evaluation. On the other hand, neurons must be evaluated by their neighboring neurons, namely, outer-evaluation. In a society of neurons, contradiction between self and outer-evaluation must be reduced as much as possible. We apply the method to self-organizing maps. Our method modifies the cooperation of neurons in such a way that difference between self and outer-evaluation is reduced. We applied the method with self-organizing maps to the visualization of dollar-yen exchange rates. Our method could produce explicit class structure in which the dollar-yen rates were divided into three specific periods and an additional one with the highest and lowest peaks.

In this paper, we propose a new type of information-theoretic method called “information-theoretic cooperative learning.” In this method, two networks, namely, cooperative and uncooperative networks are prepared. The roles of these networks are controlled by the cooperation parameter α. As the parameter is increased, the role of cooperative networks becomes more important in learning. We applied the method to the automobile data from the machine learning database. Experimental results showed that cooperation control could be used to increase mutual information on input patterns and to produce clearer class structure in SOM.

In this paper, we propose a new information-theoretic method to explicitly interpret final representations created by learning. The new method, called ldquoselective enhancement learning,rdquo aims at producing explicit representation with fewer input variables. The variable selection is performed by information enhancement in which with a specific and enhanced variable, mutual information, is measured. As this information grows larger, the importance of the variable increases. With selected and important variables, a network is retrained by free energy minimization. In this free energy minimization, we can obtain connection weights by considering the importance of specific variables. When we applied the method to the Senate problem, experimental results showed that clear representations could be obtained with a smaller number of variables. This tendency was more explicit when the network was large.

In this paper, we propose a new learning method called ldquoself-enhancement learning.rdquo In this model, a network enhances its state by itself, and this enhanced state is to be imitated by another state of the network. The word ldquotargetrdquo in our model means that a target is created spontaneously by a network, which must try to attain the target. Enhancement is realized by changing the Gaussian width or enhancement parameter. With different enhancement parameters, we can set up the different states of a network. In particular, we set up an enhanced and a relaxed state, and the relaxed state tries to imitate the enhanced state as much as possible. To demonstrate the effectiveness of this method, we apply the self-enhancement learning to the SOM. For this purpose, we introduce collectiveness into an enhanced state in which all neurons collectively respond to input patterns. Then, this enhanced and collective state should be imitated by the other non-enhanced and relaxed state. We applied the method to the Iris problem. Experimental results showed that the U-matrices obtained were significantly similar to those produced by the conventional SOM. However, much better performance could be obtained in terms of quantitative and topological errors. The experimental results suggest the possibility for the self-enhancement learning to be applied to many different neural network models.

In this paper, we propose a new self-supervised learning method for competitive learning as well as self-organizing maps. In this model, a network enhances its state by itself, and this enhanced state is to be imitated by another state of the network. We set up an enhanced and a relaxed state, and the relaxed state tries to imitate the enhanced state as much as possible by minimizing the free energy. To demonstrate the effectiveness of this method, we apply information enhancement learning to the SOM. For this purpose, we introduce collective-ness, in which all neurons collectively respond to input patterns, into an enhanced state. Then, this enhanced and collective state should be imitated by the other non-enhanced and relaxed state. We applied the method to an artificial data and three data from the well-known machine learning database. Experimental results showed that the U-matrices obtained were significantly similar to those produced by the conventional SOM. However, better performance could be obtained in terms of quantitative and topological errors. The experimental results suggest the possibility for self-supervised learning to be applied to many different neural network models.

In this paper, we propose structural enhanced information for detecting and visualizing main features in input patterns. We have so far proposed information enhancement for feature detection, where, if we want to focus upon components such as units and connection weights and interpret the functions of the components, we have only to enhance competitive units with the components. Though this information enhancement has given favorable results in feature detection, we further refine the information enhancement and propose structural enhanced information. In structural enhanced information, three types of enhanced information can be differentiated, that is, first-, second-and third-order enhanced information. The first-order information is related to the enhancement of competitive units themselves in a competitive network, and the second-order information is dependent upon the enhancement of competitive units with input patterns. Then, the third-order information is obtained by subtracting the effect of the first-order information from the second-order information. Thus, the third-order information more explicitly represents information on input patterns. With this structural enhanced information, we can estimate more detailed features in input patterns. For demonstrating explicitly and intuitively the improved performance of our method, the conventional SOM was used, and we transformed competitive unit outputs so as to improve visualization. The method was applied to the Johns Hopkins University Ionosphere database. In the problem, we succeeded in visualizing the detailed and important features of input patterns by using the third-order information.

In this paper, we propose a new information-theoretic approach to competitive learning and self-organizing maps. We use several information-theoretic measures such as conditional information and information losses to extract main features in input patterns. First, conditional information content is used to show how much information is contained in a competitive unit or an input pattern. Then, information content in each variable is detected by information losses. The information loss is defined by difference between information with all input units and information without an input unit. We applied the method to an artificial data, the Iris problem, a student survey, a CPU classification problem and a company survey. In all cases, experimental results showed that main features in input patterns were clearly detected.

In this paper, we propose a new algorithm to interpret and improve multi-layered neural networks. The new method is used to simplify and interpret final representations obtained by conventional learning methods such as BP or RBF. In addition, the method is used to retrain networks so as to produce networks with better performance in terms of generalization errors. Neural networks have so far produced effective internal representations for many problems. However, because obtained information is extremely distributed over many elements, it is difficult to interpret the meaning of representations. For this problem, we use competitive learning by which we can focus upon some elements in networks. In addition, because the number of effective elements in neural networks can significantly be reduced, we can expect improved performance in terms of generalization. Competitive processes are considered to be a process of entropy minimization in which a small number of units acquires the majority of information on input patterns. For simplifying computation, we use a free energy in which entropy and training errors are simultaneously used. We applied the methods to an artificial data and a cabinet approval rating estimation problem. In both problems, we succeeded in extracting the main features in input patterns and improved generalization performance could be obtained.

In this paper, we propose a new information-theoretic approach to self-organizing maps. Information-theoretic methods have been applied to self-organizing maps in several different ways. The methods have been considered to be powerful enough to give simple and practical procedures to replace conventional self-organizing maps. However, computational complexity in terms of mutual information and cooperation has restricted their applications to minimum points. To overcome this situation, we use a free energy similar to that in statistical mechanics in controlling information. By this introduction, computation complexity in mutual information can significantly be reduced. Experimental results showed that feature maps obtained by free energy minimization was significantly similar to those by the conventional SOM. In addition, for some cases, clearer feature maps could be obtained by the free energy method.

We have used competitive learning and information-theoretic competitive learning for data analyses. Though better interpretation of connection weights is possible, competitive learning is limited to unsupervised learning and supervised classification. In this context, we try to extend competitive learning to supervised learning, and also extend it to continuous targets. For this, we introduce controlled competitive learning in which competition is guided by error signals from an output layer. This is a new approach to extend competitive learning to supervised learning. Then, the method is considered to be a new simple computational method to train hierarchical networks with the winner-take-all algorithm. In addition, this is a new approach to train the RBF networks. We apply the method to an artificial problem as well as actual conjoint analyses of information science education. The results show that more efficient learning can be demonstrated in terms of the number of competitive units and the interpretation of final connection weights. In addition, we can say that a result by the regression analysis is only one aspect of the controlled competitive learning.

Information-theoretic competitive learning has been proved to be more general and more flexible type of competitive learning. However, one of the major shortcomings of this method is that it is sometimes very slow in learning. To overcome this problem, we introduce forced information used to force networks to increase information by supposing maximum information. We applied the method to an artificial data as well as a student survey. In both cases, we observed that information was very rapidly increased to stable points. Compared with results by the principal component analysis, our method showed more clearly the main features of input patterns. In addition, we can more easily explain a main mechanism of feature detection by forced information.

In this paper, we propose a new type of information- theoretic method called information loss to evaluate the importance of input variables. Information loss is defined by difference between information content with all input variables and without an input variable. Thus, information loss represents to what extent an input variable plays an important role in learning. By experiments, we can see that information loss extracts distinctive features by which different input patterns are separated. We applied the information loss to a semantic differential problem, that is, an image of information science education. We successfully extracted the main features of input patterns and succeeded in revealing an image of information science education.

In this paper, we propose a method to extend information-theoretic competitive learning to supervised competitive learning. We have shown that information maximization correspond to competition in neurons. However, this information maximization cannot be used to specify which neurons should be winners. Thus, it is impossible to incorporate teacher information in information maximization. For dealing with this teacher information, we use weighted distance between input patterns and connection weights. Even if distance between input patterns and connection weights is not so small, the distance are made smaller by the parameter considering teacher information. By this weighted distance, we can naturally incorporate teacher information and extend unsupervised competitive learning to supervised information-theoretic competitive learning.

We have so far demonstrated that self-organization maps can be realized by information maximization. However, one problem of information maximization has been pointed out, that is, the excessive information acquisition. In this paper, we try to show that the excessive information can be eliminated by the surface information stopping criterion. In this criterion, we propose two kinds of information, that is, the surface or the deep information. The surface information precedes the deep information, and is an indicator for the future behavior of information increase. By monitoring the surface information, the excessive information can be eliminated. We applied the method to the artificial data and show that a clear feature map can be obtained just by maximizing information. In the second and the third experiment, by the surface information stopping criterion, the maps that exhibit well the characteristics of input patterns can be obtained.

In this paper, we apply our information-theoretic method to a student survey. The information-theoretic method aims to extract main features in input patterns by condensing information contained in input patterns as much as possible. By using 2500 students' responses to the questionnaire, we could extract the main subjects in which the majority of students have interest. Especially, we could classify students into several groups by their interest. On the other hand, the conventional PCA could not demonstrate specific features in input patters. Thus, the information-theoretic neural methods can open up a new perspective for data analyses.

In this paper, we propose a new information-theoretic method to produce explicit self-organizing maps (SOMs). Competition is realized by maximizing mutual information between input patterns and competitive units. Competitive unit outputs are computed by the Gaussian function of distance between input patterns and competitive units. A property of this Gaussian function is that, as distance becomes smaller, a neuron tends to fire strongly. Cooperation processes are realized by taking into account the firing rates of neighboring neurons. We applied our method to uniform distribution learning, chemical compound classification and road classification. Experimental results confirmed that cooperation processes could significantly increase information content in input patterns. When cooperative operations are not effective in increasing information, mutual information as well as entropy maximization is used to increase information. Experimental results showed that entropy maximization could be used to increase information and to obtain clearer SOMs, because competitive units are forced to be equally used on average.

In this paper, we introduce costs in the framework of information maximization and try to maximize the ratio of information to its associated cost. We have shown that competitive learning is realized by maximizing mutual information between input patterns and competitive units. One shortcoming of the method is that maximizing information does not necessarily produce representations faithful to input patterns. Information maximizing primarily focuses on some parts of input patterns that are used to distinguish between patterns. Thus, we introduce the cost that represents average distance between input patterns and connection weights. By minimizing the cost, final connection weights by information maximization reflect well input patterns. We applied the method to a political data analysis and a Wisconsin cancer problem. Experimental results confirmed that by introducing the cost, representations faithful to input patterns were obtained. In addition, generalization performance was significantly improved.

We propose a new method for information-theoretic competitive learning that maximizes information about input patterns as well as target patterns. The method is called teacher-directed information maximization because target information directs networks to produce appropriate outputs. Target information is given in the input layer, and errors need not be back-propagated, as with conventional supervised learning methods. Thus, this is a very efficient supervised learning method. In the new method, we use information-theoretic competitive learning with Gaussian activation functions to simulate competition, because information maximization processes are accelerated by changing the width of the functions. Teacher information is added by distorting the distance between input patterns and connection weights. We applied our method to two problems: a road classification problem and a voting attitude problem. In both problems, we could show that training errors could be significantly decreased and better generalization performance could be obtained.

We propose a new information theoretic method to produce explicit self-organizing maps. Competition is realized by maximizing mutual information between input patterns and competitive units. Competitive unit outputs are computed by the Gaussian function of distance between input patterns and competitive units. By the property of this Gaussian function, as distance becomes smaller, a neuron tends to fire strongly. Cooperation processes are realized by taking into account the firing rates of neighboring neurons. We applied our method to uniform distribution learning and road classification. Experimental results confirmed that cooperation processes could significantly increase information content in input patterns. When cooperative operations are not effective in increasing information, mutual information as well as entropy maximization is used to increase information. Experimental results especially showed that entropy maximization could be used to increase information and to give clearer self-organizing maps, because competitive units are forced to use equally on average.

We propose a new network-growing method to accelerate learning and to extract explicit features in complex input patterns. We have so far proposed a new type of network-growing algorithm called greedy network growing algorithm (Kamimura, R, et al., 2002), (Kamimura, R, 2003). By this algorithm, a network can grow gradually by maximizing information on input patterns. In the algorithm, the inverse of the square of the ordinary Euclidean distance between input patterns and connection weights is used to produce competitive unit outputs. When applied to some problems, the method has shown slow learning, and sometimes the method cannot produce a state where information is large enough to produce explicit internal representations. To remedy this shortcoming, we introduce here Minkowski distance between input patterns and connection weights used to produce competitive unit outputs. When the parameter for Minkowski distance is larger, some detailed parts in input patterns can be eliminated, which enables networks to converge faster and to extract main parts of input patterns. We applied our new method to the analysis of some economic data. In the experiment, results confirm that a new method with Minkowski distance can significantly accelerate learning, and clearer features can be extracted.

We propose here a new computational method for the information-theoretic method, called the greedy network-growing algorithm, to facilitate a process of information acquisition. We have so far used the sigmoidal activation function for competitive unit outputs. The method can effectively suppress many competitive units by generating strongly negative connections. However, because methods with the sigmoidal activation function are not very sensitive to input patterns, we have observed that in some cases final representations obtained by the method do not necessarily faithfully describe input patterns. To remedy this shortcoming, we employ the inverse of distance between input patterns and connection weights for competitive unit outputs. As the distance becomes smaller, competitive units are more strongly activated. Thus, winning units tend to represent input patterns more faithfully than in the previous method with the sigmoidal activation function. We applied the new method to artificial data analysis and animal classification. Experimental results confirmed that more information can be acquired and more explicit features can be extracted by our new method.

In this paper, we extend our greedy network-growing algorithm to multi-layered networks. With multi-layered networks, we can solve many complex problems that single-layered networks fail to solve. In addition, the network-growing algorithm is used in conjunction with teacher-directed learning that produces appropriate outputs without computing errors between targets and outputs. Thus, the present algorithm is a very efficient network-growing algorithm. The new algorithm was applied to three problems: the famous vertical-horizontal lines detection problem, a medical data problem and a road classification problem. In all these cases, experimental results confirmed that the method could solve problems that single-layered networks failed to. In addition, information maximization makes it possible to extract salient features in input patterns.

In this paper, we report generalization performance by a new type of efficient learning method called teacher-directed learning. In the method, information on targets or teachers is supposed to be maximized before learning. This teacher information directs input patterns to activate correct competitive units. Because connection weights on teachers are all fixed to maximize information, we have only to update a small number of connections concerning input units and competitive units. Thus, the new method is computationally efficient. However, generalization performance of the new method has not been evaluated. In this paper, we use the Iris problem and the voting attitude problem to show that generalization performance is better than conventional methods. In addition, experimental results reconfirm that we can obtain clearer internal representations.

In this paper, we propose a new information theoretic method for self-organizing maps. The method aims to control competitive processes flexibly, that is, to produce different competitive unit activations according to information content obtained in learning. Competition is realized by maximizing mutual information between input patterns and competitive units. Competitive unit outputs are computed by the inverse of distance between input patterns and competitive unit. As distance is smaller, a neuron tends to fire strongly. Thus, winning neurons represent faithfully input patterns. We applied our method to a road classification problem. Experimental results confirmed that the new method could produce more explicit self-organizing maps than conventional self-organizing methods.

In this paper, we propose a new computational method for the information theoretic method called greedy network-growing algorithm. The method is called "greedy", because a network with the greedy algorithm tries to absorb as much information as possible from outside. We have so far used the sigmoidal activation function for competitive unit outputs. The method can effectively suppress many competitive units by generating strongly negative connections. However, because methods with the sigmoidal activation function is not so sensitive to input patterns, we have observed that in some cases final representations obtained by the method do not necessarily describe faithfully input patterns. To remedy this shortcoming, we employ the inverse of distance between input patterns and connection weights for competitive unit outputs. As the distance is smaller, competitive units are more strongly activated. Thus, winning units tend to represent input patterns more faithfully than the previous method with the sigmoidal activation function. We applied the new method to animal classification. Experimental results confirmed that more information can be acquired and more explicit features can be extracted by our new method.

In this paper, we propose self-adaptive multi-layered networks in which information in each processing layer is always maximized. Using these multi-layered networks, we can solve complex problems and discover salient features that single-layered networks fail to extract. In addition, this successive information maximization enables networks gradually to extract important features. We applied the new method to the Iris data problem, the vertical-horizontal lines detection problem, a phonological data analysis problem and a medical data problem. Experimental results confirmed that information can repeatedly be maximized in multi-layered networks and that the networks can extract features that cannot be detected by single-layered networks. In addition, features extracted in successive layers are cumulatively combined to detect more macroscopic features.

In this paper, we propose a new type of efficient learning method called teacher-directed learning. The method can accept training patterns and correlated teachers, and we need not back-propagate errors between targets and outputs into networks. Information flows always from an input layer to an output layer. In addition, connections to be updated are those from an input layer to the first competitive layer. All other connections can take fixed values. Learning is realized as a competitive process by maximizing information on training patterns and correlated teachers. Because information is maximized, information is compressed into networks in simple ways, which enables us to discover salient features in input patterns. We applied this method to the vertical and horizontal lines detection problem, the analysis of US–Japan trade relations and a fairly complex syntactic analysis system. Experimental results confirmed that teacher information in an input layer forces networks to produce correct answers. In addition, because of maximized information in competitive units, easily interpretable internal representations can be obtained.

In this paper, we proopose a new information theoretic approach to competitive learning. The new approach is called greedy information acquisition , because networks try to absorb as much information as possible in every stage of learning. In the first phase, with minimum network architecture for realizing competition, information is maximized. In the second phase, a new unit is added, and thereby information is again increased as much as possible. This proceess continues until no more increase in information is possible. Through greedy information maximization, different sets of important features in input patterns can be cumulatively discovered in successive stages. We applied our approach to three problems: a dipole problem; a language classification problem; and a phonological feature detection problem. Experimental results confirmed that information maximization can be repeatedly applied and that different features in input patterns are gradually discovered. We also compared our method with conventional competitive learning and multivariate analysis. The experimental results confirmed that our new method can detect salient features in input patterns more clearly than the other methods.

In this paper, we propose a new information theoretic method called structural information control for flexible feature discovery. The new method has three distinctive characteristics, which traditional competitive learning fails to offer. First, the new method can directly control competitive unit activation patterns, whereas traditional competitive learning does not have any means to control them. Thus, with the new method, it is possible to extract salient features not discovered by traditional methods. Second, competitive units compete witheach other by maximizing their information content about input patterns. Consequently, this information maximization makes it possible to control flexibly competition processes. Third, in structural information control, it is possible to define many different kinds of information content, and we can choose a specific type of information according to a given objective. When applied to competitive learning, structural information can be used to control the number of dead or spare units, and to extract macro as well as micro features of input patterns in explicit ways. We first applied this method to simple pattern classification to demonstrate that information can be controlled and that different neuron firing patterns can be generated. Second, a dipole problem was used to show that structural information could provide representations similar to those by the conventional competitive learning methods. Finally, we applied the method to a language acquisition problem in which networks must flexibly discover some linguistic rules by changing structural information. Especially, we attempted to examine the effect of the information parameter to control the number of dead neurons, and thus to examine how macro and micro features in input patterns can explicitly be discovered by structural information.

We propose a conditional information analysis in which information on important patterns is selectively detected. The selection is realized by /spl alpha/-information which can be used to maximize or minimize selectively conditional information, according to the important or characteristics of input patterns. The information analysis was applied to two feature detection problems: an alphabet character recognition and medical data. In both problems, experimental results confirmed that conditional information is flexibly maximized or minimized, depending upon input patterns. They also showed that the conditional information is a good measure to distinguish between different classes.

We propose a selective information acquisition device which can be used to store information selectively according to the importance or characteristics of input patterns. For selective information acquisition, we introduce /spl alpha/-information to distort the ordinary Shannon information function. We maximize or minimize the information to eliminate this distortion. Thus, the distortion elimination can be employed as a basic mechanism of the device to maximize or minimize information selectively. The information device is applied to a phonological feature detection problem. In this problem, experimental results confirmed that conditional information is flexibly maximized or minimized, depending upon input patterns. They also showed that conditional information is a good measure to distinguish between different classes, and that the strength of conditional information is used to classify input patterns.

Proposes a new information-theoretic method called structural information, and argues that this new method should be substituted for the traditional competitive method. Structural information control is a more powerful and biologically sounder model, because it uses a soft winner-takes-all model instead of a hard winner-takes-all model. Experiments were conducted to apply the structural information to linguistic rule extraction in which the choice of different donatory verbs must be inferred in an unsupervised way. We found that the structural information control can detect linguistic rules more accurately than the traditional competitive learning method.

We propose information mediators to control information in internal representations. The information mediators can control the production of internal representations. Thus we can obtain appropriate internal representations, depending on the given targets. The information mediators were applied to multi-level information processing. In this processing information minimization is related to information maximization through the information mediators. Experimental results on language acquisition problems show that simple and well-interpretable internal representations can be obtained by the information mediators.

In this paper, new information called structural information is proposed. The structural information is used to control information in internal representations, and eventually to control the production of internal representations. By controlling the structural information, we can obtain appropriate internal representations, depending on given problems. The structural information methods were applied to the XOR problem in which the utility of structural information is shown. Then, we applied the methods to language acquisition problems complex enough to test the performance. Experimental results confirmed that generalization is not concerned with total information but with the second order information.

We propose an information controller to maximize and simultaneously minimize information in neural networks. The controller can be used to improve generalization performance and interpret explicitly internal representations. An apparently contradictory operation of the simultaneous maximization and minimization is possible by assuming two kinds of information: collective information and individual information, representing collective activities of hidden units and activities of individual input-hidden connections, respectively. By maximizing the collective information and minimizing the individual information, simple networks can be generated in terms of the number of connections and number of hidden units. This method was applied to the inference of the maximum onset principle and the sonority principle of artificial languages. In these problems, it was shown that the individual information could be sufficiently minimized and simultaneously the collective information was maximized. Experimental results confirmed improved generalization performance by the suppression of overtraining. In addition, internal representations created by information maximization and minimization could easily be interpreted.

An attempt is made to generate natural language by using a recurrent neural network with the temporal supervised learning algorithm (TSLA), developed by R.J. Williams and D. Zipser (1989). As TSLA uses explicit representation of consecutive events, it can deal with time-changing phenomena without increasing the number of units in the network. However, its performance has been evaluated exclusively upon the limited short sequences or sequences with explicit regularity and not for the sequences of natural language, which show complex and long-distance correlation. It was found that TSLA showed extreme instability in the learning process, and it took a long time to finish the learning. Thus, the author proposes two methods to improve the performance of TSLA. The first is the variable learning rate method, which is used to remove the instability of the learning process. The second is Minkowski-rpower metrics, which is used to improve the learning time