We propose a new network-growing method to accelerate learning and to extract explicit features in complex input patterns. We have so far proposed a new type of network-growing algorithm called greedy network growing algorithm (Kamimura, R, et al., 2002), (Kamimura, R, 2003). By this algorithm, a network can grow gradually by maximizing information on input patterns. In the algorithm, the inverse of the square of the ordinary Euclidean distance between input patterns and connection weights is used to produce competitive unit outputs. When applied to some problems, the method has shown slow learning, and sometimes the method cannot produce a state where information is large enough to produce explicit internal representations. To remedy this shortcoming, we introduce here Minkowski distance between input patterns and connection weights used to produce competitive unit outputs. When the parameter for Minkowski distance is larger, some detailed parts in input patterns can be eliminated, which enables networks to converge faster and to extract main parts of input patterns. We applied our new method to the analysis of some economic data. In the experiment, results confirm that a new method with Minkowski distance can significantly accelerate learning, and clearer features can be extracted.

In this paper, we propose a new computational method for the information theoretic method called greedy network-growing algorithm. The method is called "greedy", because a network with the greedy algorithm tries to absorb as much information as possible from outside. We have so far used the sigmoidal activation function for competitive unit outputs. The method can effectively suppress many competitive units by generating strongly negative connections. However, because methods with the sigmoidal activation function is not so sensitive to input patterns, we have observed that in some cases final representations obtained by the method do not necessarily describe faithfully input patterns. To remedy this shortcoming, we employ the inverse of distance between input patterns and connection weights for competitive unit outputs. As the distance is smaller, competitive units are more strongly activated. Thus, winning units tend to represent input patterns more faithfully than the previous method with the sigmoidal activation function. We applied the new method to animal classification. Experimental results confirmed that more information can be acquired and more explicit features can be extracted by our new method.

We investigate the maximum generating rate of the variable-length nonuniform i.i.d. random number. Our results are generalizations of Han's results (see IEEE Trans. Inform. Theory, vol.46, p.2108-16, Sept. 2000) on the variable-length intrinsic randomness.

In this paper, we propose a new information theoretic method called structural information control for flexible feature discovery. The new method has three distinctive characteristics, which traditional competitive learning fails to offer. First, the new method can directly control competitive unit activation patterns, whereas traditional competitive learning does not have any means to control them. Thus, with the new method, it is possible to extract salient features not discovered by traditional methods. Second, competitive units compete witheach other by maximizing their information content about input patterns. Consequently, this information maximization makes it possible to control flexibly competition processes. Third, in structural information control, it is possible to define many different kinds of information content, and we can choose a specific type of information according to a given objective. When applied to competitive learning, structural information can be used to control the number of dead or spare units, and to extract macro as well as micro features of input patterns in explicit ways. We first applied this method to simple pattern classification to demonstrate that information can be controlled and that different neuron firing patterns can be generated. Second, a dipole problem was used to show that structural information could provide representations similar to those by the conventional competitive learning methods. Finally, we applied the method to a language acquisition problem in which networks must flexibly discover some linguistic rules by changing structural information. Especially, we attempted to examine the effect of the information parameter to control the number of dead neurons, and thus to examine how macro and micro features in input patterns can explicitly be discovered by structural information.

We show that an optimal source code with a cost function for code symbols can be regarded as a random number generator generating a random sequence (not necessarily a sequence of fair coin bits) as the target distribution in the sense that the normalized conditional divergence between the distribution of the generated codeword distribution and the target distribution vanishes as the block length tends to infinity.