Deep convolutional neural networks (deep CNNs) show a large power for robust recognition of visual patterns. The neocognitron, which was first proposed by Fukushima (1979), is recognized as the origin of deep CNNs. Its architecture was suggested by the neurophysiological findings on the visual systems of mammals. It acquires the ability to recognize visual patterns robustly through learning. Although the neocognitron has a long history, improvements of the network are still continuing. This article discusses the recent neocognitron, focusing on differences from the conventional deep CNNs. Some other functions of the visual system can also be realized by networks extended from the neocognitron, for example, recognition of partly occluded patterns, the mechanism of selective attention, and so on.

The neocognitron is a deep (multi-layered) convolutional neural network that can be trained to recognize visual patterns robustly. In the intermediate layers of the neocognitron, local features are extracted from input patterns. In the deepest layer, based on the features extracted in the intermediate layers, input patterns are classified into classes. A method called IntVec (interpolating-vector) is used for this purpose. This paper proposes a new learning rule called margined Winner-Take-All (mWTA) for training the deepest layer. Every time when a training pattern is presented during the learning, if the result of recognition by WTA (Winner-Take-All) is an error, a new cell is generated in the deepest layer. Here we put a certain amount of margin to the WTA. In other words, only during the learning, a certain amount of handicap is given to cells of classes other than that of the training vector, and the winner is chosen under this handicap. By introducing the margin to the WTA, we can generate a compact set of cells, with which a high recognition rate can be obtained with a small computational cost. The ability of this mWTA is demonstrated by computer simulation.

The neocognitron is a multi-layered convolutional network that can be trained to recognize visual patterns robustly. In the intermediate layers of the neocognitron, local features are extracted from input patterns. In the highest (or deepest) layers of the network, the method of Interpolating-Vector is used for classifying patterns based on the features extracted by the intermediate layers. During the learning, several reference vectors for each class are created from a set of training vectors. To recognize an input vector, we measure distances (based on similarities) between the input vector and planes that are spanned by every trio of reference vectors of the same class. The class name of the nearest plane is taken as the result of classification. To reduce the computational cost, we propose to search the nearest plane, not among all possible combinations of three reference vectors, but only among trios that contain the nearest reference vector. For reducing the computational cost, it is also important to represent the large number of training vectors accurately with a compact set of reference vectors. To create a compact set of reference vectors, the learning is carried out in two steps. In the first step, reference vectors are just chosen from vectors in the training set. We start modifying reference vectors (namely, fine tuning of connections) from the second step after an enough number of reference vectors have been chosen. The effectiveness of the proposed method for recognizing hand-written digits is demonstrated by computer simulation.

This paper proposes a powerful algorithm for pattern recognition, which usesinterpolating vectors. Labeled reference vectors in a multi-dimensional feature space are first produced by a kind of competitive learning. We then assume that interpolating vectors are densely placed along line segments connecting all pairs of reference vectors of the same label. From these interpolating vectors, we choose the one that has the largest similarity to the test vector. Its label shows the result of pattern recognition. In practice, we can get the same result with a simpler process. We applied this method to theneocognitronfor handwritten digit recognition. When the network had been trained with 5000 digits, the use of interpolating vectors reduced the error rate from 1.52% to 1.02% for a blind test set of 5000 digits. The use of interpolating vectors is not limited to the neocognitron but can be applied to various systems for pattern recognition.

When we travel in an environment, we have an optic flow on the retina. Neurons in the area MST of macaque monkeys are reported to analyze optic flow. The MST cells respond selectively to rotation, expansion/contraction and spiral motion. Previously, it was suggested that vector-field calculus is useful for analyzing optic flow. However, few MST models have been proposed so far, based on it. This paper adopts the vector-field hypothesis and explicitly proposes a novel neural network model for MST. Our model consists of hierarchically connected layers, namely, retina, V1, MT and MST. V1 cells measure local velocity. MT cells extract relative velocity with their antagonistic networks. MST cells simply collect the signals from MT cells and respond selectively to various types of optic flows. We demonstrate through a computer simulation that the simple architecture of our model is enough to explain a variety of results of neurophysiological experiments.

This paper proposes a neural network model that extracts axes of symmetry from visual patterns. The input patterns can be line drawings, plane figures or gray-scaled natural images taken by CCD cameras. The model is a multi-layered network. It has an input layer, a contrast-extracting layer, edge-extracting layers (an S-cell layer and a C-cell layer), and layers extracting symmetry axes. These layers are connected in a cascade in a hierarchical manner. The model extracts oriented edges from the input image first, and then tries to extract axes of symmetry. To reduce the computational cost, the model checks conditions of symmetry, not directly from the oriented edges, but from a blurred version (low-resolution responses covering a large area) of them. The use of blurred signals endows the network with a large tolerance to deformation of input patterns, too.

Modeling neural networks is a powerful approach to uncover the mechanism of the brain, and the results of the research are ready to use for engineering applications. This paper introduces several models for vision from recent works by the author. (1) Increased recognition rate of the neocognitron: to increase the recognition rate of the neocognitron, several modifications have been applied to the network architecture and the learning method. For example, an inhibitory surround in the connections to C-cells is useful for this purpose. When trained with 3000 patterns, the neocognitron shows a recognition rate of 98.6% for a blind test set randomly sampled from a large database of handwritten digits (ETL-1), and 100% for the training set. (2) A neocognitron that can accept incremental learning, without giving severe damage to old memories or reducing learning speed: It uses competitive learning, and the learning of all stages of the hierarchical network progresses simultaneously. (3) A model that. has an ability to recognize and restore partly occluded patterns: even the identical image is perceived differently by human beings depending on the shape of occluding objects. The model responds in a similar way to human beings. It is a multi-layered hierarchical neural network, in which visual information is processed by interaction of bottom-up and top-down signals. Occluded parts of a pattern are restored mainly by feedback signals from the highest stage of the network, while the unoccluded parts are reproduced mainly by signals from lower stages. The model does not use a simple template matching method. It can recognize and restore even deformed versions of learned patterns.

This paper proposes a hypothesis explaining why a pattern is easier to be recognized when the occluding objects are visible. A neural network model is constructed based on the hypothesis and is demonstrated that the model responds to occluded patterns in a similar way as human beings. The visual system extracts various visual features from the input pattern and then recognizes it. If the occluding objects are invisible, the visual system will have difficulty in distinguishing which features are relevant to the original pattern and which are newly generate by the occlusion. If the occluding objects are visible, however, the visual system can easily discriminate relevant from irrelevant features and recognize the occluded pattern correctly. The proposed model is an extended version of the neocognitron model. The activity of the feature-extracting S-cells whose receptive fields cover the occluding objects is suppressed in the lowest stage of the hierarchical network.

We offer a neural network model which has the ability of sift, scale, and deformation-invariant pattern recognition. The model extracts spatial relations between fixating feature and each of peripheral features. The mechanism of saccadic eye movement enables the model to analyze whole pattern-structure. The ability of the model has been tested on a computer.

This paper proposes a new learning rule by which cells with shift-invariant receptive fields are self-organized. Namely, cells similar to simple and complex cells in the primary visual cortex are generated in a network trained by the new leaning rule. To demonstrate the new learning rule, we simulate a three-layered network that consists of an input layer (retina), a layer of S-cells (simple cells), and a layer of C-cells (complex cells). During the learning, straight lines of various orientations sweep across the input layer Both S- and C-cells are created through competition. Although S-cells compete depending on their instantaneous outputs, C-cells compete depending on the traces (or temporal averages) of their outputs. For the self-organization of S-cells, only winner S-cells have LTP (long term potentiation) in their input connections. For the self-organization of S-cells, however, loser S-cells have LTD (long term depression) in their input connections, while winners have LTP. Both S- and C-cells are accompanied by inhibitory cells. Modification of inhibitory connections together with excitatory connections is important for creation of C-cells as well as S-cells.

The selective attention model proposed by the author is a neural network model which has the ability to segment patterns, as well as the function of recognizing them. The principles of this selective attention model have been extended for the recognition and segmentation of connected characters. The topics discussed include the network architecture, pattern recognition, segmentation, repairing imperfect patterns, attention focusing, search control, attention switching, size and position information, and computer simulation. Improvement of the system using bend detectors is discussed.>

A pattern recognition system which works with the mechanism of the neocognitron, a neural network model for deformation-invariant visual pattern recognition, is discussed. The neocognition was developed by Fukushima (1980). The system has been trained to recognize 35 handwritten alphanumeric characters. The ability to recognize deformed characters correctly depends strongly on the choice of the training pattern set. Some techniques for selecting training patterns useful for deformation-invariant recognition of a large number of characters are suggested.

A recognition with a large-scale network is simulated on a PDP-11/34 minicomputer and is shown to have a great capability for visual pattern recognition. The model consists of nine layers of cells. The authors demonstrate that the model can be trained to recognize handwritten Arabic numerals even with considerable deformations in shape. A learning-with-a-teacher process is used for the reinforcement of the modifiable synapses in the new large-scale model, instead of the learning-without-a-teacher process applied to a previous model. The authors focus on the mechanism for pattern recognition rather than that for self-organization.