When a neural network is trained for a specific task, activations of the hidden units encode internal representations of the inputs. Models formulated in a layer-wise fashion are believed to structure such representations in a hierarchical fashion, increasing in complexity and abstractness towards the output layer, in an analogy to both biological neural networks and artificially constructed computational models. This paper examines how the structure of classification tasks manifests itself in these internal representations, using a variety of ad hoc metrics. The results, based on feedforward neural networks trained on moderately complex datasets MNIST and SVHN, confirm our hypothesis that the hidden neurons become more correlated with class information towards the output layer, providing some evidence for an increasing bottom-up organization in representations. While various activation functions lead to noticeably different internal representations as measured by each of the methods, the differences in overall classification accuracy remain minute. This confirms the intuition that there exist qualitatively different solutions to the complex classification problem imposed by nonlinearities in the hidden layers.

It has been demonstrated that the computational capabilities of echo state networks are maximized when the recurrent layer is close to the border between a stable and an unstable dynamics regime, the so called edge of stability, or criticality. The maximization of performance is computationally useful, leading to minimal prediction error or maximal memory capacity, and has been shown to lead to maximization of information-theoretic measures, such as transfer entropy and active information storage in case of some datasets. In this paper, we take a closer look at these measures, using Kraskov-Grassberger-Stögbauer estimator with optimized parameters. We experiment with four datasets differing in the data complexity, and discover interesting differences, compared to the previous work, such as more complex behavior of the information-theoretic measures. We also investigate the effect of reservoir orthogonalization, that has been shown earlier to maximize memory capacity, on the prediction accuracy and the above mentioned measures.

Recently, we systematically investigated short-term memory of an echo state network fed with a scalar random input, using computational simulations. We studied the effect of proper reservoir initialization and its subsequent orthogonalization, using two similar gradient descent iterative procedures. It was shown that the measure defined by Jaeger as memory capacity (MC) approached its theoretical limit for orthogonalized reservoirs in most cases up to size 100 units, and at the same time, it drove the reservoir dynamics toward the critical regime. In this paper, we investigate the effect of both orthogonalization procedures for larger reservoirs, up to 1000 units. We observe almost perfect maximization of MC in both procedures for roughly up to 500 units, beyond which the MC gradually becomes suboptimal, despite our effort to find optimal parameters. We also looked at the input weights scaling that also effects the MC and we confirmed the previously encountered finding that smaller input weights allow higher maxima for MC to be reached, with the reservoir neurons operating in the linear regime. Last but not least, we show that both procedures work well, one better than the other, even in the case of very sparse reservoirs.

We use computational simulations to analyse the behavior of the recently proposed Bidirectional Activation-based Learning algorithm (BAL) which was inspired by the Generalized Recirculation algorithm (GeneRec). Both algorithms avoid biologically implausible backpropagation of the error signal, and instead use propagation of neuron activations, which drive the weight updates, using only local variables. We take a closer look at the 4-2-4 autoencoder task for which, despite the task simplicity, reliable convergence could not be achieved by either of the two models. We propose the learning mode with two, significantly different, learning rates (BAL2) that leads to considerably more successful task learning. We also analyze various factors, related to hidden activations, that contribute to further increase of the learning success. In addition, we test BAL2 also on the large scale database of handwritten digits, in which it yields relatively good performance.

Recently there has been an outburst of interest in extending topographic maps of vectorial data to more general data structures, such as sequences or trees. However, there is no general consensus as to how best to process sequences using topographic maps, and this topic remains an active focus of neurocomputational research. The representational capabilities and internal representations of the models are not well understood. Here, we rigorously analyze a generalization of the self-organizing map (SOM) for processing sequential data, recursive SOM(RecSOM) (Voegtlin, 2002), as a nonautonomous dynamical system consisting of a set of fixed input maps. We argue that contractive fixed-input maps are likely to produce Markovian organizations of receptive fields on the RecSOM map. We derive bounds on parameter beta (weighting the importance of importing past information when processing sequences) under which contractiveness of the fixed-input maps is guaranteed. Some generalizations of SOM contain a dynamic module responsible for processing temporal contexts as an integral part of the model. We show that Markovian topographic maps of sequential data can be produced using a simple fixed (nonadaptable) dynamic module externally feeding a standard topographic model designed to process static vectorial data of fixed dimensionality (e.g., SOM). However, by allowing trainable feedback connections, one can obtain Markovian maps with superior memory depth and topography preservation. We elaborate on the importance of non-Markovian organizations in topographic maps of sequential data.