We study noisy computation in randomly generated k-ary Boolean formulas. We establish bounds on the noise level above which the results of computation by random formulas are not reliable. This bound is saturated by formulas constructed from a single majority-like gate. We show that these gates can be used to compute any Boolean function reliably below the noise bound.

The problem of resource allocation in sparse graphs with real variables is studied using methods of statistical physics. An efficient distributed algorithm is devised on the basis of insight gained from the analysis and is examined using numerical simulations, showing excellent performance and full agreement with the theoretical results.

We analyze Gallager codes by employing a simple mean-field approxi(cid:173) mation that distorts the model geometry and preserves important interac(cid:173) tions between sites. The method naturally recovers the probability prop(cid:173) agation decoding algorithm as an extremization of a proper free-energy. We find a thermodynamic phase transition that coincides with informa(cid:173) tion theoretical upper-bounds and explain the practical code performance in terms of the free-energy landscape.

An analytic investigation of the average case learning and generalization properties of radial basis function (RBFs) networks is presented, utilizing online gradient descent as the learning rule. The analytic method employed allows both the calculation of generalization error and the examination of the internal dynamics of the network. The generalization error and internal dynamics are then used to examine the role of the learning rate and the specialization of the hidden units, which gives insight into decreasing the time required for training. The realizable and some over realizable cases are studied in detail: the phase of learning in which the hidden units are unspecialized (symmetric phase) and the phase in which asymptotic convergence occurs are analyzed, and their typical properties found. Finally, simulations are performed that strongly confirm the analytic results.

The generalization error is a widely used performance measure employed in the analysis of adaptive learning systems. This measure is generally critically dependent on the knowledge that the system is given about the problem it is trying to learn. In this paper we examine to what extent it is necessarily the case that an increase in the knowledge that the system has about the problem will reduce the generalization error. Using the standard definition of the generalization error, we present simple cases for which the intuitive idea of “reducivity”—that more knowledge will improve generalization—does not hold. Under a simple approximation, however, we find conditions to satisfy “reducivity.” Finally, we calculate the effect of a specific constraint on the generalization error of the linear perceptron, in which the signs of the weight components are fixed. This particular restriction results in a significant improvement in generalization performance.

We examine the fluctuations in the test error induced by random, finite, training and test sets for the linear perceptron of input dimension n with a spherically constrained weight vector. This variance enables us to address such issues as the partitioning of a data set into a test and training set. We find that the optimal assignment of the test set size scales with n2/3.

The two-layer radial basis function network, with fixed centers of the basis functions, is analyzed within a stochastic training paradigm. Various definitions of generalization error are considered, and two such definitions are employed in deriving generic learning curves and generalization properties, both with and without a weight decay term. The generalization error is shown analytically to be related to the evidence and, via the evidence, to the prediction error and free energy. The generalization behavior is explored; the generic learning curve is found to be inversely proportional to the number of training pairs presented. Optimization of training is considered by minimizing the generalization error with respect to the free parameters of the training algorithms. Finally, the effect of the joint activations between hidden-layer units is examined and shown to speed training.

Using a statistical mechanical formalism we calculate the evidence, generalisation error and consistency measure for a linear percep(cid:173) tron trained and tested on a set of examples generated by a non linear teacher. The teacher is said to be unrealisable because the student can never model it without error. Our model allows us to interpolate between the known case of a linear teacher, and an un(cid:173) realisable, nonlinear teacher. A comparison of the hyperparameters which maximise the evidence with those that optimise the perfor(cid:173) mance measures reveals that, in the non-linear case, the evidence procedure is a misleading guide to optimising performance. Finally, we explore the extent to which the evidence procedure is unreliable and find that, despite being sub-optimal, in some circumstances it might be a useful method for fixing the hyperparameters.

In supervised learning, learning from queries rather than from random examples can improve generalization performance signif(cid:173) icantly. We study the performance of query learning for problems where the student cannot learn the teacher perfectly, which occur frequently in practice. As a prototypical scenario of this kind, we consider a linear perceptron student learning a binary perceptron teacher. Two kinds of queries for maximum information gain, i.e., minimum entropy, are investigated: Minimum student space en(cid:173) tropy (MSSE) queries, which are appropriate if the teacher space is unknown, and minimum teacher space entropy (MTSE) queries, which can be used if the teacher space is assumed to be known, but a student of a simpler form has deliberately been chosen. We find that for MSSE queries, the structure of the student space deter(cid:173) mines the efficacy of query learning, whereas MTSE queries lead to a higher generalization error than random examples, due to a lack of feedback about the progress of the student in the way queries are selected.

This paper proposes a new pruning method based on merging neurons with similar functional behavior which is defined by the internal representations of each neuron for the entire training set. Classification of neurons by their functional behavior with respect to the input vectors provides a powerful tool for pruning neurons and connections, thus reducing the network complexity and increasing its generalization capability. The most remarkable property of this pruning scheme is its ability to preserve net functionality by transferring the role of every removed neuron to the most fitted neuron of the surviving ones, using a unique merging and compensation procedure. The implementation of the proposed method is demonstrated using a detailed numerical example and its performance is examined by a statistical measure calculated by repeating the training procedure several times. The influence of parameter selection on pruning performance and generalization ability is discussed and demonstrated by examining statistical results.

A modification of the binary weight CHIR algorithm is presented, whereby a zero state is added to the possible binary weight states. This method allows solutions with reduced connectivity to be obtained, by offering disconnections in addition to the excitatory and inhibitory connections. The algorithm has been examined via extensive computer simulations for the restricted cases of parity, symmetry, and teacher problems, which show convergence rates similar to those presented for the binary CHIR2 algorithm, but with reduced connectivity. Moreover, this method expands the set of problems solvable via the binary weight network configuration with no additional parameter requirements.

The Minimal Trajectory (MINT) algorithm for training recurrent neural networks with a stable end point is based on an algorithmic search for the systems’ representations in the neighbourhood of the minimal trajectory connecting the input-output representations. The said representations appear to be the most probable set for solving the global perceptron problem related to the common weight matrix, connecting all representations of successive time steps in a recurrent discrete neural networks. The search for a proper set of system representations is aided by representation modification rules similar to those presented in our former paper,1 aimed to support contributing hidden and non-end-point representations while supressing non-contributing ones. Similar representation modification rules were used in other training methods for feed-forward networks,2–4 based on modification of the internal representations. A feed-forward version of the MINT algorithm will be presented in another paper.5 Once a proper set of system representations is chosen, the weight matrix is then modified accordingly, via the Perceptron Learning Rule (PLR) to obtain the proper input-output relation. Computer simulations carried out for the restricted cases of parity and teacher-net problems show rapid convergence of the algorithm in comparison with other existing algorithms, together with modest memory requirements.

Learning by Choice of Internal Representations (CHIR) is a training algorithm presented by Grossman et al.1 based on modification of the Internal Representations (IR) along side of the direct weight matrix modification performed in conventional training methods. This algorithm was presented in several versions aimed to tackle the various training problems of nets with continuous and binary weights, multilayer and multi-output-neuron nets and training without storing the Internal Representations. The capability of one of these versions, the CHIR2 algorithm, to tackle multilayer training tasks of nets with continuous input vectors is examined in this paper. A comparison between the performance of this algorithm and of the Backpropagation algorithm2 is carried out via extensive computer simulations for the “two-spirals” problem, aimed to classify two classes of dots forming two intertwined spirals. The CHIR24 algorithm shows a rapid convergence rate for this problem, an order of magnitude faster than the results reported for the BP training algorithm (as well as those obtained by us) regarding the same training problem and network architecture.11 Moreover, the CHIR2 algorithm finds solution nets for the above mentioned problem with reduced architectures, reported as hard to solve by the BP training algorithm.11