This special issue illustrates both the scientific trends of the early work in recurrent neural networks, and the mathematics of training when at least some recurrent terms of the network derivatives can be non-zero. Herein is a brief description of each of the papers. We have organized this description into two parts. The first part contains the papers that are mainly theoretical, and the second part contains the papers that are mainly applications. The order of papers is alphabetical by first author.

Second-order recurrent networks that recognize simple finite state languages over {0,1}* are induced from positive and negative examples. Using the complete gradient of the recurrent network and sufficient training examples to constrain the definition of the language to be induced, solutions are obtained that correctly recognize strings of arbitrary length.

Results are presented from an experiment on talker-independent word recognition in which three experimental parameters were manipulated. Those parameters were 1) whether the word model had a mean vector and a variance vector, or a mean vector and a covariance matrix, at each time frame; 2) whether a single estimate was made of the word model or whether there was one re-estimate; and 3) whether word recognition was based on a Pass 1 distance score or on a novel Pass 2 log odds score. The results were 1) a multivariate word model did not do better than a word model based on a sum of univariate distributions; 2) a re-estimated word model did better than a word model based on initial estimates; and 3) the Pass 2 log odds score did better than the Pass 1 distance score.