Recent empirical research has shown conclusive advantages of multimodal interaction over speech-only interaction for map-based tasks. This paper describes a multimodal language processing architecture which supports interfaces allowing simultaneous input from speech and gesture recognition. Integration of spoken and gestural input is driven by unification of typed feature structures representing the semantic contributions of the different modes. This integration method allows the component modalities to mutually compensate for each others' errors. It is implemented in Quick-Set, a multimodal (pen/voice) system that enables users to set up and control distributed interactive simulations.

This paper advances two approaches to integrating handwritten character segmentation and recognition within one system, where the underlying function is learned by a backpropagation neural network. Integrated segmentation and recognition is necessary when characters overlap or touch, or when an individual character is broken up. The first approach exhaustively scans a field of characters, effectively creating a possible segmentation at each scan point. A neural net is trained to both identify when its input window is centered over a character, and if it is, to classify the character. This approach is similar to most recently advanced approaches to integrating segmentation and recognition, and has the common flaw of generating too many possible segmentations to be truly efficient. The second approach overcomes this weakness without reducing accuracy by training a neural network to mimic the ballistic and corrective saccades (eye movements) of human vision. A single neural net learns to jump from character to character, making corrective jumps when necessary, and to classify the centered character when properly fixated. The significant aspect of this system is that the neural net learns to both control what is in its input window as well as to recognize what is in the window. High accuracy results are reported for a standard database of handprinted digits for both approaches.

Notebook computers, using stylus input, are currently a hot topic among PC manufacturers.Handwriting recognition may be an important component of such systems, but only if everyday sloppy handwriting can be accommodated.If recognizes require unnaturally neat or boxed character input, such systems may fail in the marketplace.Neural nets have shown excellent performance at handwriting recognition.I present three neural net approaches to recognizing lines of English texti one using 2D image input, one using stroke sequence input, and one using context to combine the outputs of the other two networks.These networks can be combined to form a recognition engine that will handle natural lines of handwritten English text, including handprint, cursive script, and mixtures of both.

We report on results of training backpropagation nets with samples of hand-printed digits scanned off of bank checks and hand-printed letters interactively entered into a computer through a stylus digitizer. Generalization results are reported as a function of training set size and network capacity. Given a large training set, and a net with sufficient capacity to achieve high performance on the training set, nets typically achieved error rates of 4-5% at a 0% reject rate and 1-2% at a 10% reject rate. The topology and capacity of the system, as measured by the number of connections in the net, have surprisingly little effect on generalization. For those developing hand-printed character recognition systems, these results suggest that a large and representative training sample may be the single, most important factor in achieving high recognition accuracy. Benefits of reducing the number of net connections, other than improving generalization, are discussed.