Theoretical and practical motives alike have prompted recent investigations of multimodal speech perception. Theoretically, multimodal studies have extended the conceptualization of perceptual organization beyond the familiar modality-bound accounts deriving from Gestalt psychology. Practically, such investigations have been driven by a need to understand the proficiency of multimodal speech perception using an electrocochlear prosthesis for hearing. In each domain, studies have shown that perceptual organization of speech can occur even when the perceiver's auditory experience departs from natural speech qualities. Accordingly, our research examined auditor-visual multimodal integration of videotaped faces and selected acoustic constituents of speech signals, each realized as a single sinewave tone accompanying a video image of an articulating face. The single tone reproduced the frequency and amplitude of the phonatory cycle or of one of the lower three oral formants. Our results showed a distinct advantage for the condition pairing the video image of the face with a sinewave replicating the second formant, despite its unnatural timbre and its presentation in acoustic isolation from the rest of the speech signal. Perceptual coherence of multimodal speech in these circumstances is established when the two modalities concurrently specify the same underlying phonetic attributes.

A series of experiments was conducted in which listeners were presented with audio-visual sentences in a transcription task.The visual components of the stimuli consisted of a male talker's face.The acoustic components consisted of : (1) natural speech (2) envelope-shaped noise which preserved the duration and amplitude of the original speech waveform and (3) various types of sinewave speech signals that followed the formant frequencies of a natural utterance.Sinewave speech is a skeletonized version of a natural utterance which contains frequency and amplitude variation of the formants, but lacks any fine-grained acoustic structure of speech.Intelligibility of the present set of sinewave sentences was relatively low in contrast to previous findings (Remez, Rubin, Pisoni, & Carrell 1981).However, intelligibility was greatly increased when visual information from a talkers face was presented along with the auditory stimuli.Further experiments demonstrated that the intelligibility of single tones increased differentially depending on which formant analog was presented.It was predicted that the increase in intelligibility for the sinewave speech with an added video display would be greater than the gain observed with envelope-shaped noise.This prediction is based on the assumption that the information-bearing phonetic properties of spoken utterances are preserved in the audio+visual sine-wave conditions.This prediction was borne out for the tonal analog of the second formant (T2), but not the tonal analog of the first formant (T1) or third formant (T3), suggesting that the information contained in the T2 analog is relevant for audiovisual integration.

Acoustical analyses were carried out on the digits 0-9 spoken by two male talkers in the quiet and in 90 dB SPL of masking noise in their headphones. The results replicated previous studies demonstrating reliable increases in amplitude, duration and vocal pitch while talking in noise. We also found reliable differences in the tilt of the short-term spectrum of consonants and vowels. The results are discussed in terms of: (1) the development of algorithms for recognition of speech in noise; (2) the nature of the acoustic changes that take place when talkers produce speech under adverse conditions such as noise, stress or high cognitive load; and, (3) the role of training and feedback in controlling and modifying a talker's speech to improve performance of current speech recognizers.

As the use of voice response systems employing synthetic speech becomes more widespread in consumer products, industrial and military applications, and aids for the handicapped, it will be necessary to develop reliable methods of comparing different synthesis systems and of assessing how human observers perceive and respond to the speech generated by these systems. The selection of a specific voice response system for a particular application depends on a wide variety of factors only one of which is the inherent intelligibility of the speech generated by the synthesis routines. In this paper, we describe the results of several studies that applied measures of phoneme intelligibility, word recognition, and comprehension to assess the perception of synthetic speech. Several techniques were used to compare performance of different synthesis systems with natural speech and to learn more about how humans perceive synthetic speech generated by rule. Our findings suggest that the perception of synthetic speech depends on an interaction of several factors including the acoustic-phonetic properties of the speech signal, the requirements of the perceptual task, and the previous experience of the listener. Differences in perception between natural speech and high-quality synthetic speech appear to be related to the redundancy of the acoustic-phonetic information encoded in the speech signal.

With the rapid increase in the use of voice response systems in commercial, industrial, military, and educational applications, it has become important to understand how humans interact with devices that produce synthetic speech output. This paper describes a series of experiments that have examined the differences in perception between natural and synthetic speech. Our results demonstrate that important perceptual and cognitive limitations are imposed when synthetic speech is used in a variety of psychological tasks ranging from phoneme recognition to word recognition to spoken language comprehension. Moreover, these differences are manifest not only in terms of measures of response accuracy but also in estimates of the cognitive processing time required to execute responses to synthetic speech signals.

Telesensory Systems, Inc. (TSI) has developed an English text-to-speech system based on MITalk [1], but which runs in real time on microprocessor hardware and in main memory. Important structural parts of the MITalk-79 algorithm that are omitted in the TSI system include: (1) MITalk's parser and its sophisticated phrase-structure dependent fundamental frequency generator, and (2) The morpheme decomposition module with its 12,000 morpheme lexicon. These omitted modules, which use about 50% of MITalk's memory, have been replaced in the TSI system by an 1100 word dictionary and a simple "hat and declination" F0 routine. Results from intelligibility and comprehension tests identical to those used in evaluating MITalk [10] indicate that for common paragraphic text listener performance on TSI's speech is comparable to performance on MITalk speech. Analysis of the results suggests that accurate acoustic realization of segmental information is the crucial factor in intelligibility, with simple F0 patterns and conventional letter-tophoneme conversion adequate for many purposes.

A detailed perceptual evaluation of MITalk has been carried out to obtain measures of intelligibility and comprehension of the synthetic speech produced by the system. Phoneme recognition as measured by the Modified Rhyme Test showed an average error rate of only 6.9% overall. Correct word recognition assessed with a set of Harvard psychoacoustic sentences was 93.2% whereas performance with a set of semantically anomalous sentences developed at Haskins Laboratories was 78.7%. Comprehension performance on multiple-choice questions designed to assess understanding of continuous fluent synthetic speech showed good-to-excellent levels of performance and continued improvement over time. No major problems were encountered during the generation of the test materials used in this evaluation nor were any serious errors identified in the conceptual design of the MIT text-to-speech system. These perceptual results suggest that very high-quality and natural sounding synthetic speech can now be produced automatically from unrestricted English text and that such a text-to-speech system could well be implemented in applied settings such as devices for computer aided instruction or a reading machine for the blind in the very near future.