Lifelike talking faces for interactive services are an exciting new modality for man-machine interactions. Recent developments in speech synthesis and computer animation enable the real-time synthesis of faces that look and behave like real people, opening opportunities to make interactions with computers more like face-to-face conversations. This paper focuses on the technologies for creating lifelike talking heads, illustrating the two main approaches: model-based animations and sample-based animations. The traditional model-based approach uses three-dimensional wire-frame models, which can be animated from high-level parameters such as muscle actions, lip postures, and facial expressions. The sample-based approach, on the other hand, concatenates segments of recorded videos, instead of trying to model the dynamics of the animations in detail. Recent advances in image analysis enable the creation of large databases of mouth and eye images, suited for sample-based animations. The sample-based approach tends to generate more naturally looking animations at the expense of a larger size and less flexibility than the model-based animations. Beside lip articulation, a talking head must show appropriate head movements, in order to appear natural. We illustrate how such "visual prosody" is analyzed and added to the animations. Finally, we present four applications where the use of face animation in interactive services results in engaging user interfaces and an increased level of trust between user and machine. Using an RTP-based protocol, face animation can be driven with only 800 bits/s in addition to the rate for transmitting audio.

The AT&T text-to-speech (TTS) synthesis system has been used as a framework for experimenting with a perceptuallyguided data-driven approach t o s p e e c h s y n thesis, with primary focus on data-driven elements in the \back end".Statistical training techniques applied to a large corpus are used to make decisions about predicted speech e v ents and selected speech i n ventory units.Our recent a d v ances in automatic phonetic and prosodic labeling and a new faster harmonic plus noise model (HNM) and unit preselection implementations have signi cantly improved TTS quality and speeded up both development time and runtime.

In the future, the world of telecommunications will be vastly different than it is today. The driving force will be the seamless integration of real time communications (e.g. voice, video, music, etc.) and data into a single network, with ubiquitous access to that network anywhere, anytime, and by a wide range of devices. The only currently available ubiquitous access device to the network is the telephone, and the only ubiquitous user access technology mode is spoken voice commands and natural language dialogues with machines. In the future, new access devices and modes will augment speech in this role, but are unlikely to supplant the telephone and access by speech anytime soon. Speech technologies have progressed to the point where they are now viable for a broad range of communications services, including: compression of speech for use over wired and wireless networks; speech synthesis, recognition, and understanding for dialogue access to information, people, and messaging; and speaker verification for secure access to information and services. The paper provides brief overviews of these technologies, discusses some of the unique properties of wireless, plain old telephone service, and Internet protocol networks that make voice communication and control problematic, and describes the types of voice services available in the past and today, and those that we foresee becoming available over the next several years.

In an effort to select a speech representation for our next generation concatenative text-to-speech synthesizer, the use of two candidates is investigated; TD-PSOLA and the harmonic plus noise model, HNM. A formal listening test has been conducted and the two candidates have been rated regarding intelligibility, naturalness and pleasantness. Ability for database compression and computational load is also discussed. The results show that HNM consistently outperforms TD-PSOLA in all the above features except for computational load. HNM allows for high-quality speech synthesis without smoothing problems at the segmental boundaries and without buzziness or other oddities observed with TD-PSOLA.

Software engineering for research and development in the area of signal processing is by no means unimportant. For speech processing, in particular, it should be a priority: given the intrinsic complexity of text-to-speech or recognition systems, there is little hope to do state-of-the-art research without solid and extensible code. This paper describes a simple and efficient methodology for the design of maximally reusable and extensible software components for speech and signal processing. The resulting programming paradigm allows software components to be advantageously combined with each other in a way that recalls the concept of hardware plug-andplay, without the need for incorporating complex schedulers to control data flows. It has been successfully used for the design of a software library for high-level speech processing systems at AT&T Labs, as well as for several other large-scale software projects.

In this paper we present a high-quality text-to-speech system using diphones. The system is based on a Harmonic plus Noise (HNM) representation of the speech signal. HNM is a pitch-synchronous analysis-synthesis system but does not require pitch marks to be determined as necessary in PSOLA-based methods. HNM assumes the speech signal to be composed of a periodic part and a stochastic part. As a result, different prosody and spectral envelope modification methods can be applied to each part, yielding more natural-sounding synthetic speech. The fully parametric representation of speech using HNM also provides a straightforward way of smoothing diphone boundaries. Informal listening tests, using natural prosody, have shown that the synthetic speech quality is close to the quality of the original sentences, without smoothing problems and without buzziness or other oddities observed with other speech representations used for TTS. 1. INTRODUCTION Many current Text-To-Speech (TTS) systems a...

This paper reviews methods for mapping from the acoustical properties of a speech signal to the geometry of the vocal tract that generated the signal. Such mapping techniques are studied for their potential application in speech synthesis, coding, and recognition. Mathematically, the estimation of the vocal tract shape from its output speech is a so-called inverse problem, where the direct problem is the synthesis of speech from a given time-varying geometry of the vocal tract and glottis. Different mappings are discussed: mapping via articulatory codebooks, mapping by nonlinear regression, mapping by basis functions, and mapping by neural networks. Besides being nonlinear, the acoustic-to-geometry mapping is also nonunique, i.e., more than one tract geometry might produce the same speech spectrum. The authors show how this nonuniqueness can be alleviated by imposing continuity constraints.>

Item does not contain fulltext

The authors present an LPC (linear predictive coding) speech coder that classifies speech into periodic and nonperiodic intervals. The coder uses a stochastic codebook, as in CELP (code excited linear prediction), to synthesize nonperiodic speech and single-pulse excitation (SPE) to synthesize periodic speech. This coder is called SPE-CELP. The optimization of the single-pulse excitation is based on a new algorithm that determines the time instants of pitch periods within a short interval of periodic speech of approximately 32 ms. It thus causes a coding delay that is acceptable for many applications. Results using a fixed delta-impulse shape and time-varying pulse shapes obtained from codebooks are reported.>

A pitch-synchronous scheme for analysis/synthesis of speech is presented that uses a parametric model of the time-derivative of the glottal area function. Improved articulatory codebooks are used to start up an analysis-by-synthesis procedure to adapt parameters representing the vocal tract shape and the glottal geometry. The authors also present results of multiframe parameter optimization of articulatory parameters for potential application to low-bit-rate speech coding. In this scheme, they optimize the parameters once for each appropriately selected segment of speech and interpolate to obtain parameter values at intermediate time-instants. They study three different vocal-tract representations for linear and arctan interpolation functions for multiframe parameter optimization. Experiments indicate that multiframe parameter optimization results in some degradation in speech quality as compared to single-frame parameter optimization.>

The authors describe an efficient procedure for acoustic-to-articulatory parameter mapping using neural networks. An assembly of multilayer perceptrons, each designated to a specific region in the articulatory space, is used to map acoustic parameters of the speech into tract areas. The training of this model is executed in two stages; in the first stage a codebook of suitably normalized articulatory parameters is used and in the second stage real speech data are used to further improve the mapping. In general, acoustic-to-articulatory parameter mapping is nonunique; several vocal tract shapes can result in identical spectral envelopes. The model accommodates this ambiguity. During synthesis, neural networks are selected by dynamic programming using a criterion that ensures smoothly varying vocal tract shapes while maintaining a good spectral match.>

An improved method for generating articulatory codebooks is introduced. The distance measure used for codebook access is optimized. The improvements are evaluated and compared in terms of a perceptually motivated formant distance measure. It is found that the codebooks derived from the articulatory model of P. Mermelstein (1973) are more robust against model/speaker mismatches than the codebooks derived from the model of C. Coker (1976).>

The authors introduce a delayed-decision method based on dynamic programming for accessing such codebooks. The idea is to retrieve a sequence of shapes for a given sequence of spectra. It is shown that dynamic programming provides smooth parameter trajectories and phonetically reasonable tract shapes, even for difficult segments of speech, for example, during low-energy transitional portions of an utterance. >

The authors address the problem of how to accommodate varying speaker characteristics by selecting appropriate control parameters for a vocal tract/vocal cord model. This problem is basic to speech coding based on articulatory modeling. Experience shows that, for example, nominal vocal-tract length of the speaker, speech amplitude scale factor, as well as some glottal characteristics, cannot be accurately estimated on a frame-by-frame basis. The solution to this problem is found by a search procedure employing several frames of speech for estimating global speaker characteristics, in the form of scaling parameters, which is separate from the local (frame-to-frame) optimization of other parameters.>

This paper proposes the use of a vocal cord and tract model for speech coding at bit rates below 4.8 kb/s. For this, a key requirement is the ability to derive model parameters from an input speech signal. Our approach to this problem employs an acoustic analysis front-end, a linked codebook of vocal-tract configurations and related acoustic characteristics, and an optimizing articulatory synthesizer. While the acoustic front-end is relatively straight-forward involving LPC, pitch, and voicing analyses, the codebook design and usage, as well as the specific method for optimizing the model parameters are new. The codebook is intended to provide good starting values for an iterative optimization, thus alleviating the problem of locking on to a locally optimum solution. In a first stage of optimization, the best vocal tract configuration found in the codebook is refined by varying only the vocal tract parameters. Then, in a second stage of optimization, the best match is found between the glottal waveform of the model and the inverse filtered input speech.

High quality speech at low bit rates (e.g., 2400 bits per second) is one of the important objectives of current speech research. As part of long range activity on this problem we have developed an efficient computer program that will serve as a tool for investigating whether articulatory speech synthesis may achieve this low bit rate. At a sampling frequency of 20 kHz the most comprehensive version of the program, including nasality and frication, runs at about five times real time on a Cray-1 computer.