Methods for speech compression aim at reducing the transmission bit rate while preserving the quality and intelligibility of speech. These objectives are antipodal in nature since higher compression presupposes preserving less information about the original speech signal. This paper presents a method for compressing speech based on polynomial approximations of the trajectories in time of various speech features (i.e., spectrum, gain, and pitch). The compression method can be integrated into frame-based speech coders, and can also be applied to features that can be represented as temporal series greater in duration than the frame interval. Theoretical issues and experimental results regarding this type of compression are addressed in this paper. Experimental implementation into a 2400 b/s standard speech coder is reported along with objective and subjective evaluations of operation in various noise environments. The new speech coder operates at a transmission rate of 1533 b/s, and for all noisy conditions tested performs better than the 2400 b/s standard speech coder

This paper presents a new method of multi-frame speech coding based upon polynomial approximation of speech feature trajectories incorporating multiple sensor signals from microphones, accelerometer, electro-glottograph, and micro-radar. The trajectory polynomial approximation exploits the inter-frame information redundancy encountered in natural speech. The trajectory method is applicable to features such as spectral parameters, gain, and pitch. The method is suitable for application to a frame vocoder to further reduce the transmission bit rate. Multiple transducers increase the intelligibility and quality of the coded speech in noisy environments. Experimental results are obtained by embedding the new method into an enhanced mixed-excitation linear prediction vocoder. The resulting vocoder operates at 1533 bps and preliminary intelligibility and quality tests show results comparable to those of the original 2400 bps vocoder. 1.

Recent efforts in the field of mobile computing are directed toward speech-enabling portable computers. This paper presents a method of multimodal interaction and an application which integrates speech and pen on mobile computers. The application is designed for documenting traffic accident diagrams by police. The novelty of this application is due to a) its method of fusing the speech and pen inputs, and b) its fully embedded speech engine. Preliminary experiments showed flexibility, versatility and increased naturalness and user satisfaction during multimodal interaction.

Communicating by voice with speech-enabled computer applications based on preprogrammed rule grammars suffers from constrained vocabulary and sentence structures. Deviations from the allowed language result in an unrecognized utterance that will not be understood and processed by the system. One way to alleviate this restriction consists in allowing the user to expand the computer's recognized and understood language by teaching the computer system new language knowledge. We present an adaptive dialog system capable of learning from users new words, phrases and sentences, and their corresponding meanings. User input incorporates multiple modalities, including speaking, typing, pointing, drawing and image capturing. The allowed language can thus be expanded in real time by users according to their preferences. By acquiring new language knowledge the system becomes more capable in specific tasks, although its language is still constrained.

Pervasive networking and sophisticated computing open opportunities for collaborative information processing independent of time and space. In this instance the information system becomes an enhancer of human intellect, as well as a mediator for communication among participants. The human user favors the sensory dimensions of sight, sound, and touch as primary channels of communication. Machines that can accommodate these modes promise flexibilities and functionalities that transcend the traditional mouse and keyboard. The paper describes research to establish human-computer interfaces that capture attributes of natural face-to-face communication. An experimental multimodal system is developed to study several aspects of natural style human-computer communication. While as yet primitive, the technologies of image and gaze processing, hands-free conversation, and force feedback tactile transduction are combined and used simultaneously for manipulating objects in a shared workspace. Software agents fuse the sensory signals to estimate and interpret user intent. Current areas of experimental application include disaster relief/crisis management, telemedicine/rehabilitation, and mobile office/wearable computers.

Performance of automatic speech recognition systems trained on close-talking data suffers when used in a distant-talking environment due to the mismatch in the training and testing conditions. Microphone array sound capture can reduce some mismatch by removing ambient noise and reverberation but offers insufficient improvement in performance. However, using array signal capture in conjunction with a hidden Markov model (HMM) adaptation on the clean-speech models can result in improved recognition accuracy. This paper describes an experiment in which the output of an 8-element microphone array system using MFA processing is used for speech recognition with LT-MLLR adaptation. The recognition is done in two passes. In the first pass, an HMM trained on clean data is used to recognize the speech. Using the results of this pass, the HMM model is adapted to the environment using the LT-MLLR algorithm. This adapted model, a product of MFA and LT-MLLR, results in improved recognition performance.

The performance of well-trained speech recognizers using high quality full bandwidth speech data is usually degraded when used in real world environments. In particular, telephone speech recognition is extremely difficult due to the limited bandwidth of the transmission channels. In this paper, neural network based adaptation methods are applied to telephone speech recognition and a new unsupervised model adaptation method is proposed. The advantage of the neural network based approach is that the retraining of speech recognizers for telephone speech is avoided. Furthermore, because the multi-layer neural network is able to compute nonlinear functions, it can accommodate for the non-linear mapping between full bandwidth speech and telephone speech. The new unsupervised model adaptation method does not require transcriptions and can be used with the neural networks. Experimental results on TIMIT/NTIMIT corpora show that the performance of the proposed methods is comparable to that of recognizers retained on telephone speech.

Matched filter array processing (MFA) has been shown to improve the signal-to-noise (SNR) quality for array speech capture in reverberant environments. However, under non-optimum conditions, MFA processing is computationally costly, and may produce little improvement or even subjective quality degradation as compared with simple time delay compensation (TDC). Appropriate truncation of the MFA filter bank is shown to reduce the computational burden without significantly reducing the capture SNR. This work attempts to find an optimal truncation time with respect to room size, wall absorption and the number of microphones used for the system. Simulations were conducted to evaluate MFA performance as a function of truncation length as these parameters were varied in situations typical of teleconferencing applications. It was demonstrated that judicious MFA truncation allows a reduction in computation load without sacrificing capture SNR.

Multimedia interfaces are rapidly evolving to facilitate human/machine communication. Most of the technologies on which they are based are, as yet, imperfect. But, the interfaces do begin to allow information exchange in ways familiar and comfortable to the human-principally through natural actions in the sensory dimensions of sight, sound and touch. Further, as digital networking becomes ubiquitous, the opportunity grows for collaborative work through conferenced computing. In this context the machine takes on the role of mediator in human/machine/human communication-the ideal being to extend the intellectual abilities of humans through access to distributed information resources and collective decision making. The challenge is how to design machine mediation so that it extends, not impedes, human abilities. This report describes evolving work to incorporate multimodal interfaces into a networked system for collaborative distributed computing. It also addresses strategies for quantifying the synergies that may be gained.

The Huge Microphone Array (HMA) project started in February 1994 to design, construct, and test a real-time 512-microphone array system and to develop algorithms for use on it. Analysis of known algorithms showed that signal-processing performance of over 6 Gigaflops would be required; at the same time, there was a need for "portability", i.e., fitting into a small van. These tradeoffs and many others have led to a unique design in both hardware and software. This paper presents the design and its justifications. Performance data for a few important algorithms relative to usage of processing-capability, response latency, and difficulty of programming are discussed.

VOICEMIMICSYSTEMUSINGANARTICULATORYCODEBOOKFORESTIMATIONOFVOCALTRACTSHAPES. Chennoukh, D. Sinder, G. Richard* and J.L. FlanaganCenter for Computer Aids for Industrial Pro ductivity (CAIP), Rutgers University,Piscataway, NJ 08855-1390, USA*Matra-Communication, rue J.P. Timbaud, 78392 Bois d'Arcy,FranceTel.+1 908-445-0080, FAX: +1 908 445-4775, E-mail:[email protected] mimic systems using articulatory co deb o oks re-quireaninitialestimateofthevo caltractshap einthe vicinity of the global optimum.For this purp ose,we need to gather a large set of corresp onding articu-latory and acoustic data in the articulatory co deb o ok.Thus, searching and accessing the co deb o ok b ecomesa dicult task.In this pap er, the design of an artic-ulatory co deb o ok is presented where an acoustic net-work sub-samples the acoustic space such that vo caltract mo del shap es are ordered and clustered in thenetwork according toacousticparameters.Anotherissue addressed in this pap er concerns estimating thetra jectory of vo cal tract shap es as they change withtime.Sincetheinversemappingfromacousticpa-rameters to mo del shap e do es not have a unique so-lution, several vo cal tract shap e variations are p ossi-ble.Therefore, a dynamic optimization of tra jectorieshas b een develop ed.This optimization uses dynamicprop erties of each articulatory parameter to estimatethe next p osition.1.INTRODUCTIONThestudyofsp eechp erceptionandpro duc-tion has b een enhanced in the last two decades by thedevelopment of computers capable of large amountofcomputation.As a result, Stevens' study towards anarticulatorymo delforsp eechrecognition-synthesisb ecomesmorefeasiblethanitwasintheearlysix-ties([9]).However, an incomplete understanding ofsp eechpro ductionandtheacousticsofpre-ventedusfromachievingStevens'goal.Thegoalwastomimicinputsp eechsignalsbyrecognition-synthesis using a mo del of the vo cal tract area func-tionthatcanmimicthesp eechsignalswithoutun-derstanding their structure or meaning.An early attempt at creating a complete computersimulation of articulatory mo del sp eech co ding usingan optimization technique was rep orted by Flanaganetal.([4]).Thesimulationiscalled\voice mimic.The voice mimic attempts to provide an articulatorydescription of the vo cal tract that corresp onds to anarbitrary natural sp eech input and to generate a syn-thetic signal that, within p erceptual accuracy, dupli-catesthenaturalone.Centraltoe ortisinverse mapping from an acoustic signal to an articu-latory description.However, acoustic-to-articulatorymappings are non-unique and, given a cost function,the optimization techniques converge only to a lo calextremum that may b e near the vicinity of the initialparameters.Therefore, one needs to cho ose accuratestartup parameters to initialize the optimization pro-cedure.Schro eter andSondhi([8]),whocontinuedalong the same lines of Flanagan et al.'s study, usedan articulatory co deb o ok prop osed earlier byAtal etal.([1]).Since a co deb o ok is used to obtain the rstestimates of the vo cal tract shap e that may pro ducea given combination of acoustic parameters, it mustbedesignedsuchthatitspansthenatural articula-tory space of a sp eaker.Furthermore, sampling of thespace must b e ne enough so that an acoustic entryalways exists very close to the global optimum.Suchco deb o oks require a large set of matching pairs of vo-cal tract and acoustic parameters.The complexityofsearching a large co deb o ok for all p ossible vo cal tractmo del shap es b ecomes an issue.For this reason, thevoice mimic system needs, in addition to a go o d artic-ulatory co deb o ok, an ecient pro cedure for accessingthe co deb o ok ([6],[7]).The numb er and p osition of the co deb o ok vectorsa ect the p erformance of the voice mimic system ac-cording to two compromising problems.On one hand,increasing the size of the co deb o ok increases the dif- cultyoftheaccesstaskand,onotherhand,reductionofthissizecomplicatestheinverseprob-lemsolution.Inthesecond sectionofthispap er,anew design of the articulatory co deb o ok is presentedfor which the inversion of the articulatory-to-acousticmapping is pro cessed during the building of the co de-b o ok.Thisco deb o okdesignallowsreal-time accessto the set of acoustically equivalent shap es, regardlessthe size of the co deb o ok.Sincetheinversemappingfromacousticparam-eterstomo delshap edo esnothaveauniquesolu-tion,severalvo caltractshap eariationsarep ossi-ble.Schro eter and Sondhi([7]) prop osed the use ofdynamicprogrammingtoestimatetheoptimaltra-jectory of the vo cal tract mo del shap e variation path.The dynamic programming requires a delay of severaldata frames for the sp eech output ([8]).In the thirdsection, a metho d is prop osed where the articulatoryparameters are estimated within one frame.Section

This report describes microphone arrays and parallel signal processing for high-quality sound capture in noisy, reverberant enclosures. The technique incorporates matched-filtering of individual sensors and parallel processing to provide spatial volume selectivity that mitigates effects of noise interference and multipath distortion. Truncated causal approximations to ideal matched-filters and arrays composed of randomly distributed transducers reduce complexity and improve the quality of sound capture. The method outperforms traditional delay-and-sum beamformers which provide only directional selectivity.

Most contemporary speech recognizers are designed to operate with close-talking speech and they work best in a quiet laboratory condition. There is an apparent need to render environment robustness to these systems. The objective of the paper is to explore utility of existing speech recognition technology in adverse "real-world" environments for distant-talking applications. A synergistic system consisting of microphone array and neural network (MANN) is utilized to mitigate environmental interference introduced by reverberation, ambient noise, and channel mismatch between training and testing conditions. The MANN system is evaluated with experiments on continuous distant-talking speech recognition. The results show that the MANN system elevates the word recognition accuracy to a level which is competitive with a retrained speech recognizer and that the neural network compensation performs better than some previously researched techniques.

An extensive speech database was collected in two experimental enclosures for research on microphone arrays. The first enclosure is a digitally controlled variable acoustics facility where the reverberation time can be adjusted from 1.7 s to 0.1 s, and the second is a regular hard-walled laboratory. Two different sets of data were collected to address source location and spatial volume selectivity issues. Preliminary results on Matched-filter processing and on Source location are reported for the new database. I. Introduction Performance of microphone sound pick up is degraded by deleterious properties of the acoustic environment, such as multipath distortion (reverberation) and ambient noise. The degradation becomes more prominent in a teleconferencing environment in which the microphone is positioned far away from the speaker. Besides, the ideal teleconference should feel as easy and natural as face-to-face communication with another person. This suggests hands-free sound capture w...

Performance of a delay-and-sum beamformer is typically degraded in a reverberant enclosure because the beam captures not only the desired source signal (direct path) but also all images along the beam axis. Matched-filter processing of microphone arrays is shown to improve the quality of sound capture in reverberant environments. This paper also reports the performance of matched-filter array processing in a real room. Significant improvements are shown over single microphones and the traditional delay-and-sum beamformer.

A number of technologies/spl minus/based upon the sensory modalities of sight, sound, and touch/spl minus/are becoming well enough established that their coordinated application can expand capabilities for human/machine interaction. This paper examines evolving advances in natural language communication with machines, video, and audio compression, spatially realistic conferencing, tactile interaction, and high-speed data communication. It considers the integration of these technologies into networked information systems to achieve synergistic benefits for the human user.>

Hands-free operation of speech processing equipment is sometimes desired so that the user is unencumbered by hand-held or body-worn microphones. This paper explores the use of array microphones to capture speech under adverse acoustic conditions, and provide input to a system for automatic speaker identification. The system is evaluated using reverberated speech signals, generated by a computer model of room acoustics and transduced by different simulated microphone-arrays. For comparison, the system is also evaluated using close-talking microphone input. It is found that 2-D matched-filter microphone arrays are capable of producing high speaker identification scores in a hostile acoustic environment, such as multipath distortion and competing noise sources. The paper also explores the influence of vector quantization techniques, codebook size, and order of cepstrum coefficients on the performance of the speaker identification system.>

It is shown that beamforming with microphone arrays can be used to significantly reduce the effects of noise and reverberation. The Frost algorithm and a new vectorized algorithm called the row-action projection (RAP) method are shown to yield approximately 11 dB of interference suppression. Computer simulations demonstrate the performance of the beamforming techniques for various locations of interference sources. The scenarios used for the computer simulations consist of a speech signal arriving normal to the array contaminated with speech interference from an unknown direction off the normal. The direction of the desired speaker is taken to the normal to the array for the simulation; however, it could be at any specified 'look direction'.>

A novel method of characterizing speech signals is presented in which the signal is modeled in terms of the mean radian frequency, the amplitude modulating function and the frequency modulating function of each frequency domain component (i.e., formants). This model defines the component structure and represents the local characteristics of the component and its bandwidth. A vowel classification task is presented where a distance metric of the relative modulation functions is used. This technique has been found to be robust to inter- and intraspeaker variability.>

The design and implementation of a digital teleconferencing system that integrates a number of speech technologies together with image and data facilities is described. The aim is to provide a variety of sophisticated communication features that are easy to learn and use. The system is called HuMaNet, for human-machine network. The system is controlled totally and interactively hands-free by natural speech. The system combines the technologies of speech recognition, text synthesis, and talker verification with autodirective microphone arrays, image compression, data, and hypertext management to provide high-quality audio and image conferencing. The present public-switched transport capacity provides 2B+D, or two 64 kB/s circuit-switched channels (2B), and one 16 Kb/s packet-switched channel (D).>

Automatic delay-steering of uniform arrays of microphones, under control of on-site microprocessors, offers attractive possibilities for large-group teleconferencing. But, steering to wave-arrival directions away from the normal reduces the useable bandwidth over which spatial discrimination is achieved. This report, using the line array as the expository vehicle, derives relations that specify the useful bandwidth as a function of the steering direction, and gives design formulae for the number and spacing of the receiving elements.

We describe digital encoding and automatic silenceediting techniques for PCM and ADPCM voice storage in the MC68000 microprocessor. We give program details for encoding, silence elimination, storage of active speech bursts and reconstitution of the signal upon read-back. Coding rates in the range 64 down to 16 kbits/s are implemented. We find that silent intervals corresponding, typically, to 20-40 percent of the duration of continuous input speech can be usefully eliminated. Available storage is thereby used more efficiently. The eliminated intervals are automatically reinserted upon read-back. We further demonstrate automatic scaling of the silent intervals in the reconstituted output. We confirm that a variation of ± 50 percent in the reinserted silent intervals is typically acceptable. This latitude can significantly alleviate buffer requirements in a packet transmission system for voice.

A rationale is advanced for digitally coding speech signals in terms of sub-bands of the total spectrum. The approach provides a means for controlling and reducing quantizing noise in the coding. Each sub-band is quantized with an accuracy (bit allocation) based upon perceptual criteria. As a result, the quality of the coded signal is improved over that obtained from a single full-band coding of the total spectrum. In one implementation, the individual sub-bands are low-pass translated before coding. In another, "integer-band" sampling is employed to alias the signal in an advantageous way before coding. Other possibilities extend to complex demodulation of the sub-bands, and to representing the subband signals in terms of envelopes and phase-derivatives. In all techniques, adaptive quantization is used for the coding, and a parsimonious allocation of bits is made across the bands. Computer simulations are made to demonstrate the signal qualities obtained for codings at 16 and 9.6 Kbits/sec.