Abstract Unlike automatic speech recognition systems, humans canunderstand speech when other competing sounds are presentAlthough the theory of auditory scene analysis (ASA)may helpto explain this ability, some perceptual experiments show fu-sion of the speech signal under circumstances in which ASAprinciples might be expected to cause segregation. We pro-pose a model of multi-resolution ASA that uses both high- andlow- resolution representations of the auditory signal in paral-lel in order to resolve this conflict. The use of parallel repre-sentationsreduces variabilityforpattern-matching whileretain-ing the ability to identify and segregate low-level features ofthe signal. An important feature of the model is the assump-tion that features of the auditory signal are fused together un-less there is good reason to segregate them. Speech is recog-nised by matching the low-resolution representation to previ-ously learned speech templates without prior segregation of thesignal into separate perceptual streams; this contrasts with theappr oach gener a l l y used by comput at i onal m odels of A S A . Wedescribe an implementation of the multi-resolution model, us-ing hidden Markov models, that illustrates the feasibility ofthis approach and achieves much higher identification perfor-mance than standard techniques used for computer recognitionof speech mixed with other sounds.

Audio-visual speech recognition systems can be divided into systems that integrate audio-visual features before decisions are made (feature fusion) and those that integrate decisions of separate recognisers for each modality (decision fusion).

A commonly held view of auditory scene analysis is that complex auditory environments are segregated into separate perceptual streams using primitive cues that can be attended to separately. We argue that this view is inconsistent with the majority of perceptual data reported in the literature and propose an alternative model that is based on a primary, low resolution signal representation used in a passive pattern matching stage, augmented by secondary, high resolution representations that can be used in an active pattern matching stage to formulate hypotheses about the auditory scene.

An improvement of the speech spectrogram based on the method of reassignment is presented. This method consists of moving each point of the spectrogram to a new point that represents the distribution of the energy in the time-frequency window more accurately. Examples of natural speech show an improvement of the energy localization in both time and frequency domains. This allows a better description of speech features.

Modulation maps provide an effective method for the segregation of voiced speech sounds from competing background activity. The maps are constructed by computing modulation spectra in a bank of auditory filters. If the modulation spectra are computed using a conventional DFT, windows of 200 ms duration are necessary. The reassigned spectrum, a new time frequency representation (Kodera et al., 1978, and Auger and Flandrin, 1995), allows a reduction in the window size to 50 ms without loss of performance. The algorithm is tested on a 'double vowel' identification task that has been used extensively in psychophysical experiments.

A model that is able to predict human performance in a simultaneous glide recognition task is described. The model combines a primitive, F0 guided, segregation stage and a schema driven stage with a heuristic that models whether listeners perceive a single or two simultaneous sounds. Introduction Previous studies [1,2,3] suggest that human listeners use simple cues, such as signal harmonicity, speaker location or segmental onset and offset to aid in the segregation of simultaneous sounds. These cues are called ‘primitive’ grouping cues because they can be applied without prior knowledge. The only heuristic is that segments in an ‘auditory scene’ that share the same features are likely to be produced by the same speaker. In addition to the primitive segregation process human listeners use high-level knowledge, schemata, to deal with mixtures of sounds [1]. One of the most intensively studied primitive grouping cues is harmonicity. Figure 1 shows human performance for a recognition task involving simultaneous vowels. Each of the panels shows the percentage of pairs that listeners correctly recognise. The stimuli were pairs of the French long vowels /#,G,K,Q,W,[/. One of the vowels always had a fundamental frequency (F0) of 100Hz, the fundamental frequency of the second vowel is plotted along the x-axis. The only primitive segregation cue is the vowel fundamental frequency. The three panels show subject performance for vowels of 200ms, 100ms and 50ms duration. For signals of at least 100ms duration subject performance improves significantly as the frequency difference between the vowels increases. If signals are only 50ms long no improvement in performance is seen. The perceptual data is surprising considering the dynamic nature of speech sounds where stationary segments of more than 100ms duration are very rare. Another important feature that emerges from the data is that humans are able to recognise both constituents of a pair in around 65% of all cases independent of the signal duration and without any grouping cues. 100106112 126 2nd vowel F0 40 50 60 70 80 90 % p ai rs c or re ct 204.8ms 100106112 126 2nd vowel F0 40 50 60 70 80 90 Human Performance