We motivate the need for and describe the key components of real world audio indexing systems. In particular, we discuss the various flavors of such systems, the advantages and disadvantages of each, user interfaces, system architectures and evaluation issues. Throughout the paper, we give examples from our own experience of audio indexing using SpeechBot and its successor NewsTuner.

We present several novel approaches to the Out of Vocabulary (OOV) query problem for spoken audio: indexing based on syllable-like units called particles and query expansion according to acoustic confusability for a word index. We also examine linear and OOV-based combination of indexing schemes. We experiment on 75 h of broadcast news, comparing our techniques to a word index, a phoneme index and a phoneme index queried with phoneme sequences. Our results show that our approaches are superior to both a word index and a phoneme index for OOV words, and have comparable performance to the sequence of phonemes scheme. The particle system has worse performance than the acoustic query expansion scheme. The best system uses word queries for in-vocabulary words and a linear combination of the phoneme sequence scheme and acoustic query expansion for OOV words. Using the best possible weights for linear combination, this system improves the average precision from 0.35 for a word index to 0.40, a result only obtainable if the weights could be learnt on a development query set. The next best system used a word index for in-vocabulary words and the phoneme sequence system otherwise and had average precision of 0.39.

We explore the use of song lyrics for automatic indexing of music. Using lyrics mined from the Web, we apply a standard text processing technique to characterize their semantic content. We then determine artist similarity in this space. We found lyrics can be used to discover natural genre clusters. Experiments on a publicly available set of 399 artists showed that determining artist similarity using lyrics is better than random, but inferior to a state-of-the-art acoustic similarity technique. However the approaches made different errors, suggesting they could be profitably combined.

We present A new Web-based application, called the News Tuner, for searching and browsing large radio archives. While popular search engines provide means for finding text and images, our approach combines semantic and acoustic search for efficient retrieval of audio documents. Semantic search allows the user to retrieve stories for a given concept, while acoustic search allows random access within stored audio files. Our experiments on over 1700 programs show that our method is effective at quickly retrieving stories that would be difficult to find otherwise. The News Tuner paradigm is intended primarily for news and talk radio programs, however it may be applied to browsing and searching any spoken word audio content.

We present a novel approach to the out of vocabulary (OOV) query problem for audio indexing. Our technique first builds a word index for the audio using speech recognition. It then expands query words into in-vocabulary phrases according to intrinsic acoustic confusability and language model scores. The aim is to mimic the mistakes the speech recognizer makes when transcribing the OOV words.

As the Web transforms from a text-only medium into a more multimedia-rich medium, the need arises to perform searches based on the multimedia content. In this paper, we present an audio and video search engine to tackle this problem. The engine uses speech recognition technology to index spoken audio and video files from the World Wide Web (WWW) when no transcriptions are available. If transcriptions (even imperfect ones) are available, we can also take advantage of them to improve the indexing process. Our engine indexes several thousand talk and news radio shows covering a wide range of topics and speaking styles from a selection of public Web sites with multimedia archives. Our Web site is similar in spirit to normal Web search sites; it contains an index, not the actual multimedia content. The audio from these shows suffers in acoustic quality due to bandwidth limitations, coding, compression, and poor acoustic conditions. Our word error rate (WER) results using appropriately trained acoustic models show remarkable resilience to the high compression, although many factors combine to increase the average WERs over standard broadcast news benchmarks. We show that, even if the transcription is inaccurate, we can still achieve good retrieval performance for typical user queries (77.5%).

We present a method to compare songs based solely on their audio content. Our technique forms a signature for each song based on K-means clustering of spectral features. The signatures can then be compared using the Earth Mover's Distance [1] which allows comparison of histograms with disparate bins. Preliminary objective and subjective results on a database of over 8000 songs are encouraging. For 20 songs judged by two users, on average 2.5 out of the top 5 songs returned were judged similar. We also found that our measure is robust to simple corruption of the audio signal.

In this paper we present the application of a boosting classification algorithm to confidence scoring. We derive feature vectors from speech recognition lattices and feed them into a boosting classifier. This classifier combines hundreds of very simple `weak learners' and derives classification rules that can reduce the confidence error rate by up to 34%. We compare our results to those obtained using two other standard classification techniques, Support Vector Machines (SVMs) and Classification and Regression Trees (CART), and show significant improvements. Furthermore, the nature of the boosting algorithm allows us to combine the best single classifier and improve its performance. We present experimental results on real world corpora derived from our SpeechBot Web index http://www.speechbot.com and from the HUB4 DARPA evaluation sets. We believe these results have wide applicability to audio indexing and to acoustic and language modeling adaptation where word confidence scores can be used in iterative adaptation schemes. 1.

Systems to automatically provide a representative summary or 'key phrase' of a piece of music are described. For a 'rock' song with 'verse' and 'chorus' sections, we aim to return the chorus or in any case the most repeated and hence most memorable section. The techniques are less applicable to music with more complicated structure although possibly our general framework could still be used with different heuristics. Our process consists of three steps. First we parameterize the song into features. Next we use these features to discover the song structure, either by clustering fixed-length segments or by training a hidden Markov model (HMM) for the song. Finally, given this structure, we use heuristics to choose the key phrase. Results for summaries of 18 Beatles songs evaluated by ten users show that the technique based on clustering is superior to the HMM approach and to choosing the key phrase at random.

We have developed a speech recognition based audio search engine for indexing spoken documents found on the World Wide Web. Our site (http://www.compaq.com/speechbot) indexes around 20 news and talk radio shows covering a wide range of topics, speaking styles and acoustic conditions from a selection of public Web sites with multimedia archives. In this paper, we describe our system and its performance, focusing on the speech recognition and retrieval aspects. We describe our training procedure in some detail and report our historical error rate since the site launch. We also investigate the impact of Out Of Vocabulary (OOV) words. Finally we report the results of retrieval experiments which demonstrate that our system can index effectively.

In the machine learning research field several extensions of hidden Markov models (HMMs) have been proposed. In this paper we study their possibilities and potential benefits for the field of acoustic modeling. We describe preliminary experiments using an alternative modeling approach known as factorial hidden Markov models (FHMMs). We present these models as extensions of HMMs and detail a modification to the original formulation which seems to allow a more natural fit to speech. We present experimental results on the phonetically balanced TIMIT database comparing the performance of FHMMs with HMMs. We also study alternative feature representations that might be more suited to FHMMs.

We have previously developed a speech enhancement scheme which can adapt to unknown additive noise. We model speech and noise using perceptual frequency or `warped' autoregressive HMMs (AR-HMMs) and estimate the clean speech and noise parameters within this framework. In this current work, we investigate the use of our system as a front end to a MFCC recognition system trained on clean speech. To use our system as a front end, we make two modifications. First, we use minimum mean squared error (MMSE) spectral rather than time domain estimators for enhancement. Second, for computational reasons, we form estimators from non-warped AR-HMMs. To avoid mismatch introduced when converting between warped and nonwarped models, we use parallel sets of models. Results are presented for small and medium vocabulary tasks. On the simple task, we are able to approach the performance of a matched system when language model information is included. On the second task, we are not able to incorporate a l...

This paper describes a new algorithm to enhance and recognise noisy speech when only the noisy signal is available. The system uses autoregressive hidden Markov models (HMMs) to model the clean speech and noise and combines these to form a model for the noisy speech. The probability framework developed is then used to reestimate the noise models from the corrupted speech waveform and the process is repeated. Enhancement is performed using the Wiener filters formed from the final clean speech models and noise estimates. Results are presented for additive stationary Gaussian and coloured noise.

A new method to improve the accuracy of Autoregressive Hidden Markov Model (AR-HMM) based recognition systems is proposed. The technique uses the bilinear transform to warp the frequency scale of the observation vectors, hence it uses a better perceptual measure to compare the observation vectors to the trained models. Results presented for the E-set letters from the ISOLET database and the first speaker dependent task of the Resource Management (RM) database show that this technique improves recognition accuracy considerably. However, in the case of the RM system, the recognition results still fall short of those obtained from a similar melfrequency cepstral (MFCC) based system without delta parameters. Reasons for the inferior performance of the AR-HMM system are proposed and future research directions are suggested. The models built for the RM task are incorporated into an existing enhancement algorithm to form a large vocabulary speaker dependent enhancement system. Preliminary res...

This paper presents a real-time speech recognition system used to transcribe broadcast radio speech. The system is based on ABBOT, the hybrid connectionist-HMM large vocabulary continuous speech recognition system developed at the Cambridge University Engineering Department. Developments designed to make the system more robust to acoustic variability and to improve performance when decoding spontaneous speech are described. Modifications necessary to increase the speed of the system so that it operates in real-time are also described. Recognition results and latency figures are presented for speech collected from broadcast news segments on BBC Radio 4.