We motivate the need for and describe the key components of real world audio indexing systems. In particular, we discuss the various flavors of such systems, the advantages and disadvantages of each, user interfaces, system architectures and evaluation issues. Throughout the paper, we give examples from our own experience of audio indexing using SpeechBot and its successor NewsTuner.

We present several novel approaches to the Out of Vocabulary (OOV) query problem for spoken audio: indexing based on syllable-like units called particles and query expansion according to acoustic confusability for a word index. We also examine linear and OOV-based combination of indexing schemes. We experiment on 75 h of broadcast news, comparing our techniques to a word index, a phoneme index and a phoneme index queried with phoneme sequences. Our results show that our approaches are superior to both a word index and a phoneme index for OOV words, and have comparable performance to the sequence of phonemes scheme. The particle system has worse performance than the acoustic query expansion scheme. The best system uses word queries for in-vocabulary words and a linear combination of the phoneme sequence scheme and acoustic query expansion for OOV words. Using the best possible weights for linear combination, this system improves the average precision from 0.35 for a word index to 0.40, a result only obtainable if the weights could be learnt on a development query set. The next best system used a word index for in-vocabulary words and the phoneme sequence system otherwise and had average precision of 0.39.

We present A new Web-based application, called the News Tuner, for searching and browsing large radio archives. While popular search engines provide means for finding text and images, our approach combines semantic and acoustic search for efficient retrieval of audio documents. Semantic search allows the user to retrieve stories for a given concept, while acoustic search allows random access within stored audio files. Our experiments on over 1700 programs show that our method is effective at quickly retrieving stories that would be difficult to find otherwise. The News Tuner paradigm is intended primarily for news and talk radio programs, however it may be applied to browsing and searching any spoken word audio content.

We present a novel approach to the out of vocabulary (OOV) query problem for audio indexing. Our technique first builds a word index for the audio using speech recognition. It then expands query words into in-vocabulary phrases according to intrinsic acoustic confusability and language model scores. The aim is to mimic the mistakes the speech recognizer makes when transcribing the OOV words.

As the Web transforms from a text-only medium into a more multimedia-rich medium, the need arises to perform searches based on the multimedia content. In this paper, we present an audio and video search engine to tackle this problem. The engine uses speech recognition technology to index spoken audio and video files from the World Wide Web (WWW) when no transcriptions are available. If transcriptions (even imperfect ones) are available, we can also take advantage of them to improve the indexing process. Our engine indexes several thousand talk and news radio shows covering a wide range of topics and speaking styles from a selection of public Web sites with multimedia archives. Our Web site is similar in spirit to normal Web search sites; it contains an index, not the actual multimedia content. The audio from these shows suffers in acoustic quality due to bandwidth limitations, coding, compression, and poor acoustic conditions. Our word error rate (WER) results using appropriately trained acoustic models show remarkable resilience to the high compression, although many factors combine to increase the average WERs over standard broadcast news benchmarks. We show that, even if the transcription is inaccurate, we can still achieve good retrieval performance for typical user queries (77.5%).

We have developed a speech recognition based audio search engine for indexing spoken documents found on the World Wide Web. Our site (http://www.compaq.com/speechbot) indexes around 20 news and talk radio shows covering a wide range of topics, speaking styles and acoustic conditions from a selection of public Web sites with multimedia archives. In this paper, we describe our system and its performance, focusing on the speech recognition and retrieval aspects. We describe our training procedure in some detail and report our historical error rate since the site launch. We also investigate the impact of Out Of Vocabulary (OOV) words. Finally we report the results of retrieval experiments which demonstrate that our system can index effectively.

In this paper we address the problem of aligning very long (of-ten more than one hour) audio files to their corresponding textual transcripts in an effective manner. We present an efficient recur-sive technique to solve this problem that works well even on noisy speech signals. The key idea of this algorithm is to turn the forced alignment problem into a recursive speech recognition problem with a gradually restricting dictionary and language model. The algorithm is tolerant to acoustic noise and errors or gaps in the text transcript or audio tracks. We report experimental results on a 3 hour audio file containing TV and radio broadcasts. We will show accurate alignments on speech under a variety of real acoustic conditions such as speech over music and speech over telephone lines. We also report re-sults when the same audio stream has been corrupted with white additive noise or compressed using a popular web encoding for-mat such as RealAudio. This algorithm has been used in our internal multimedia indexing project. It has processed more than 200 hours of audio from var-ied sources, such as WGBH NOVA documentaries and NPR web audio files. The system aligns speech media content in about one to five times realtime, depending on the acoustic conditions of the audio signal. 1.

The goal of this work is to use phonetic recognition to drive a synthetic image with speech. Phonetic units are identified by the phonetic recognition engine and mapped to mouth gestures, known as visemes, the visual counterpart of phonemes. The acoustic waveform and visemes are then sent to a synthetic image player, called FaceMe! where they are rendered synchronously. This paper provides background for the core technologies involved in this process and describes asynchronous and synchronous prototypes of a combined phonetic recognition/FaceMe! system which we use to render mouth gestures on an animated face. 1. Introduction This paper addresses the problem of driving an animated face using audio data. We present a phonetic recognition system as a front-end process to generate visemes, the visual analog of phonemes. The paper provides background for the core phonetic recognition technology, based on Statistical Trajectory Modeling [1] and the core FaceMe! animation technology. It th...

The capability to view detailed events taking place inside of a speech search engine can be essential for reducing the computational complexity and enhancing the accuracy of a speech recognition system.This paper describes a generic search visualization and performance tuning tool, Vista, which permits a user to interactively examine data within the search module of a speech recognition system.Vista's key attributes include the ability to display a large number of active search paths simultaneously, to display data relevant to these paths, and to update the paths and data as the search progresses in time.

A planar map is a figure formed by a set of intersecting lines and curves. Such an object captures both the geometrical and the topological information implicitly defined by the data. In the context of 2D drawing it provides a new interaction paradigm, map sketching, for editing graphic shapes.To build a planar map, one must compute curve intersections and deduce from them the map they define. The computed topology must be consistent with the underlying geometry. Robustness of geometric computations is a key issue in this process. We present a robust solution to Bézier curve intersection that uses exact forward differencing and bounded rational arithmetic. Then, we describe data structure and algorithms to support incremental insertion of Bézier curves in a planar map. A prototype illustration tool using this method is also discussed.