We introduce a system to automatically manage photocopies made from copyrighted printed materials. The system monitors photocopiers to detect the copying of pages from copyrighted publications. Such activity is tallied for billing purposes. Access rights to the materials can be verified to prevent printing. Digital images of the copied pages are checked against a database of copyrighted pages. To preserve the privacy of the copying of non-copyright materials, only digital fingerprints are submitted to the image matching service. A problem with such systems is creation of the database of copyright pages. To facilitate this, our system maintains statistics of clusters of similar unknown page images along with copy sequence. Once such a cluster has grown to a sufficient size, a human inspector can determine whether those page sequences are copyrighted. The system has been tested with 100,000s of pages from conference proceedings and with millions of randomly generated pages. Retrieval accuracy has been around 99% even with copies of copies or double-page copies.

New technologies arise in a number of ways. They may come from advances in scientific research, through new combinations of existing technologies, or by simply imagining what might be possible in the future. This video describes the evolution of Tabletop Telepresence, a system for remote collaboration through desktop videoconferencing combined with a digital desk. Tabletop Telepresence began as a collection of camera, projector, videoconferencing and user interaction technologies. Working together; artists and research scientists combined these technologies into a means of sharing paper documents between remote desktops, interacting with those documents, requesting services (such as translation), and communicating through a videoconference.

While synchronous meetings are an important part of collaboration, it is not always possible for all stakeholders to meet at the same time. We created the concept of hypermeetings for meetings with asynchronous attendance. Such hypermeetings consist of a chain of video-recorded meetings with hyperlinks for navigating through them. Our HyperMeeting system supports the viewing of prior meetings during a videoconference. Natural viewing behavior such as pausing video generates hyperlinks between previous and current meetings. During playback, automatic link-following guided by playback plans present the relevant content to users. Playback plans take into account the user's meeting attendance and viewing history and match them with features such as topic and speaker segmentation. A user study showed that participants found hyperlinks useful but did not always understand where the links would take them. Experiences from longer-term use and the study results provide a good basis for future system improvements.

Although people are capturing more video with their mobile phones, digital cameras, and other devices, they rarely watch all that video. More commonly, users extract a still image from the video to print or a short clip to share with others. We created a novel interface for browsing through a video keyframe hierarchy to find frames or clips. The interface is shown to be more efficient than scrolling linearly through all keyframes. We developed algorithms for selecting quality keyframes and for clustering keyframes hierarchically. At each level of the hierarchy, a single representative keyframe from each cluster is shown. Users can drill down into the most promising cluster and view representative keyframes for the sub-clusters. Our clustering algorithms optimize for short navigation paths to the desired keyframe. A single keyframe is located using a non-temporal clustering algorithm. A video clip is located using one of two temporal clustering algorithms. We evaluated the clustering algorithms using a simulated search task. User feedback provided us with valuable suggestions for improvements to our system.

Embedded Media Markers (EMMs) are nearly transparent icons printed on paper documents that link to associated digital media. By using the document content for retrieval, EMMs are less visually intrusive than barcodes and other glyphs while still providing an indication for the presence of links. An initial implementation demonstrated good overall performance but exposed difficulties in guaranteeing the creation of unambiguous EMMs. We developed an EMM authoring tool that supports the interactive authoring of EMMs via visualizations that show the user which areas on a page may cause recognition errors and automatic feedback that moves the authored EMM away from those areas. The authoring tool and the techniques it relies on have been applied to corpora with different visual characteristics to explore the generality of our approach.

PACER is a gesture-based interactive paper system that supports fine-grained paper document content manipulation through the touch screen of a cameraphone. Using the phone's camera, PACER links a paper document to its digital version based on visual features. It adopts camera-based phone motion detection for embodied gestures (e.g. marquees, underlines and lassos), with which users can flexibly select and interact with document details (e.g. individual words, symbols and pixels). The touch input is incorporated to facilitate target selection at fine granularity, and to address some limitations of the embodied interaction, such as hand jitter and low input sampling rate. This hybrid interaction is coupled with other techniques such as semi-real time document tracking and loose physical-digital document registration, offering a gesture-based command system. We demonstrate the use of PACER in various scenarios including work-related reading, maps and music score playing. A preliminary user study on the design has produced encouraging user feedback, and suggested future research for better understanding of embodied vs. touch interaction and one vs. two handed interaction.

Browsing and searching for documents in large, online enterprise document repositories are common activities. While internet search produces satisfying results for most user queries, enterprise search has not been as successful because of differences in document types and user requirements. To support users in finding the information they need in their online enterprise repository, we created DocuBrowse, a faceted document browsing and search system. Search results are presented within the user-created document hierarchy, showing only directories and documents matching selected facets and containing text query terms. In addition to file properties such as date and file size, automatically detected document types, or genres, serve as one of the search facets. Highlighting draws the user's attention to the most promising directories and documents while thumbnail images and automatically identified keyphrases help select appropriate documents. DocuBrowse utilizes document similarities, browsing histories, and recommender system techniques to suggest additional promising documents for the current facet and content filters.

Embedded Media Markers, or simply EMMs, are nearly transparent iconic marks printed on paper documents that signify the existence of media associated with that part of the document. EMMs also guide users' camera operations for media retrieval. Users take a picture of an EMM-signified document patch using a cell phone, and the media associated with the EMM-signified document location is displayed on the phone. Unlike bar codes, EMMs are nearly transparent and thus do not interfere with the document appearance. Retrieval of media associated with an EMM is based on image local features of the captured EMM-signified document patch. This paper describes a technique for semi-automatically placing an EMM at a location in a document, in such a way that it encompasses sufficient identification features with minimal disturbance to the original document.

An Embedded Media Marker (EMM) is a transparent mark printed on a paper document that signifies the availability of additional media associated with that part of the document. Users take a picture of the EMM using a camera phone, and the media associated with that part of the document is displayed on the phone. Unlike bar codes, EMMs are nearly transparent and thus do not interfere with the document appearance. Retrieval of media associated with an EMM is based on image features of the document within the EMM boundary. Unlike other feature-based retrieval methods, the EMM clearly indicates to the user the existence and type of media associated with the document location. A semi-automatic authoring tool is used to place an EMM at a location in a document, in such a way that it encompasses sufficient identification features with minimal disturbance to the original document. We will demonstrate how to create an EMM-enhanced document, and how the EMM enables access to the associated media on a cell phone.

This paper presents a tool and a novel Fast Invariant Transform (FIT) algorithm for language independent e-documents access. The tool enables a person to access an e-document through an informal camera capture of a document hardcopy. It can save people from remembering/exploring numerous directories and file names, or even going through many pages/paragraphs in one document. It can also facilitate people's manipulation of a document or people's interactions through documents. Additionally, the algorithm is useful for binding multimedia data to language independent paper documents. Our document recognition algorithm is inspired by the widely known SIFT descriptor [4] but can be computed much more efficiently for both descriptor construction and search. It also uses much less storage space than the SIFT approach. By testing our algorithm with randomly scaled and rotated document pages, we can achieve a 99.73% page recognition rate on the 2188-page ICME06 proceedings and 99.9% page recognition rate on a 504-page Japanese math book [2].

We designed an interactive visual workspace, MediaGLOW, that supports users in organizing personal and shared photo collections. The system interactively places photos with a spring layout algorithm using similarity measures based on visual, temporal, and geographic features. These similarity measures are also used for the retrieval of additional photos. Unlike traditional spring-based algorithms, our approach provides users with several means to adapt the layout to their tasks. Users can group photos in stacks that in turn attract neighborhoods of similar photos. Neighborhoods partition the workspace by severing connections outside the neighborhood. By placing photos into the same stack, users can express a desired organization that the system can use to learn a neighborhood-specific combination of distances.

Retail establishments want to know about traffic flow and patterns of activity in order to better arrange and staff their business. A large number of fixed video cameras are commonly installed at these locations. While they can be used to observe activity in the retail environment, assigning personnel to this is too time consuming to be valuable for retail analysis. We have developed video processing and visualization techniques that generate presentations appropriate for examining traffic flow and changes in activity at different times of the day. Taking the results of video tracking software as input, our system aggregates activity in different regions of the area being analyzed, determines the average speed of moving objects in the region, and segments time based on significant changes in the quantity and/or location of activity. Visualizations present the results as heat maps to show activity and object counts and average velocities overlaid on the map of the space.

This demo introduces a tool for accessing an e-document by capturing one or more images of a real object or document hardcopy. This tool is useful when a file name or location of the file is unknown or unclear. It can save field workers and office workers from remembering/exploring numerous directories and file names. Frequently, it can convert tedious keyboard typing in a search box to a simple camera click. Additionally, when a remote collaborator cannot clearly see an object or a document hardcopy through remote collaboration cameras, this tool can be used to automatically retrieve and send the original e-document to a remote screen or printer.

Hyper-Hitchcock consists of three components for creating and viewing a form of interactive video called detail-on-demand video: a hypervideo editor, a hypervideo player, and algorithms for automatically generating hypervideo summaries. Detail-on-demand video is a form of hypervideo that supports one hyperlink at a time for navigating between video sequences. The Hyper-Hitchcock editor enables authoring of detail-on-demand video without programming and uses video processing to aid in the authoring process. The Hyper-Hitchcock player uses labels and keyframes to support navigation through and back hyperlinks. Hyper-Hitchcock includes techniques for automatically generating hypervideo summaries of one or more videos that take the form of multiple linear summaries of different lengths with links from the shorter to the longer summaries. User studies on authoring and viewing provided insight into the various roles of links in hypervideo and found that player interface design greatly affects people's understanding of hypervideo structure and the video they access.

A common video surveillance task is to keep track of people moving around the space being monitored. It is often difficult to track activity between cameras because locations such as hallways in office buildings can look quite similar and do not indicate the spatial proximity of the cameras. We describe a spatial video player that orients nearby video feeds with the field of view of the main playing video to aid in tracking between cameras. This is compared with the traditional bank of cameras with and without interactive maps for identifying and selecting cameras. We additionally explore the value of static and rotating maps for tracking activity between cameras. The study results show that both the spatial video player and the map improve user performance when compared to the camera-bank interface. Also, subjects change cameras more often with the spatial player than either the camera bank or the map, when available.

DOTS (Dynamic Object Tracking System) is an indoor, real-time, multi-camera surveillance system, deployed in a real office setting. DOTS combines video analysis and user interface components to enable security personnel to effectively monitor views of interest and to perform tasks such as tracking a person. The video analysis component performs feature-level foreground segmentation with reliable results even under complex conditions. It incorporates an efficient greedy-search approach for tracking multiple people through occlusion and combines results from individual cameras into multi-camera trajectories. The user interface draws the users. attention to important events that are indexed for easy reference at a later time. Different views within the user interface provide spatial information for easier navigation. Our system, with over twenty video cameras installed in hallways and other public spaces in our office building, has been in constant use for almost a year.

Organizing digital photograph collections according to events such as holiday gatherings or vacations is a common practice among photographers. To support photographers in this task, we present similarity-based methods to cluster digital photos by time and image content. The approach is general and unsupervised, and makes minimal assumptions regarding the structure or statistics of the photo collection. We present several variants of an automatic unsupervised algorithm to partition a collection of digital photographs based either on temporal similarity alone, or on temporal and content-based similarity. First, interphoto similarity is quantified at multiple temporal scales to identify likely event clusters. Second, the final clusters are determined according to one of three clustering goodness criteria. The clustering criteria trade off computational complexity and performance. We also describe a supervised clustering method based on learning vector quantization. Finally, we review the results of an experimental evaluation of the proposed algorithms and existing approaches on two test collections.

We introduced detail-on-demand video as a simple type of hypervideo that allows users to watch short video segments and to follow hyperlinks to see additional detail. Such video lets users quickly access desired information without having to view the entire contents linearly. A challenge for presenting this type of video is to provide users with the appropriate affordances to understand the hypervideo structure and to navigate it effectively. Another challenge is to give authors tools that allow them to create good detail-on-demand video. Guided by user feedback, we iterated designs for a detail-on-demand video player. We also conducted two user studies to gain insight into people's understanding of hypervideo and to improve the user interface. We found that the interface design was tightly coupled to understanding hypervideo structure and that different designs greatly affected what parts of the video people accessed. The studies also suggested new guidelines for hypervideo authoring.

No abstract available.

We created an alternative approach to existing video summaries that gives viewers control over the summaries by selecting hyper-links to other video with additional information. We structure such summaries as "detail-on-demand" video, a subset of general hypervideo in which at most one link to another video sequence is available at any given time. Our editor for such video, Hyper-Hitchcock, provides a workspace in which an author can select and arrange video clips, generate composites from clips and from other composites, and place links between composites. To simplify dealing with a large number of clips, Hyper-Hitchcock generates iconic representations for composites that can be used to manipulate the composite as a whole. In addition to providing an authoring environment, Hyper-Hitchcock can automatically generate multi-level hypervideo summaries for immediate use or as the starting point for author modification.

The FXPAL Photo Application is designed to faciliate the organization of digital images from digital cameras and other sources through automated organization and intuitive user interfaces.

We present similarity-based methods to cluster digital photos by time and image content. The approach is general, unsupervised, and makes minimal assumptions regarding the structure or statistics of the photo collection. We present results for the algorithm based solely on temporal similarity, and jointly on temporal and content-based similarity. We also describe a supervised algorithm based on learning vector quantization. Finally, we include experimental results for the proposed algorithms and several competing approaches on two test collections.

We demonstrate the use of detail-on-demand hypervideo in interactive training and video summarization. Detail-on-demand video allows viewers to watch short video segments and to follow hyperlinks to see additional detail. The player for detail-on-demand video displays keyframes indicating what links are available at each point in the video. The Hyper-Hitchcock authoring tool helps users create hypervideo by automatically dividing video into clips that can be combined in a direct manipulation interface. Clips can be grouped into composites and hyperlinks can be placed between clips and composites. A summarization algorithm creates multi-level hypervideo summaries from linear video by automatically selecting clips and placing links between them.

To simplify the process of editing interactive video, we developed the concept of "detail-on-demand" video as a subset of general hypervideo. Detail-on-demand video keeps the authoring and viewing interfaces relatively simple while supporting a wide range of interactive video applications. Our editor, Hyper-Hitchcock, provides a direct manipulation environment in which authors can combine video clips and place hyperlinks between them. To summarize a video, Hyper-Hitchcock can also automatically generate a hypervideo composed of multiple video summary levels and navigational links between these summaries and the original video. Viewers may interactively select the amount of detail they see, access more detailed summaries, and navigate to the source video through the summary.

We present a system that allows remote and local participants to control devices in a meeting environment using mouse or pen based gestures "through" video windows. Unlike state-of-the-art device control interfaces that require interaction with text commands, buttons, or other artificial symbols, our approach allows users to interact with devices through live video of the environment. This naturally extends our video supported pan/tilt/zoom (PTZ) camera control system, by allowing gestures in video windows to control not only PTZ cameras, but also other devices visible in video images. For example, an authorized meeting participant can show a presentation on a screen by dragging the file on a personal laptop and dropping it on the video image of the presentation screen. This paper presents the system architecture, implementation tradeoffs, and various meeting control scenarios.

In this paper, we describe how a detail-on-demand representation for interactive video is used in video summarization. Our approach automatically generates a hypervideo composed of multiple video summary levels and navigational links between these summaries and the original video. Viewers may interactively select the amount of detail they see, access more detailed summaries, and navigate to the source video through the summary. We created a representation for interactive video that supports a wide range of interactive video applications and Hyper-Hitchcock, an editor and player for this type of interactive video. Hyper-Hitchcock employs methods to determine (1) the number and length of levels in the hypervideo summary, (2) the video clips for each level in the hypervideo, (3) the grouping of clips into composites, and (4) the links between elements in the summary. These decisions are based on an inferred quality of video segments and temporal relations those segments.

We present a method for rapidly and robustly extracting audio excerpts without the overhead of speech recognition or speaker segmentation. An immediate application is to automatically augment keyframe-based video summaries with informative audio excerpts associated with the video segments represented by the keyframes. Short audio clips combined with keyframes comprise an extremely lightweight and Web-browsable interface for auditioning video or similar media, without using bandwidth-intensive streaming video or audio.

This paper presents a camera system called FlySPEC. In contrast to a traditional camera system that provides the same video stream to every user, FlySPEC can simultaneously serve different video-viewing requests. This flexibility allows users to conveniently participate in a seminar or meeting at their own pace. Meanwhile, the FlySPEC system provides a seamless blend of manual control and automation. With this control mix, users can easily make tradeoffs between video capture effort and video quality. The FlySPEC camera is constructed by installing a set of pan/tilt/zoom (PTZ) cameras near a high-resolution panoramic camera. While the panoramic camera provides the basic functionality of serving different viewing requests, the PTZ camera is managed by our algorithm to improve the overall video quality that may affect users watching details. The video resolution improvements from using different camera management strategies are compared in the experimental section.

Hitchcock is a semi-automatic video editing system. This video shows users collaboratively authoring a home video.

FlySPEC is a video camera system designed for real-time remote operation. A hybrid design combines the high resolution of an optomechanical video camera with the wide field of view always available from a panoramic camera. The control system integrates requests from multiple users so that each controls a virtual camera. The control system seamlessly integrates manual and fully automatic control. It supports a range of options from untended automatic to full manual control. The system can also learn control strategies from user requests. Additionally, the panoramic view is always available for an intuitive interface, and objects are never out of view regardless of the zoom factor. We present the system architecture, an information-theoretic approach to combining panoramic and zoomed images to optimally satisfy user requests, and experimental results that show the FlySPEC system significantly assists users in a remote inspection tasks.

Hitchcock is a system that allows users to easily create custom videos from raw video shot with a standard video camera.In contrast to other video editing systems, Hitchcock uses automatic analysis to determine the suitability of portions of the raw video.Unsuitable video typically has fast or erratic camera motion.Hitchcock first analyzes video to identify the type and amount of camera motion: fast pan, slow zoom, etc.Based on this analysis, a numerical "unsuitability" score is computed for each frame of the video.Combined with standard editing rules, this score is used to identify clips for inclusion in the final video and to select their start and end points.To create a custom video, the user drags keyframes corresponding to the desired clips into a storyboard.Users can lengthen or shorten the clip without specifying the start and end frames explicitly.Clip lengths are balanced automatically using a spring-based algorithm.

This paper describes a method for finding segments in video-recorded meetings that correspond to presentations. These segments serve as indexes into the recorded meeting. The system automatically detects intervals of video that correspond to presentation slides. We assume that only one person speaks during an interval when slides are detected. Thus these intervals can be used as training data for a speaker spotting system. An HMM is automatically constructed and trained on the audio data from each slide interval. A Viterbi alignment then resegments the audio according to speaker. Since the same speaker may talk across multiple slide intervals the acoustic data from these intervals is clustered to yield an estimate of the number of distinct speakers and their order. This allows the individual presentations in the video to be identified from the location of each presenter's speech. Results are presented for a corpus of six meeting videos.

This paper describes a technique for automatically creating an index for handwritten notes captured as digital ink. No text recognition is performed. Rather, a dictionary of possible index terms is built by clustering groups of ink strokes corresponding roughly to words. Terms whose distribution varies significantly across note pages are selected for the index. An index page containing the index terms is created, and terms are hyper-linked back to their original location in the notes. Further, index terms occurring in a note page are highlighted to aid browsing.

NoteLook is a client-server system designed and built to support multimedia note taking in meetings with digital video and ink. It is integrated into a conference room equipped with computer controllable video cameras, video conference camera, and a large display rear video projector. The NoteLook client application runs on wireless pen-based notebook computers. Video channels containing images of the room activity and presentation material are transmitted by the NoteLook servers to the clients, and the images can be interactively and automatically incorporated into the note pages. Users can select channels, snap in large background images and sequences of thumbnails, and write freeform ink notes. A smart video source management component enables the capture of high quality images of the presentation material from a variety of sources. For accessing and browsing the notes and recorded video, NoteLook generates Web pages with links from the images and ink strokes correlated to the video.

No abstract available.

This paper describes a technique for segmenting video using hidden Markov models (HMM). Video is segmented into regions defined by shots, shot boundaries, and camera movement within shots. Features for segmentation include an image-based distance between adjacent video frames, an audio distance based on the acoustic difference in intervals just before and after the frames, and an estimate of motion between the two frames. Typical video segmentation algorithms classify shot boundaries by computing an image-based distance between adjacent frames and comparing this distance to fixed, manually determined thresholds. Motion and audio information is used separately. In contrast, our segmentation technique allows features to be combined within the HMM framework. Further, thresholds are not required since automatically trained HMMs take their place. This algorithm has been tested on a video data base, and has been shown to improve the accuracy of video segmentation over standard threshold-based systems.

Many techniques can extract information from an multimedia stream, such as speaker identity or shot boundaries. We present a browser that uses this information to navigate through stored media. Because automatically-derived information is not wholly reliable, it is transformed into a time-dependent “confidence score. ” When presented graphically, confidence scores enable users to make informed decisions about regions of interest in the media, so that non-interesting areas may be skipped. Additionally, index points may be determined automatically for easy navigation, selection, editing, and annotation and will support analysis types other than the speaker identification and shot detection used here. 1.1 Keywords Content-based retrieval, video, speaker identification, automatic analysis, visualization, skimming 2.

In this paper, we describe a technique for dynamically grouping digital ink and audio to support user interaction in freeform note-taking systems.For ink, groups of strokes might correspond to words, lines, or paragraphs of handwritten text.For audio, groups might be a complete spoken phrase or a speaker turn in a conversation.Ink and audio grouping is important for editing operations such as deleting or moving chunks of ink and audio notes.The grouping technique is based on hierarchical agglomerative clustering.This clustering algorithm yields groups of ink or audio in a range of sizes, depending on the level in the hierarchy, and thus provides structure for simple interactive selection and rapid non-linear expansion of a selection.Ink and audio grouping is also important for marking portions of notes for subsequent browsing and retrieval.Integration of the ink and audio clusters provides a flexible way to browse the notes by selecting the ink cluster and playing the corresponding audio cluster.

Dynomitc is a portable electronic notebook for the capture and retrieval of handwritten and audio notes.The goal of Dynomitc is to merge the organization, search, and data acquisition capabilities of a computer with the benefits of a paper-based notebook.Dynomite provides novel solutions in tour key problem areas.First, Dynomite uses a casual, low cognitive overhead interface.Second, for content indexing of notes, Dynomite uses ink properties and keywords.Third, to assist organization, Dynomite's properties and keywords define views, presenting a subset of the notebook content that dynamically changes as users add new information.Finally, to augment handwritten notes with audio on devices with limited storage, Dynomite continuously records audio, but only permanently stores those parts highlighted by the user.

In spotting for phrases in text images, speed and accuracy are important considerations. In a hidden Markov model (HMM) based spotter recognition time is dominated by the time required to compute the state conditional observation probabilities. These probabilities are a measure of how well the data match each state in the model. In this paper discrete and continuous hidden Markov models are compared based on speed and accuracy in spotting for phrases in text images. For the discrete HMM, vector quantization is used to associate each continuous feature vector with a discrete value. For the continuous HMMs, the observation distributions for the feature vectors are modeled by either a single Gaussian, or a mixture of two Gaussians. Comparisons were made on a subset of the UW English Document Image Database I. The best accuracy was observed when a mixture of two Gaussians was used in the continuous HMM. The discrete HMM provides for faster spotting particularly when long phrases are used.

This paper describes techniques for segmentation of conversational speech based on speaker identity. Speaker segmentation is performed using Viterbi decoding on a hidden Markov model network consisting of interconnected speaker sub-networks. Speaker sub-networks are initialized using Baum-Welch training on data labeled by speaker, and are iteratively retrained based on the previous segmentation. If data labeled by speaker is not available, agglomerative clustering is used to approximately segment the conversational speech according to speaker prior to Baum-Welch training. The distance measure for the clustering is a likelihood ratio in which speakers are modeled by Gaussian distributions. The distance between merged segments is recomputed at each stage of the clustering, and a duration model is used to bias the likelihood ratio. Segmentation accuracy using agglomerative clustering initialization matches accuracy using initialization with speaker labeled data.>

A hidden Markov model (HMM) based system for detecting locating, or spotting, user-specified keywords in scanned images is described. The system is font-independent, and no pre-segmentation of text and graphics is required. The bounding boxes of potential lines of text are extracted from the image using morphology. Feature vectors based on the external shape and internal structure of characters are computed for each bounding box. A keyword HMM is created by concatenating appropriate context-dependent character HMMs. The non-keyword HMM is based on context-dependent sub-character models. Keywords are spotted using Viterbi decoding on an HMM network created from the keyword and non-keyword HMMs. This model allows detection of keywords embedded in a line without pre-segmentation of the line into words or characters. Thus keywords may be specified by a baseform and variants of the keyword can be detected.>

No abstract available.

Error-correcting training is extended to spotting algorithms. An error measure for spotting tasks is defined, and the derivatives of that error with respect to the parameters of the spotter are derived. The derivatives are computed by modifying the backward pass of the forward-backward algorithm. The training algorithm is applicable to any hidden Markov model (HMM)-based classification task in which a frame-by-frame classifier error can be defined. Here it is applied to phoneme spotting. Preliminary experimental results have been obtained for spotting the phonemes mod r mod , mod s mod , mod t mod in the TIMIT database. Error-correcting training generally improved the spotter accuracy, reducing the miss rate by up to 30%.>

Algorithms for a speaker-dependent wordspotting system based on hidden Markov models (HMMs) are described. The system allows a user to specify keywords dynamically and to train the associated HMMs via a single repetition of a keyword. Nonkeyword speech is modeled using an HMM trained from a prerecorded sample of continuous speech. The wordspotter is intended for interactive applications, such as the editing of voice mail or mixed-media documents, and for keyword indexing in audio or video recordings. The forward-backward search algorithm used in the wordspotter is compared with the Viterbi decoder on the basis of speed and accuracy. In addition, an algorithm for speaker adaptation is described which allows indexing by a user into another talker's speech.>

The theory of Markov random fields (MRFs) is used to extract formants from linear predictive coding (LPC) pole locations. In this model, the spectral peaks are presented in a time-frequency plot, using LPC poles as noisy indications of these peaks, and the formants are represented as lines connecting the spectral peaks. The spectral peaks and formants are modeled by a site process MRF and a line process MRF, respectively. The a posteriori probability of the spectral peaks and formants, given the LPC poles, quantifies the continuity constraints on the formants, the relationship between spectral peaks and formant locations, and the noise of the LPC poles. The formants are chosen by maximizing the a posteriori probability using simulated annealing. The parameters characterizing the MRFs are estimated by maximizing a pseudolikelihood function derived from LPC pole data with correctly labeled formants.>

In this paper, we describe a robust technique for segmenting an utterance into a sequence of coarse phonetic classes. The resulting coarse class string is used to provide contextual information for further phonetic analysis, and in lexical access to limit the number of word candidates. Each 10 ms interval of the utterance is first given a probability of belonging to each of five classes: silence, vowel, nasal-like, strong fricative and weak fricative. The probabilities are assigned using a hierarchical classification scheme with Gaussian classifiers at each node. A fuzzy C-means clustering procedure is used to learn the class means and variances from unlabeled data. Dynamic programming is used to align the utterance with all possible coarse class strings in the lexicon. The performance of the classifier has been evaluated on the TI speaker independent isolated digits data. The correct word is hypothesized by a coarse class sequence more than 99 percent of the time.

Performance degradation of isolated word recognition systems on vocabularies with acoustically similar words is a well known problem. In many cases, vocabularies can be designed to eliminate this confusability. However, the English alphabet combined with digits is a fixed vocabulary which cannot be easily modified to be less confusable. We show how a priori phonetic knowledge of the alpha-digits vocabulary can be used during template generation to reduce error rate. Unlike methods proposed by Lamel and Zue or Rabiner and Wilpon, our technique requires only a single pass for recognition. Thus it is usable in a real time system. The alpha-digits vocabulary is partitioned into subclasses of words with common vowel phonemes, or in some cases allophones. Each subclass consists of words which differ only in the starting or final consonants. Templates are created so the common vowel in each subclass has one representation. For words in the same subclass, the match from the dynamic programming score reflects only initial or final consonant differences. Template storage space is reduced because the vowel portion of each template in the subclass is stored only once.

Dynamic programming (DP) has become a popular means of performing the necessary time warping in isolated word recognition. DP offers the advantage of always finding the optimal fit of a known reference pattern against an unknown input utterance. Unfortunately, DP has two major disadvantages: 1) The amount of data needed to store the known reference patterns can be very large and 2) the computation needed to do the full DP search is intensive. Because of these problems, DP has not yet become feasible for commercial low cost speaker dependent isolated word recognition systems. We present two means of reducing both the storage requirements and the computation needed for the DP search: 1) a segmentation scheme for the reduction of data (and thereby computation) which typically reduces the speech data by a factor of between 3 and 4 and 2) a modification of the Beam search used by the Harpy speech recognition system which provides a substantial speedup over conventional DP. Together, computation is reduced by an order of magnitude.