This paper is concerned with the generation, acquisition, standardized representation and transport of video metadata. The use of MPEG standards in the design and development of interoperable media architectures and web services is discussed. A high-level discussion of several algorithms for metadata extraction is presented. Some architectural and algorithmic issues encountered when designing services for real-time processing of video streams, as opposed to traditional offline media processing, are addressed. A prototype real-time video analysis system for generating MPEG-7 Audiovisual Description Profile from MPEG-2 transport stream encapsulated video is presented. Such a capability can enable a range of new services such as content-based personalization of live broadcasts given that the MPEG-7 based data models fit in well with specifications for advanced television services such as TV-Anytime and Alliance for Telecommunications Industry Solutions IPTV Interoperability Forum.

Combining content analysis of television programs with quantitative audience measurement can provide insights into customer reactions to advertisements and program content. This work introduces a system architecture that incorporates anonymous audience metrics from an operational IPTV environment with metadata from a content-based analysis of recorded programs. Evaluated on a collection of news programs, the system verifies that events derived from the audience metrics data stream correspond to media segmentation boundaries such as commercial breaks and topic changes. An automated system for executing multimodal media segmentation algorithms for commercial break and topic change detection is also discussed. Better understanding of audience reaction can help IPTV service providers plan infrastructure investments and help in managing multimedia content delivery networks.

In this paper, we describe a novel architecture for content and delivery validation for IPTV systems. The system generates multimedia events derived from IPTV content analysis and its delivery. Events are aggregated, filtered and, through a rules-based system, processed and prioritized to provide monitoring and alerting capabilities. Offline event processing along with a video archiving capability allows for detailed forensic analysis. The novelty of the approach includes the correlation and the combination of content-specific and network-specific events in a normalized representation to allow for complex forensic monitoring and alerting. Applications include content security assurance and metadata verification.

Internet protocol TV (IPTV) is a system to deliver TV content to customers by using IP. It has been deployed worldwide by many IPTV service provides, including AT&T, BT, etc. In addition to the core video services provided by the traditional cable TV and satellite TV systems, IPTV provides much richer user interactions and enables additional value added services. This paper presents a prototype for such kind of service: uninterrupted recording and real time media content indexing service for IPTV platforms. While electronic program guides (EPG) list the TV programs that will be broadcast in the future and allow the users to schedule recording for shows that they are interested in, our prototype empowers the users to search and browse any moment of all recent broadcast content for certain channels. Based on the business interest and requirements, this service can be realized in either a centralized architecture (on the network side) or a distributed one (on the individual client side). The developed user interface can be seamlessly integrated on IPTV platforms, and it has been successfully demonstrated on the set top box (STB).

Abstract Building natural language spoken dialogue systems requires large amounts of human transcribed and labeled speech utterances to reach useful operational service performances. Furthermore, the design of such complex systems consists of several manual steps. The User Experience (UE) expert analyzes and defines by hand the system core functionalities: the system semantic scope (call-types) and the dialogue manager strategy that will drive the human–machine interaction. This approach is extensive and error-prone since it involves several nontrivial design decisions that can be evaluated only after the actual system deployment. Moreover, scalability is compromised by time, costs, and the high level of UE know-how needed to reach a consistent design. We propose a novel approach for bootstrapping spoken dialogue systems based on the reuse of existing transcribed and labeled data, common reusable dialogue templates, generic language and understanding models, and a consistent design process. We demonstrate that our approach reduces design and development time while providing an effective system without any application-specific data.

IPTV customers will have access to thousands of video content sources and will require powerful yet intuitive tools to locate desired content. We propose a solution based on stored user interest profiles and multimodal

The proposed shot boundary determination (SBD) algorithm contains a set of finite state machine (FSM) based detectors for pure cut, fast dissolve, fade in, fade out, dissolve, and wipe. Support vector machines (SVM) are applied to the cut and dissolve detectors to further boost performance. Our SBD system was highly effective when evaluated in TRECVID 2006 (TREC video retrieval evaluation) and its performance was ranked highest overall.

This paper presents some of the searching and browsing features of the MIRACLE Video Search Engine. The rapid increase in the generation and dissemination of information and entertainment in video form has created a need for video search engines that facilitate finding and browsing relevant information. MIRACLE is an ongoing project at AT&T Labs aimed at addressing this need. This video search engine combines existing metadata with content-based information that is automatically extracted from the media components, or is obtained from other sources, to achieve this goal. A Web-based user interface provides browsing mechanisms that take advantage of the user’s perceptual abilities to refine the search results. The MIRACLE search engine currently operates on an archive of more than 32,000 hours of video that have been collected and automatically indexed over a ten year period.

This paper describes the content acquisition, indexing and repurposing components of the MIRACLE system. MIRACLE is an ongoing research project at AT&T Labs aimed at creating automated content-based media processing algorithms and systems to collect, organize, index, and repurpose video and multimedia information. While the retrieval engine and the user interface are the most visible part of the query process, performing the required processing to extract the relevant information and generate the indices for the retrieval engine is the most crucial and challenging component of the overall system. Moreover, the utility of the system is highly dependent on the quality and quantity of the acquired data. We describe an acquisition system that takes advantage of Personal Video Recorder (PVR) functionality of Windows OS to acquire high-quality video and the associated metadata. We discuss several processing algorithms for extracting information that enable efficient searching and browsing of more than 32,000 video programs. 1.

Media consumers who are overwhelmed by multitudinous news content demand a system that would sift through tens or hundreds of broadcast TV channels on a daily basis to capture the most important clips that match users' interests and deliver these personalized clips for easy viewing on a typical TV or home PC. In this paper, we present a personal media alert system that extracts video segments for the interested users. In addition, alerts with pointers to content stored on the home server can be sent to mobile users who are authorized to access the content at home. To make content sharing feasible and scalable, we also propose a P2P payment scheme that would allow a consumer to have access to media clips aggregated on other consumers' personal media systems

This paper introduces a methodology for speech data mining along with the tools that the methodology requires. We show how they increase the productivity of the analyst who seeks relationships among the contents of multiple utterances and ultimately must link some newly discovered context into testable hypotheses about new information. While in its simplest form, one can extend text data mining to speech data mining by using text tools on the output of a speech recognizer, we have found that it is not optimal. We show how data mining techniques that are typically applied to text should be modified to enable an analyst to do effective semantic data mining on a large collection of short speech utterances.

We present a system for automatic monitoring and timely dissemination of multimedia information to a range or mobile information appliances based on each user's interest profile. Multimedia processing algorithms detect and isolate relevant video segments from over twenty television broadcast programs based on a collection or words and phrases specified by the user. Content repurposing techniques are then used to convert the information into a form that is suitable for delivery to the user's mobile devices. Alerts are sent using a number of application messaging and network access protocols including email, short message service (SMS), multimedia messaging service (MMS), voice, session initiation protocol (SIP), fax, and pager protocols. The system is evaluated with respect to performance and user experiences. The MediaAlert system provides an effective and low-cost solution for the timely generation of alerts containing personal, business, and security information.

This paper introduces a methodology for speech data mining along with the tools that the methodology requires. We show how they increase the productivity of the analyst who seeks relationships among the contents of multiple utterances and ultimately must link some newly discovered context into testable hypotheses about new information. While, in its simplest form, one can extend text data mining to speech data mining by using text tools on the output of a speech recognizer, we have found that it is not optimal. We show how data mining techniques that are typically applied to text should be modified to enable an analyst to do effective semantic data mining on a large collection of short speech utterances. For the purposes of this paper, we examine semantic data mining in the context of semantic parsing and analysis in a specific situation involving the solution of a business problem that is known to the analyst. We are not attempting a generic semantic analysis of a set of speech. Our tools and methods allow the analyst to mine the speech data to discover the semantics that best cover the desired solution. The coverage, in this case, yields a set of Natural Language Understanding (NLU) classifiers that serve as testable hypotheses.

This paper addresses the problem of generating semantically meaningful content by integrating information from different media. The goal is to automatically construct a compact yet meaningful abstraction of the multimedia data that can serve as an effective index table, allowing users to browse through large amounts of data in a non-linear fashion with flexibility, efficiency, and confidence. We propose an integrated solution in the context of broadcast news that simultaneously utilizes cues from video, audio, and test to achieve the goal. Some experimental results are presented and discussed.

Several years have passed since the research topic of content based multimedia retrieval emerged.We have witnessed the burgeoning research activities into a plenitude of new indexing, retrieval, and filtering tools for images, video, audio, music, graphics, and their combinations with text-based information.Exciting research opportunities arise when integrating knowledge from multiple disciplines, such as media content processing, database, information retrieval, and machine user interface.In the commercial domain, we have also witnessed several impressive efforts moving technologies into practical arenas.This panel includes experts from industry, research labs, and academia.The panel will assess the state of the art and articulate the important future directions in the general field of multimedia access and retrieval.

The challenge of multimedia processing is to provide services that seamlessly integrate text, sound, image, and video information and to do it in a way that preserves the ease of use and interactivity of conventional plain old telephone service (POTS) telephony. To achieve this goal, there are a number of technological problems that must be considered, including: compression and coding of multimedia signals, including algorithmic issues, standards issues, and transmission issues; synthesis and recognition of multimedia signals, including speech, images, handwriting, and text; organization, storage, and retrieval of multimedia signals, including the appropriate method and speed of delivery, resolution, and quality of service; access methods to the multimedia signal, including spoken natural language interfaces, agent interfaces, and media conversion tools; searching by text, speech, and image queries; browsing by accessing the text, by voice, or by indexed images. In each of these areas, a great deal of progress has been made in the past few years, driven in part by the relentless growth in multimedia personal computers and in part by the promise of broad-band access from the home and from wireless connections. Standards have also played a key role in driving new multimedia services, both on the POTS network and on the Internet. It is the purpose of this paper to review the status of the technology in each of the areas listed above and to illustrate current capabilities by describing several multimedia applications that have been implemented at AT&T Labs over the past several years.

The challenge of multimedia processing is to seamlessly integrate text, sound, image, and video information into a single communications channel, and to do it in a way that provides high quality communications while preserving the ease-of-use and interactivity of conventional telephony. There are a number of technology drivers that are pushing the technology forward, as well as a number of technological problems that must be overcome before multimedia becomes as ubiquitous as voiceband telephony. A key issue with any practical multimedia system has to do with standards that insure connectivity between customers and a range of service providers. Multimedia processing is an area of communications that is rapidly evolving. However, a number of interesting and important multimedia communications applications have evolved over the past several years, and some of these applications are described.

This paper describes a working system for the automated archiving and selective retrieval of textual, pictorial and auditory information contained in video programs. Video processing performs the task of representing the visual information using a small subset of the video frames. Linguistic processing refines the closed caption text, generates table of contents, and creates links to relevant multimedia documents. Audio and video information are compressed and indexed based on their temporal association with the selected video frames and processed text. The derived information is used to automatically generate a hypermedia rendition of the program contents. This provides a compact representation of the information contained in the video program. It also serves as a textual and pictorial index for selective retrieval of the full-motion video program. This fully automatic system generates HyperText Markup Language (HTML) renditions of television programs, and makes them available for access over the Internet within seconds of their broadcast. This digital library currently contains over 2200 hours of television programs.

The problem of estimating the properties of smooth, continuous contours from discrete, noisy samples is used as vehicle to demonstrate the robustness of cross-validated regularization applied to a vision problem. A method for estimation of contour properties based on smoothing spline approximations is presented. Generalized cross-validation is to devise an automatic algorithm for finding the optimal value of the smoothing (regularization) parameter from the data. The cross-validated smoothing splines are then used to obtain optimal estimates of the derivatives of quantized contours. Experimental results are presented which demonstrate the robustness of the method applied to the estimation of curvature of quantized contours under variable scale, rotation, and partial occlusion. These results suggest the application of generalized cross-validation to other computer-vision algorithms involving regularization.>

This paper addresses the problem of the nonuniform intersample distance of digital curves obtained as a result of quantization of continuous contours on a square lattice. A resampling algorithm based on variable-factor interpolation and decimation is presented, and its performance is evaluated analytically and by computer simulations assuming the grid-intersect quantization. It is shown that the output of the resampling algorithm outperforms the original digital curve in terms of the average and maximum error in the measurement of length by 50 and 73 percent, respectively. The resampling algorithm can be used as a preprocessing stage in shape analysis systems to enhance their performance by increasing the accuracy and consistency of both local and global features such as curvature and the Fourier shape descriptors.