Streaming media is gaining popularity, with numerous new services for video on demand and live broadcast. These services stream requested media content to user devices like smart TVs, personal computers and smartphones. Many of these devices are mobile devices, yet streaming media services do not adapt the delivered content to the context of the request, e.g., the location of the user, the time of the request or who are the people near the viewer. In this paper we illustrate how geofencing and geoblocking can be combined with adaptive streaming media to create a new technology of contextually-adapted streaming media. The new technology has the potential to create a new type of experience for streaming media by applying real-time modification of streamed content according to the viewer and the context, e.g., modify inappropriate content when the media is played in public places, change the length of a video played by a train passenger according to a train schedule, etc. The suggested technology can also be used for preventing password sharing in a non-intrusive way and for hyperlocal geoblocking, to facilitate copyright protection. We discuss the vision of modifying streaming media in real time based on the context, elaborate on some of the challenges in implementing this vision, and present novel applications of this new technology.

This paper is concerned with the generation, acquisition, standardized representation and transport of video metadata. The use of MPEG standards in the design and development of interoperable media architectures and web services is discussed. A high-level discussion of several algorithms for metadata extraction is presented. Some architectural and algorithmic issues encountered when designing services for real-time processing of video streams, as opposed to traditional offline media processing, are addressed. A prototype real-time video analysis system for generating MPEG-7 Audiovisual Description Profile from MPEG-2 transport stream encapsulated video is presented. Such a capability can enable a range of new services such as content-based personalization of live broadcasts given that the MPEG-7 based data models fit in well with specifications for advanced television services such as TV-Anytime and Alliance for Telecommunications Industry Solutions IPTV Interoperability Forum.

Combining content analysis of television programs with quantitative audience measurement can provide insights into customer reactions to advertisements and program content. This work introduces a system architecture that incorporates anonymous audience metrics from an operational IPTV environment with metadata from a content-based analysis of recorded programs. Evaluated on a collection of news programs, the system verifies that events derived from the audience metrics data stream correspond to media segmentation boundaries such as commercial breaks and topic changes. An automated system for executing multimodal media segmentation algorithms for commercial break and topic change detection is also discussed. Better understanding of audience reaction can help IPTV service providers plan infrastructure investments and help in managing multimedia content delivery networks.

Content indexing has become necessary, not just optional, in the era where broadcast, cable and Internet produce huge amounts of media daily. Text information from spoken audio is still a key feature to understand content along with other meta-data and video features. In this paper, a new method is introduced to improve transcription quality, which allows more accurate content indexing. Our method finds phonetic similarities between two imperfect sources, closed captions and ASR outputs, and aligns them together to make quality transcriptions. In the process, even out-of-vocabulary words could be learned automatically. Given broadcast news audio and closed captions, our experimental results show that the proposed method, on average, improves word correct rates 11 % from the ASR output using the baseline language model and 6 % from the one using the adapted language model.

In this paper, we describe a novel architecture for content and delivery validation for IPTV systems. The system generates multimedia events derived from IPTV content analysis and its delivery. Events are aggregated, filtered and, through a rules-based system, processed and prioritized to provide monitoring and alerting capabilities. Offline event processing along with a video archiving capability allows for detailed forensic analysis. The novelty of the approach includes the correlation and the combination of content-specific and network-specific events in a normalized representation to allow for complex forensic monitoring and alerting. Applications include content security assurance and metadata verification.

GeoTV is a project that explores seamless integration of mobile phones, HDTV sets, and computers in the living room to enrich the user experience of existing services. The three- screen service allows a user to navigate a world map on a smart phone to track geo-located media RSS content that matches her personal interests. The user can show a matching video clip on her phone or direct a nearby HDTV set to play the video. In addition, the user can bring up a world map on a nearby computer screen to navigate areas of interest related to the video clip. GeoTV allows all three screens to be used for what they are best for: HDTV for high resolution video, computer screen for browsing a world map, and a smart phone for personalized control at hand to select media of interest.

Internet protocol TV (IPTV) is a system to deliver TV content to customers by using IP. It has been deployed worldwide by many IPTV service provides, including AT&T, BT, etc. In addition to the core video services provided by the traditional cable TV and satellite TV systems, IPTV provides much richer user interactions and enables additional value added services. This paper presents a prototype for such kind of service: uninterrupted recording and real time media content indexing service for IPTV platforms. While electronic program guides (EPG) list the TV programs that will be broadcast in the future and allow the users to schedule recording for shows that they are interested in, our prototype empowers the users to search and browse any moment of all recent broadcast content for certain channels. Based on the business interest and requirements, this service can be realized in either a centralized architecture (on the network side) or a distributed one (on the individual client side). The developed user interface can be seamlessly integrated on IPTV platforms, and it has been successfully demonstrated on the set top box (STB).

Abstract Building natural language spoken dialogue systems requires large amounts of human transcribed and labeled speech utterances to reach useful operational service performances. Furthermore, the design of such complex systems consists of several manual steps. The User Experience (UE) expert analyzes and defines by hand the system core functionalities: the system semantic scope (call-types) and the dialogue manager strategy that will drive the human–machine interaction. This approach is extensive and error-prone since it involves several nontrivial design decisions that can be evaluated only after the actual system deployment. Moreover, scalability is compromised by time, costs, and the high level of UE know-how needed to reach a consistent design. We propose a novel approach for bootstrapping spoken dialogue systems based on the reuse of existing transcribed and labeled data, common reusable dialogue templates, generic language and understanding models, and a consistent design process. We demonstrate that our approach reduces design and development time while providing an effective system without any application-specific data.

IPTV customers will have access to thousands of video content sources and will require powerful yet intuitive tools to locate desired content. We propose a solution based on stored user interest profiles and multimodal

This paper investigates a novel concept of providing seamless control and portability of an IPTV viewing session. A solution employing a middleware system, a secure hardware token and a cell phone are used to demonstrate how an IPTV session can be securely controlled remotely and moved between multiple viewing stations. We have built a prototype of the system and demonstrated its flexible features. Depending on the user's protocol of choice, most remote control operations from a mobile device took less than 5 seconds to execute. An interesting capability of previewing content of other channels via the user's device while still continuing to watch the program on the viewing station differentiates it from today's IPTV offers. Finally for mobile content delivery, we address the problem of dynamic device profile selection and content adaptation using a classification algorithm to match the best content alternative destined for a mobile.

The proposed shot boundary determination (SBD) algorithm contains a set of finite state machine (FSM) based detectors for pure cut, fast dissolve, fade in, fade out, dissolve, and wipe. Support vector machines (SVM) are applied to the cut and dissolve detectors to further boost performance. Our SBD system was highly effective when evaluated in TRECVID 2006 (TREC video retrieval evaluation) and its performance was ranked highest overall.

The Web is rapidly moving towards a platform for mass collaboration in content production and consumption from three screens: computers, mobile phones, and TVs. While there has been a surge of interests in making Web content accessible from mobile devices, there is a significant lack of progress when it comes to making the web experience suitable for viewing on a television. Towards this end, we describe a novel concept, namely GeoTV, where we explore a framework by which web content can be presented or pushed in a meaningful manner to create an entertainment experience for the TV audience. Fresh content on a variety of topics, people, and places is being created and made available on the Web at breathtaking speed. Navigating fresh content effectively on TV demands a new browsing paradigm that requires fewer mouse clicks or user interactions from the remote control. Novel geospatial and temporal browsing techniques are provided in GeoTV that allow users the capability of aggregating and navigating RSS-enabled content in a timely, personalized and automatic manner for viewing in an IPTV environment. This poster is an extension of our previous work on GeoTracker that utilizes both a geospatial representation and a temporal (chronological) presentation to help users spot the most relevant updates quickly within the context of a Web-enabled environment. We demonstrate 1) the usability of such a tool that greatly enhances a user.s ability in locating and browsing videos based on his or her geographical interests and 2) various innovative interface designs for showing RSS-enabled information in an IPTV environment.

The Web is rapidly moving towards a platform for mass collaboration in content production and consumption. Fresh content on a variety of topics, people, and places is being created and made available on the Web at breathtaking speed. Navigating the content effectively not only requires techniques such as aggregating various RSS-enabled feeds, but it also demands a new browsing paradigm. In this paper, we present novel geospatial and temporal browsing techniques that provide users with the capability of aggregating and navigating RSS-enabled content in a timely, personalized and automatic manner. In particular, we describe a system called GeoTracker that utilizes both a geospatial representation and a temporal (chronological) presentation to help users spot the most relevant updates quickly. Within the context of this work, we provide a middleware engine that supports intelligent aggregation and dissemination of RSS feeds with personalization to desktops and mobile devices. We study the navigation capabilities of this system on two kinds of data sets, namely, 2006 World Cup soccer data collected over two months and breaking news items that occur every day. We also demonstrate that the application of such technologies to the video search results returned by YouTube and Google greatly enhances a user.s ability in locating and browsing videos based on his or her geographical interests. Finally, we demonstrate that the location inference performance of GeoTracker compares well against machine learning techniques used in the natural language processing/information retrieval community. Despite its algorithm simplicity, it preserves high recall percentages.

This paper presents some of the searching and browsing features of the MIRACLE Video Search Engine. The rapid increase in the generation and dissemination of information and entertainment in video form has created a need for video search engines that facilitate finding and browsing relevant information. MIRACLE is an ongoing project at AT&T Labs aimed at addressing this need. This video search engine combines existing metadata with content-based information that is automatically extracted from the media components, or is obtained from other sources, to achieve this goal. A Web-based user interface provides browsing mechanisms that take advantage of the user’s perceptual abilities to refine the search results. The MIRACLE search engine currently operates on an archive of more than 32,000 hours of video that have been collected and automatically indexed over a ten year period.

This paper describes the content acquisition, indexing and repurposing components of the MIRACLE system. MIRACLE is an ongoing research project at AT&T Labs aimed at creating automated content-based media processing algorithms and systems to collect, organize, index, and repurpose video and multimedia information. While the retrieval engine and the user interface are the most visible part of the query process, performing the required processing to extract the relevant information and generate the indices for the retrieval engine is the most crucial and challenging component of the overall system. Moreover, the utility of the system is highly dependent on the quality and quantity of the acquired data. We describe an acquisition system that takes advantage of Personal Video Recorder (PVR) functionality of Windows OS to acquire high-quality video and the associated metadata. We discuss several processing algorithms for extracting information that enable efficient searching and browsing of more than 32,000 video programs. 1.

Media consumers who are overwhelmed by multitudinous news content demand a system that would sift through tens or hundreds of broadcast TV channels on a daily basis to capture the most important clips that match users' interests and deliver these personalized clips for easy viewing on a typical TV or home PC. In this paper, we present a personal media alert system that extracts video segments for the interested users. In addition, alerts with pointers to content stored on the home server can be sent to mobile users who are authorized to access the content at home. To make content sharing feasible and scalable, we also propose a P2P payment scheme that would allow a consumer to have access to media clips aggregated on other consumers' personal media systems

This paper introduces a methodology for speech data mining along with the tools that the methodology requires. We show how they increase the productivity of the analyst who seeks relationships among the contents of multiple utterances and ultimately must link some newly discovered context into testable hypotheses about new information. While in its simplest form, one can extend text data mining to speech data mining by using text tools on the output of a speech recognizer, we have found that it is not optimal. We show how data mining techniques that are typically applied to text should be modified to enable an analyst to do effective semantic data mining on a large collection of short speech utterances.

We present a system for automatic monitoring and timely dissemination of multimedia information to a range or mobile information appliances based on each user's interest profile. Multimedia processing algorithms detect and isolate relevant video segments from over twenty television broadcast programs based on a collection or words and phrases specified by the user. Content repurposing techniques are then used to convert the information into a form that is suitable for delivery to the user's mobile devices. Alerts are sent using a number of application messaging and network access protocols including email, short message service (SMS), multimedia messaging service (MMS), voice, session initiation protocol (SIP), fax, and pager protocols. The system is evaluated with respect to performance and user experiences. The MediaAlert system provides an effective and low-cost solution for the timely generation of alerts containing personal, business, and security information.

This paper introduces a methodology for speech data mining along with the tools that the methodology requires. We show how they increase the productivity of the analyst who seeks relationships among the contents of multiple utterances and ultimately must link some newly discovered context into testable hypotheses about new information. While, in its simplest form, one can extend text data mining to speech data mining by using text tools on the output of a speech recognizer, we have found that it is not optimal. We show how data mining techniques that are typically applied to text should be modified to enable an analyst to do effective semantic data mining on a large collection of short speech utterances. For the purposes of this paper, we examine semantic data mining in the context of semantic parsing and analysis in a specific situation involving the solution of a business problem that is known to the analyst. We are not attempting a generic semantic analysis of a set of speech. Our tools and methods allow the analyst to mine the speech data to discover the semantics that best cover the desired solution. The coverage, in this case, yields a set of Natural Language Understanding (NLU) classifiers that serve as testable hypotheses.

In this paper we discuss a simple method to improve the lighting conditions of a scene or video sequence. We focus on modifying the intensities of existing light sources and on inserting additional virtual lights into a scene. We target video conferencing applications where typical lighting conditions (i e the average office or home environment) are poor. Our model is much simpler and is a departure from approaches such as re-lighting, developed in computer-augmented reality (CAR) where the scene is first reconstructed geometrically by means of computer vision techniques We rely on a well-known illumination equation that takes into account light attenuation, plus lambertian and specular reflection Our first scenario models a 3D space in which the image or video frame lies on a 2D plane. Virtual lights are placed in the 3D space and illuminate the 2D image We then propose a refined model for talking head video sequences where the talking head is detected, tracked and its surface is modeled with a 3D ellipsoid. Results show that convincing lighting improvements can be achieved in both scenarios.

This paper addresses the problem of generating semantically meaningful content by integrating information from different media. The goal is to automatically construct a compact yet meaningful abstraction of the multimedia data that can serve as an effective index table, allowing users to browse through large amounts of data in a non-linear fashion with flexibility, efficiency, and confidence. We propose an integrated solution in the context of broadcast news that simultaneously utilizes cues from video, audio, and test to achieve the goal. Some experimental results are presented and discussed.

In this paper, we describe the implementation of a system for reliable recording and on-demand indexed playback of multimedia conferences using the services provided by RTP (Real-time Transport Protcol) and RTCP (Real-Time Control Protocol) over an intranet. Implementation issues are underlined regarding the file structure used to support random access during playback, the indexing techniques and the database searching capabilities.

This paper describes a working system for the automated archiving and selective retrieval of textual, pictorial and auditory information contained in video programs. Video processing performs the task of representing the visual information using a small subset of the video frames. Linguistic processing refines the closed caption text, generates table of contents, and creates links to relevant multimedia documents. Audio and video information are compressed and indexed based on their temporal association with the selected video frames and processed text. The derived information is used to automatically generate a hypermedia rendition of the program contents. This provides a compact representation of the information contained in the video program. It also serves as a textual and pictorial index for selective retrieval of the full-motion video program. This fully automatic system generates HyperText Markup Language (HTML) renditions of television programs, and makes them available for access over the Internet within seconds of their broadcast. This digital library currently contains over 2200 hours of television programs.

We designed a modular system using a combination of shape analysis, color segmentation and motion information for locating reliably heads and faces of different sizes and orientations in complex images. The first of the system's three channels does a shape analysis on gray-level images to determine the location of individual facial features as well as the outlines of heads. In the second channel the color space is analyzed with a clustering algorithm to find areas of skin colors. The color space is first calibrated, using the results from the other channels. In the third channel motion information is extracted from frame differences. Head outlines are determined by analyzing the shapes of areas with large motion vectors. All three channels produce lists of shapes, each marking an area of the image where a facial feature or a part of the outline of a head may be present. Combinations of such shapes are evaluated with n-gram searches to produce a list of likely head positions and the locations of facial features. We tested the system for tracking faces of people sitting in front of terminals and video phones and used it to track people entering through a doorway.