The papers in this special section is to broadly engage the machine learning and multimedia communities on the emerging yet challenging interpretable machine learning. Multimedia is increasingly becoming the “biggest big data,” among the most important and valuable source for insight and information. Many powerful machine learning algorithms, especially deep learning models such as convolutional neural networks (CNNs), have recently achieved outstanding predictive performance in a wide range of multimedia applications, including visual object classification, scene understanding, speech recognition, and activity prediction. Nevertheless, most deep learning algorithms are generally conceived as blackbox methods, and it is difficult to intuitively and quantitatively understand the results of their prediction and inference. Since this lack of interpretability is a major bottleneck in designing more successful predictive models and exploring wider-range useful applications, there has been an explosion of interest in interpreting the representations learned by these models, with profound implications for research into interpretable machine learning in the multimedia community.

Similarity-based image hashing represents crucial technique for visual data storage reduction and expedited image search. Conventional hashing schemes typically feed hand-crafted features into hash functions, which separates the procedures of feature extraction and hash function learning. In this paper, we propose a novel algorithm that concurrently performs feature engineering and non-linear supervised hashing function learning. Our technical contributions in this paper are two-folds: 1) deep network optimization is often achieved by gradient propagation, which critically requires a smooth objective function. The discrete nature of hash codes makes them not amenable for gradient-based optimization. To address this issue, we propose an exponentiated hashing loss function and its bilinear smooth approximation. Effective gradient calculation and propagation are thereby enabled; 2) pre-training is an important trick in supervised deep learning. The impact of pre-training on the hash code quality has never been discussed in current deep hashing literature. We propose a pre-training scheme inspired by recent advance in deep network based image classification, and experimentally demonstrate its effectiveness. Comprehensive quantitative evaluations are conducted. On all adopted benchmarks, our proposed algorithm generates new performance records by significant improvement margins.

This paper is concerned with the generation, acquisition, standardized representation and transport of video metadata. The use of MPEG standards in the design and development of interoperable media architectures and web services is discussed. A high-level discussion of several algorithms for metadata extraction is presented. Some architectural and algorithmic issues encountered when designing services for real-time processing of video streams, as opposed to traditional offline media processing, are addressed. A prototype real-time video analysis system for generating MPEG-7 Audiovisual Description Profile from MPEG-2 transport stream encapsulated video is presented. Such a capability can enable a range of new services such as content-based personalization of live broadcasts given that the MPEG-7 based data models fit in well with specifications for advanced television services such as TV-Anytime and Alliance for Telecommunications Industry Solutions IPTV Interoperability Forum.

Combining content analysis of television programs with quantitative audience measurement can provide insights into customer reactions to advertisements and program content. This work introduces a system architecture that incorporates anonymous audience metrics from an operational IPTV environment with metadata from a content-based analysis of recorded programs. Evaluated on a collection of news programs, the system verifies that events derived from the audience metrics data stream correspond to media segmentation boundaries such as commercial breaks and topic changes. An automated system for executing multimodal media segmentation algorithms for commercial break and topic change detection is also discussed. Better understanding of audience reaction can help IPTV service providers plan infrastructure investments and help in managing multimedia content delivery networks.

The seven papers in this special section focus om recent advances in the growing research field of television.

Graphical probabilistic models play an important role in modern machine learning and pattern recognition. In this half-day tutorial, we introduce the fundamentals of graphical probabilistic modeling, including Bayesian networks, Markov random fields, hidden conditional Random field, etc. We also discuss some applications of graphical probabilistic modeling in multimedia content analysis, for example, video segmentation, video event detection, video sequence matching, and image labeling. The audience will learn the basic of graphical models, and get familiar with the state of the art in multimedia content analysis systems.

The 15 papers in this special issue are extended versions of papers presented at the 2010 IEEE International Conference on Multimedia and Expo (ICME), held in Singapore on July 19-23, 2010. These papers cover a wide range of topics in multimedia including user interface, content understanding, mobility, 3-D processing, storage, and forensics.

In this paper, we describe a novel architecture for content and delivery validation for IPTV systems. The system generates multimedia events derived from IPTV content analysis and its delivery. Events are aggregated, filtered and, through a rules-based system, processed and prioritized to provide monitoring and alerting capabilities. Offline event processing along with a video archiving capability allows for detailed forensic analysis. The novelty of the approach includes the correlation and the combination of content-specific and network-specific events in a normalized representation to allow for complex forensic monitoring and alerting. Applications include content security assurance and metadata verification.

IPTV, enabled by aggregating new technologies in computing, networking, and storage, provides a new information access and entertainment platform in the world of digital evolution. From this perspective where information and entertainment are part of each other, we write this paper to address the issues of IPTV towards seamless infotainment. We focus our discussions to explore such IPTV potentials from three aspects: information access and distribution, improving quality of entertainment, and enabling seamless infotainment. In addition, relevant topics such as IPTV standards and future directions are also discussed.

We propose an approach to searching and browsing video and multimedia using the results of face clustering. Once a set of faces has been detected we use various techniques to create a set of clusters of faces from a particular video. That set of clusters encourages new approaches to browsing and searching video that are not linguistic and can apply to videos internationally, in any language.

Internet protocol TV (IPTV) is a system to deliver TV content to customers by using IP. It has been deployed worldwide by many IPTV service provides, including AT&T, BT, etc. In addition to the core video services provided by the traditional cable TV and satellite TV systems, IPTV provides much richer user interactions and enables additional value added services. This paper presents a prototype for such kind of service: uninterrupted recording and real time media content indexing service for IPTV platforms. While electronic program guides (EPG) list the TV programs that will be broadcast in the future and allow the users to schedule recording for shows that they are interested in, our prototype empowers the users to search and browse any moment of all recent broadcast content for certain channels. Based on the business interest and requirements, this service can be realized in either a centralized architecture (on the network side) or a distributed one (on the individual client side). The developed user interface can be seamlessly integrated on IPTV platforms, and it has been successfully demonstrated on the set top box (STB).

Abstract Building natural language spoken dialogue systems requires large amounts of human transcribed and labeled speech utterances to reach useful operational service performances. Furthermore, the design of such complex systems consists of several manual steps. The User Experience (UE) expert analyzes and defines by hand the system core functionalities: the system semantic scope (call-types) and the dialogue manager strategy that will drive the human–machine interaction. This approach is extensive and error-prone since it involves several nontrivial design decisions that can be evaluated only after the actual system deployment. Moreover, scalability is compromised by time, costs, and the high level of UE know-how needed to reach a consistent design. We propose a novel approach for bootstrapping spoken dialogue systems based on the reuse of existing transcribed and labeled data, common reusable dialogue templates, generic language and understanding models, and a consistent design process. We demonstrate that our approach reduces design and development time while providing an effective system without any application-specific data.

IPTV customers will have access to thousands of video content sources and will require powerful yet intuitive tools to locate desired content. We propose a solution based on stored user interest profiles and multimodal

This paper provides a timely review of influential work in the area of peer-to-peer (P2P) content search. We begin with a survey of text-based P2P search mechanisms and continue with an exposition of content-based approaches followed by a discussion of future directions.

The proposed shot boundary determination (SBD) algorithm contains a set of finite state machine (FSM) based detectors for pure cut, fast dissolve, fade in, fade out, dissolve, and wipe. Support vector machines (SVM) are applied to the cut and dissolve detectors to further boost performance. Our SBD system was highly effective when evaluated in TRECVID 2006 (TREC video retrieval evaluation) and its performance was ranked highest overall.

In this paper, we address the problem of how to improve the automatic speech recognition (ASR) performance on audio conference data by speaker segmentation and speaker adaptation. A new speaker segmentation method is proposed, where the speaker turns and speaker labels are automatically determined. For speaker adaptation, we use Vocal Tract Length Normalization and Maximum Likelihood Linear Regression. On a corpus of multi-speaker teleconferences, the word error rate of the ASR system improves over 4% absolute.

Major casts, for example, the anchor persons or reporters in news broadcast programs and the principle characters in movies, play an important role in video, and their occurrences provide meaningful indices for organizing and presenting video content. This paper describes a new approach for automatically generating a list of major casts in a video sequence based on multiple modalities, specifically, speaker information in audio track and face information in video track. The core algorithm is composed of three steps. First, speaker boundaries are detected and speaker segments are clustered in audio stream. Second, face appearances are tracked and face tracks are clustered in video stream. Finally, correspondences between speakers and faces are determined based on their temporal co-occurrence. A list of major casts is constructed and ranked in an order that reflects each cast's importance, which is determined by the accumulative temporal and spatial presence of the cast. The proposed algorithm has been integrated in a major cast based video browsing system, which presents the face icon and marks the speech locations in time stream for each detected major cast. The system provides a semantically meaningful summary of the video content, which helps the user to effectively digest the theme of the video

This paper presents some of the searching and browsing features of the MIRACLE Video Search Engine. The rapid increase in the generation and dissemination of information and entertainment in video form has created a need for video search engines that facilitate finding and browsing relevant information. MIRACLE is an ongoing project at AT&T Labs aimed at addressing this need. This video search engine combines existing metadata with content-based information that is automatically extracted from the media components, or is obtained from other sources, to achieve this goal. A Web-based user interface provides browsing mechanisms that take advantage of the user’s perceptual abilities to refine the search results. The MIRACLE search engine currently operates on an archive of more than 32,000 hours of video that have been collected and automatically indexed over a ten year period.

This paper describes the content acquisition, indexing and repurposing components of the MIRACLE system. MIRACLE is an ongoing research project at AT&T Labs aimed at creating automated content-based media processing algorithms and systems to collect, organize, index, and repurpose video and multimedia information. While the retrieval engine and the user interface are the most visible part of the query process, performing the required processing to extract the relevant information and generate the indices for the retrieval engine is the most crucial and challenging component of the overall system. Moreover, the utility of the system is highly dependent on the quality and quantity of the acquired data. We describe an acquisition system that takes advantage of Personal Video Recorder (PVR) functionality of Windows OS to acquire high-quality video and the associated metadata. We discuss several processing algorithms for extracting information that enable efficient searching and browsing of more than 32,000 video programs. 1.

Media consumers who are overwhelmed by multitudinous news content demand a system that would sift through tens or hundreds of broadcast TV channels on a daily basis to capture the most important clips that match users' interests and deliver these personalized clips for easy viewing on a typical TV or home PC. In this paper, we present a personal media alert system that extracts video segments for the interested users. In addition, alerts with pointers to content stored on the home server can be sent to mobile users who are authorized to access the content at home. To make content sharing feasible and scalable, we also propose a P2P payment scheme that would allow a consumer to have access to media clips aggregated on other consumers' personal media systems

This paper provides a brief overview of recent progress of peer-to-peer (P2P) technologies for multimedia applications. We provide an overview of the technical challenges, creative solutions and results related to P2P file sharing and content search, and P2P media streaming

This paper introduces a methodology for speech data mining along with the tools that the methodology requires. We show how they increase the productivity of the analyst who seeks relationships among the contents of multiple utterances and ultimately must link some newly discovered context into testable hypotheses about new information. While in its simplest form, one can extend text data mining to speech data mining by using text tools on the output of a speech recognizer, we have found that it is not optimal. We show how data mining techniques that are typically applied to text should be modified to enable an analyst to do effective semantic data mining on a large collection of short speech utterances.

Spoken dialogue systems, which provide automated customer services at call centers, have become more prevalent. It is time consuming to determine a set of call types for the dialogue system by analyzing a large volume of unstructured spoken utterances. Traditional hierarchical agglomerative clustering (HAC) algorithms can bootstrap the call types in an unsupervised way, yet the time and space complexities are huge, especially for a large data set. Based on our observation that spoken utterances containing less than ten terms are common in the spoken dialogue system, we propose an efficient HAC algorithm for short utterances. By utilizing the particular properties of short utterances, we significantly reduce both the time and the space complexities of the clustering algorithm.

We present a system for automatic monitoring and timely dissemination of multimedia information to a range or mobile information appliances based on each user's interest profile. Multimedia processing algorithms detect and isolate relevant video segments from over twenty television broadcast programs based on a collection or words and phrases specified by the user. Content repurposing techniques are then used to convert the information into a form that is suitable for delivery to the user's mobile devices. Alerts are sent using a number of application messaging and network access protocols including email, short message service (SMS), multimedia messaging service (MMS), voice, session initiation protocol (SIP), fax, and pager protocols. The system is evaluated with respect to performance and user experiences. The MediaAlert system provides an effective and low-cost solution for the timely generation of alerts containing personal, business, and security information.

This paper introduces a methodology for speech data mining along with the tools that the methodology requires. We show how they increase the productivity of the analyst who seeks relationships among the contents of multiple utterances and ultimately must link some newly discovered context into testable hypotheses about new information. While, in its simplest form, one can extend text data mining to speech data mining by using text tools on the output of a speech recognizer, we have found that it is not optimal. We show how data mining techniques that are typically applied to text should be modified to enable an analyst to do effective semantic data mining on a large collection of short speech utterances. For the purposes of this paper, we examine semantic data mining in the context of semantic parsing and analysis in a specific situation involving the solution of a business problem that is known to the analyst. We are not attempting a generic semantic analysis of a set of speech. Our tools and methods allow the analyst to mine the speech data to discover the semantics that best cover the desired solution. The coverage, in this case, yields a set of Natural Language Understanding (NLU) classifiers that serve as testable hypotheses.

Scene classification and segmentation are fundamental steps for efficient accessing, retrieving and browsing large amount of video data. We have developed a scene classification scheme using a Hidden Markov Model (HMM)-based classifier. By utilizing the temporal behaviors of different scene classes, HMM classifier can effectively classify presegmented clips into one of the predefined scene classes. In this paper, we describe three approaches for joint classification and segmentation based on HMM, which search for the most likely class transition path by using the dynamic programming technique. All these approaches utilize audio and visual information simultaneously. The first two approaches search optimal scene class transition based on the likelihood values computed for short video segment belonging to a particular class but with different search constrains. The third approach searches the optimal path in a super HMM by concatenating HMM's for different scene classes.

A process for segmenting 2-speaker telephone conversations by speaker with no prior speaker models is described and evaluated. The process consists of an initial segmentation using acoustic change and pause detection, segment clustering, and iterative modeling of segment clusters and resegmentation. The technique has been evaluated on 6, approximately 3 min long, customer care conversations. The technique does not resolve short ( 2 secs) or overlapping segments very well, but is capable of detecting longer segments ( 4 secs) with miss rates of the order of 10% and confusion rates 2% or less.

Major casts, for example, the anchor persons or reporters in news broadcast programs and principle characters in movies play an important role in video, and their occurrences provide good indices for organizing and presenting video content. This paper describes a new approach for automatically generating the list of major casts in a video sequence based on multiple modalities, specifically, both speaker and face information. A list of major casts is created and ordered by the accumulative temporal and spatial presence of corresponding casts. Preliminary simulation results show that the detected major casts are meaningful and the proposed approach is promising.

Evaluating the similarity between two probability distribution functions (PDF) is very important in various research problems. This paper proposes a new metric that computes the distance between two PDFs of mixture type directly from their parameters. It is posed as a linear programming problem and its theoretical properties and performance are analyzed, experimented, and compared with existing measures. In addition, as a proof of concept, we applied the new metric to the problem of audio retrieval where involved PDFs are GMMs (Gaussian mixture model) with 4 mixtures. Experimental results on both synthetic and real data show that this new distance measure is quite promising.

Face detection and tracking are important in video content analysis since the most important objects in most video are human beings. This paper proposes a new approach for combined face detection and tracking in video. The face detection algorithm is a fast template matching procedure using iterative dynamic programming (DP). Although the face detection algorithm is designed for frontal face, the same mechanism can also be applied to track non-frontal faces with online adapted face models. Due to the essence of template matching, the algorithm is capable of comparing the similarity among different faces, which makes it suitable for tracking the same face that occur at disjointed temporal locations in video. While the proposed face detection method provides comparable accuracy as the neural network based approach, it is much faster.

We first present new concepts applicable to the design of multimedia search and retrieval schemes in general, and to MPEG-7 in particular, the multimedia description standard in progress. Raw multimedia data is assumed to exist in the form of programs that typically consist of a combination of media types such as visual, audio, and text. We partition each such media stream into smaller units based on actual physical events. These physical events within each media stream can then be effectively indexed for retrieval. The concept of logical events is introduced next; we define logical events as those that can provide different "views" of the content as may be desired by a user. Such events usually result from either the correlation of events that cross different media types, of by merging recursively chosen events from a lower level within each media type. We then address the related issue of how to develop a practical multimedia information retrieval system that exploits the aforementioned concepts of physical and logical events as well as other aspects such as storage, representation and indexing to enable efficient search, retrieval, and browsing. Finally, we implement the proposed concepts and solutions within a multimedia system that addresses a real application, effective browsing of broadcast news, and evaluate its performance.

This paper addresses the problem of generating semantically meaningful content by integrating information from different media. The goal is to automatically construct a compact yet meaningful abstraction of the multimedia data that can serve as an effective index table, allowing users to browse through large amounts of data in a non-linear fashion with flexibility, efficiency, and confidence. We propose an integrated solution in the context of broadcast news that simultaneously utilizes cues from video, audio, and test to achieve the goal. Some experimental results are presented and discussed.

This paper proposes an integrated approach to discriminate news reporting from everything else in broadcast news data based on both audio and visual information. The separation of news reporting segments from others not only can provide useful indices for video streams but also serves as a pre-processing step for tasks such as speaker identification and speech recognition so that only speech segments are passed on for further processing. A set of audio and visual features are adopted that aim at capturing the intrinsic properties of the underlying classes. Four types of classifiers (threshold fuzzy, Gaussian Mixture Model based, and Support Vector Machine) are tested. Some of the experimental results are presented and discussed in the paper.

Along with the advances in multimedia and Internet technology, a huge amount of data, including digital video and audio, are generated daily. Tools for the efficient indexing and retrieval of such data are indispensable. With multi-modal information present in the data, effective integration is necessary and is still a challenging problem. In this paper, we present four different methods for integrating audio and visual information for video classification based on a hidden Markov model (HMM): direct concatenation, product HMM, two-stage HMM, and integration by neural network. Our results have shown significant improvements over using a single modality.

This paper describes an approach to discriminate news report from others such as commercials and music in broadcast news programs based on audio information. The reported work is part of the effort at AT&T to hierarchically segment broadcast news programs into semantically meaningful units at different levels of abstraction. At the coarse level, using the described approach we preprocess the audio data to pass only the news segments as input to a speaker identification system. To develop a lightweight preprocessing scheme for efficiency, we adopted a set of audio features that are simple to compute yet, based on our observation, statistically capture the intrinsic properties of the audio events to be classified. To improve the performance of the classifier, fuzzy membership functions associated with the features are introduced. Preliminary experimental results are reported which demonstrate the usefulness of the approach.

This paper describes a technique for classifying TV broadcast video using a hidden Markov model (HMM). Here we consider the problem of discriminating five types of TV programs, namely commercials, basketball games, football games, news reports, and weather forecasts. Eight frame-based audio features are used to characterize the low-level audio properties, and fourteen clip-based audio features are extracted based on these frame-based features to characterize the high-level audio properties. For each type of these five TV programs, we build an ergodic HMM using the clip-based features as observation vectors. The maximum likelihood method is then used for classifying testing data using the trained models.

Analysis and classification of the scene content of a video sequence are very important for content-based indexing and retrieval of multimedia databases. We report our research on using the associated audio information for video scene classification. We describe several audio features that have been found effective in distinguishing audio characteristics of different scene classes. Based on these features, a neural net classifier was quite successful in separating audio clips from different TV programs.