In this paper, we analyze the performance of an end-to-end Mortgage Origination (MO) process. The process begins with the submission of a mortgage application by an applicant to a lender and ends with one of the following outcomes: closing, i.e., loan approved by the lender and accepted by the applicant or non-closing, i.e., loan either rejected by the lender, or approved by the lender and not accepted by the applicant. Ranking mortgage applications by their predicted likelihood of closing at various steps in the process is useful for process efficiency and identification of actionable insights to convert applications likely to non-close into those that are likely to close. To build models for ranking applications at any step of the MO process, we take into account customer and product specific attributes of the applications as well as environment attributes and the history of the applications or workflow.The large state-space of the workflow makes the ranking problem challenging. We propose two workflow attributes, each with a state-space of dimension one, based on the number of visits to any step and a particular step (re-work) respectively. We find that incorporating these workflow attributes into the density modeling technique that we develop results in improvement of 4:8 percent in Average Precision over models that only incorporate customer, product and environment attributes. The simple and scalable density modeling technique allows for easy identification of applications that are likely to non-close and consequent corrective action such as change in the attributes of the mortgage product being offered. Further, our results indicate that the model is comparable to Support Vector Machines and superior to Logistic Regression for ranking.

We present an efficient algorithm for providing transactional guarantees when federating a transactional object store with a stream processing engine. The object store allows transactions to programmatically access and update persistent tables as inmemory map objects. The stream processing engine processes continuous queries on events from designated map objects to generate continuously updated derived views. Stream processing traditionally provides an “eventual correctness” guarantee; however our clients insisted upon a uniform transactional quality of service for the federated system: in particular, the system must guarantee that no transaction can observe an updated map object and a stale value of a derived view. A straightforward approach would entail executing all stream processing within the scope of transactions, and this comes at a performance cost. Our approach, based on a hybrid optimistic concurrency control technique, allows stream processing to proceed in parallel, usually after commit time. We present our algorithm along with experimental performance evaluations of both approaches.

We present and evaluate a low-overhead approach for achieving high-availability in distributed event-processing middleware systems consisting of networks of stateful software components that communicate by either one-way (send) or two-way (call) messages. The approach is based on transparently augmenting each component to produce a deterministic component whose state can be recovered by checkpoint and replay. Determinism is achieved by augmenting messages with virtual times, and by scheduling message handling in virtual time order. Scheduling delays are reduced by computing virtual times with estimators: deterministic functions that approximate the expected real times of arrival. We describe our algorithms, show how Java components can be transparently augmented with checkpointing code and with good estimators, discuss how our deterministic runtime can be tuned to reduce overhead, and provide experimental results to measure the overhead of determinism relative to non-determinism.

Recent growth in broadband access and proliferation of small personal devices that capture images and videos has led to explosive growth of multimedia content available everywhere - from personal disks to the Web. While digital media capture and upload has become nearly universal with newer device technology, there is still a need for better tools and technologies to search large collections of multimedia data and to find and deliver the right content to a user according to her current needs and preferences. A renewed focus on the subjective dimension in the multimedia lifecycle, from creation, distribution, to delivery and consumption, is required to address this need beyond what is feasible today. Integration of the subjective aspects of the media itself - its affective, perceptual, and physiological potential (both intended and achieved), together with those of the users themselves will allow for personalizing the content access, beyond today's facility. This integration, transforming the traditional multimedia information retrieval (MIR) indexes to more effectively answer specific user needs, will allow a richer degree of personalization predicated on user intention and mode of interaction, relationship to the producer, content of the media, and their history and lifestyle. In this paper, we identify the challenges in achieving this integration, current approaches to interpreting content creation processes, to user modelling and profiling, and to personalized content selection, and we detail future directions. The structure of the paper is as follows: In Section I, we introduce the problem and present some definitions. In Section II, we present a review of the aspects of personalized content and current approaches for the same. Section III discusses the problem of obtaining metadata that is required for personalized media creation and present eMediate as a case study of an integrated media capture environment. Section IV presents the MAGIC system as a case study of capturing effective descriptive data and putting users first in distributed learning delivery. The aspects of modelling the user are presented as a case study in using user's personality as a way to personalize summaries in Section V. Finally, Section VI concludes the paper with a discussion on the emerging challenges and the open problems.

In this paper, we describe the SMILE (Smart MIddleware, Light Ends) system which is one of the earliest systems built in the area of distributed event stream processing. SMILE unites the "publish-subscribe" model of messaging middleware with the "continuous query" model of database systems. In SMILE, information producers, which may be sensors, applications, or databases, generate streams of events, such as RFID data, news items or stock trades; consumers specify stateful subscriptions to derived views, such as "individuals trading top 5 total volume of stock Y within x minutes before a major news story about the same company Y"; the SMILE system constructs and deploys a dataflow network of computations which process events from producers, compute derived views and deliver continuous and timely updates of subscribed views to consumers. Research challenges addressed by SMILE include: rigorously defining the semantics for correct state delivery; implementing this semantics in the presence of failure; distributing computations optimally over a network of multiple servers; scheduling and controlling message flow in this network; smoothly integrating this technology with other components, e.g., databases, services, user interfaces. SMILE is applicable to any "sense and respond" scenario that requires monitoring and processing of events as they are generated. Applications span multiple industries, e.g., monitoring financial opportunities and detecting potential fraud in financial industry, systems management and alerts in data centers, inventory management in RFID applications, and business performance management. In this paper, we describe the SMILE system, its architecture, services supported, and implementation details, and its use in multiple application scenarios.

For the auto insurance claims process, improvements in the First Notice of Loss and rapidity in the investigation and evaluation of claims could drive significant values by reducing loss adjustment expense. This paper proposes a novel application where advanced technologies in image analysis and pattern recognition are applied to automatically identify and characterize automobile damage. Success in this will allow some cases to proceed without human adjusters, while others to proceed more efficiently, thus ultimately shortening the time between the first Notice of Loss and the final payout. To investigate its feasibility, we built a prototype system which automatically identifies the damaged area(s) based on the comparison of before-and after-accident automobile images. Performance of the prototype system has been evaluated on images taken from forty scaled model cars under reasonably controlled environments, and encouraging results were obtained. It is our belief that, with the advancement of image analysis and pattern recognition technologies, the proposed idea could evolve into a very promising application area where the auto insurance industry could significantly benefit.

The "Tools for Automatic Generation of Learning Object Metadata" project addresses the requirement of developing advanced distributed learning delivery architecture and services for a large US government agency. We have developed a Webbased system called MAGIC (Metadata Automated Generation for Instructional Content) to assist content authors and course developers in generating metadata for learning objects and information assets to enable wider reuse of these objects across departments and organizations. Using the MAGIC system, content authors review and edit automatically-generated metadata sufficient to register and describe their assets for use and discovery in current and future distributed learning applications complying with the ADL SCORM standard. Course developers can use the system to assist in the conversion of existing courses to SCORM format or in developing new SCORM courses. The MAGIC system includes software tools to analyze and extract descriptive metadata from instructional videos, training documents, and other information assets. The tools generate some of the most critical SCORM metadata completely automatically. Benefits of MAGIC include easier reuse and repurposing, improved interoperability, and more timely registration of content for use by course developers. In this paper, we describe the system architecture, analysis tools developed, and services supported. A live demonstration of the system illustrating several use cases of the system will be presented at the conference, with a discussion of results from user studies and evaluation of the system.

This paper presents our latest work on structuring instructional videos into units of atomic topical segment so as to facilitate topic-based video browsing and offer efficient video authoring. Specifically, we developed a comprehensive text analysis component to first extract informative text cues such as keyword synonym set and sentence boundary information, from a video's transcript. These text cues are then applied with various audiovisual cues such as silence/music break and speech similarity, to identify topical segments. Early experiments carried out on collections of real data from targeted user communities have yielded good results, and the user feedback on using the generated topical segment information is very encouraging.

Automatic media content analysis and understanding for efficient topic searching and browsing are current challenges in the management of e-learning content repositories. This paper presents our current work on analyzing and structuralizing instructional videos using pure audio information. Specifically, an audio classification scheme is first developed to partition the sound-track of an instructional video into homogeneous audio segments where each segment has a unique sound type such as speech or music. We then apply a statistical approach to extract discussion scenes in the video by modeling the instructor with a Gaussian mixture model (GMM) and updating it on the fly. Finally, we categorize obtained discussion scenes into either two-speaker or multispeaker discussions using an adaptive mode-based clustering approach. Experiments carried out on four training videos and five IBM MicroMBA class videos have yielded encouraging results. It is our belief that by detecting and identifying various types of discussions, we are able to better understand and annotate the learning media content and subsequently facilitate its content access, browsing, and retrieval

This paper presents our latest work on identifying frame content types for understanding learning media content. In particular, we categorize frames into six classes namely, slide, Web-page, instructor, audience, picture-in-picture and miscellaneous, which make up salient narrative modes in learning videos. Various image and video analysis approaches are explored to achieve this task. Preliminary experiments carried out on three recorded seminars have yielded encouraging results. The identification of fine-grained visual content types can assist us in content understanding, access, browsing and searching of generic learning videos

This paper presents our latest work on building a system called MAGIC (Metadata Automated Generation for Instructional Content) that will automatically identify segments and generate critical metadata conforming with the SCORM (Sharable Content Object Reference Model) standard for instructional content. Various content analytics engines are utilized to automatically generate key metadata, which include audiovisual analysis modules that recognize semantic sound categories and identify narrators and informative text segments; text analysis modules that extract title, keywords and summary from text documents; and a text categorizer that classifies a document according to a pre-generated taxonomy. With MAGIC, instructional content developers can generate and edit SCORM metadata to richly describe their content asset for use in distributed learning applications. Experimental results obtained from collections of real data from targeted user communities will be presented.

Automatic content analysis and annotation for efficient search and browsing of topics in instructional videos are current challenges in the management of e-learning content repositories. This paper presents our current work on classifying the soundtrack of instructional videos into seven distinct audio classes using the support vector machine (SVM) technology. The classification results are then used to partition a video into homogeneous audio segments, which forms the fundamental basis for its higher-level content analysis and exploration. Initial experiments carried out on three education and four training videos totalling 185 minutes have yielded an average 97.9% classification accuracy. The performance comparisons between the SVM-based, the decision tree (DT)-based and the threshold-based audio classification schemes further demonstrates the superiority of the proposed scheme.

This paper addresses the problem of detecting discussion scenes in instructional videos using statistical approaches. Specifically, given a series of speech segments separated from the audio tracks of educational videos, we first model the instructor using a Gaussian mixture model (GMM), then a four-state transition machine is designed to extract discussion scenes in real-time, based on detected instructor-student speaker change points. Meanwhile, we keep updating the GMM model to accommodate the instructor's voice variation along time. Promising experimental results have been achieved on five educational (IBM MicroMBA program) videos, and very interesting instruction/teaching patterns have been observed. The extracted scene information would facilitate the semantic indexing and structuralization of instructional video content.

This paper describes our current effort on analyzing the contents of discussion scenes in instructional videos based on a clustering technique. Specifically, given a discussion scene pre-detected from an education or training video, we first apply a mode-based clustering approach to group all speech segments into an optimal number of clusters where each cluster contains speech from one speaker; we then analyze the discussion patterns in the scene, and subsequently classify it into either a 2-speaker or multi-speaker discussion. Encouraging classification results have been achieved on 122 discussion scenes detected from five IBM MicroMBA videos. Moreover, we have also observed fairly good performance on the speaker clustering scheme, which demonstrates the superiority of the proposed clustering approach. Undoubtedly, the discussion scene information output from this analysis scheme would facilitate the content browsing, searching and understanding of instructional videos.

In this paper, we study the application of a scene structure visualizing technique called double-Ring Take-Transition-Diagram (DR-TTD). This technique presents takes and their transitions during a film scene via nodes and edges of a 'graph' consisting of two rings as its back-bone. We describe how certain filmic elements such as montage, centre/cutaway, dialogue, temporal flow, zone change, dramatic progression, shot association, scene introduction, scene resolution, master shot and editing orchestration can be identified from a scene through the signature arrangements of nodes and edges in the DR-TTD.

Traditional fingerprint acquisition is limited to single-image capture and processing. With the advent of faster capture hardware, faster processors, and advances in video compression standards, newer systems can capture and exploit video signals for tasks that are difficult using a single image. We propose the use of fingerprint video sequences to investigate detecting two aspects of the dynamic behavior of fingerprints. Specifically, we are interested in the detection of distortion of fingerprint impressions due to excessive force and the detection of the positioning of fingers during image capture. These issues often lead to difficulties in establishing a precise match between acquired images. The proposed techniques investigate dynamic characteristics of fingerprints across video sequence frames. A significant advantage of our approach for distortion analysis is that it works directly on MPEG-1,-2 encoded fingerprint video bitstreams. The proposed methods have been tested on the NIST-24 live-scan fingerprint video database and the results are promising. We also describe a new concept called the "resultant biometrics", a new type of biometrics which has both a physiological, physical (e.g., force, torque, linear motion, rotation) component and/or a temporal characteristic, added by a subject to an existing biometric. This resultant biometric is both desirable and efficient in terms of easy modification of compromised biometrics and is harder to produce with spoof body parts.

This work addresses the challenge of extracting structure in educational and training media based on the type of material that is presented during lectures and training sessions. The narrative structure that arises out of a use of different types of presentation content such as slides, web pages, and white board writings is useful in segmenting an educational video for easy content access and nonlinear browsing of the material presented. Automatically detecting sections of videos as delineated by the use of supplementary teaching/instructional visual aids allows for structuralizing educational video with high level of semantics, and provides a concise means for organizing learning content according to the needs of different users in e-learning scenarios. Experiments on the videos from classrooms show encouraging results with discriminating different narrative sections in the proposed presented-material based video structuralization scheme.

This work constitutes the first attempt to extract an important narrative structure, the 3-Act story telling paradigm, in film. This narrative structure is prevalent in the domain of film as it forms the foundation and framework in which the film can be made to function as an effective tool for story telling, and its extraction is a vital step in automatic content management for film data. A novel act boundary likelihood function for Act 1 is derived using a Bayesian formulation under guidance from film grammar, tested under many configurations and the results are reported for experiments involving 25 full length movies. The formulation is shown to be a useful tool in both the automatic and semi-interactive setting for semantic analysis of film.

We examine localised sound energy patterns, or events, that we associate with high level affect experienced with films. The study of sound energy events in conjunction with their intended affect enable the analysis of film at a higher conceptual level, such as genre. The various affect/emotional responses we investigate in this paper are brought about by well established patterns of sound energy dynamics employed in audio tracks of horror films. This allows the examination of the thematic content of the films in relation to horror elements. We analyse the frequency of sound energy and affect events at a film level as well as at a scene level, and propose measures indicative of the film genre and scene content. Using 4 horror, and 2 non-horror movies as experimental data we establish a correlation between the sound energy event types and horrific thematic content within film, thus enabling an automated mechanism for genre typing and scene content labeling in film.

In this paper, we propose novel computational models for the extraction of high level expressive constructs related to, namely thematic and dramatic functions of the content shown in educational and training videos. Drawing on the existing knowledge of film theory, and media production rules and conventions used by the filmmakers, we hypothesize key aesthetic elements contributing to convey these functions of the content. Computational models to extract them are then formulated and their performance evaluated on a set of ten educational and training videos is presented.

In this paper, we focus on the 'reverse editing' problem in movie analysis, i.e., the extraction of film takes, original camera shots that a film editor extracts and arranges to produce a finished scene. The ability to disassemble final scenes and shots into takes is essential for nonlinear browsing, content annotation and the extraction of higher order cinematic constructs from film. In this work, we investigate agglomerative hierarchical clustering methods along with different similarity metrics and group distances for this task, and demonstrate our findings with 10 movies.

In this paper, we propose a novel solution for segmenting an instructional video into hierarchical topical sections. Incorporating the knowledge of education-oriented film theory with our previous study of expressive functions namely the content density and the thematic functions, we develop an algorithm to effectively structuralize an instructional video into a two-tiered hierarchy of topical sections at the main and sub-topic levels. Our experimental results on a set of ten industrial instructional videos demonstrate the validity of the detection scheme.

This paper addresses the challenge of bridging the semantic gap between the rich meaning users desire when they query to locate and browse media and the shallowness of media descriptions that can be computed in today's content management systems. To facilitate high-level semantics-based content annotation and interpretation, we tackle the problem of automatic decomposition of motion pictures into meaningful story units, namely scenes. Since a scene is a complicated and subjective concept, we first propose guidelines from film production to determine when a scene change occurs. We then investigate different rules and conventions followed as part of film grammar that would guide and shape an algorithmic solution for determining a scene. Two different techniques using intershot analysis are proposed as solutions in this paper. In addition, we present different refinement mechanisms, such as film-punctuation detection founded on film grammar, to further improve the results. These refinement techniques demonstrate significant improvements in overall performance. Furthermore, we analyze errors in the context of film-production techniques, which offer useful insights into the limitations of our method.

Research in automated analysis of digital media content has led to a large collection of low-level feature extractors, such as face detectors, videotext extractors, speech and speaker identifiers, people/vehicle trackers, and event locators. These media metadata are often symbolic rather than continuous-valued, and pose significant difficulty to subsequent tasks such as classification and dimensionality reduction which traditionally deal with continuous-valued data. This paper proposes a novel mechanism that extends tasks traditionally limited to continuous-valued feature spaces, such as (a) dimensionality reduction, (b) de-noising, and (c) clustering, to domains with symbolic features. To this end, we introduce new kernels based on well-known distance metrics, and prove Mercer validity of these kernels for analyzing symbolic feature spaces. We demonstrate their usefulness within the context of kernel-space methods such as Kernel PCA and SVM, in classifying machine learning datasets from the UCI repository and in temporal clustering and tracking of videotext in multimedia. We show that the generalized kernels help capture information from symbolic feature spaces, visualize symbolic data, and aid tasks such as classification and clustering, and therefore are useful in multimodal analysis of multimedia.

To enable high-level semantic indexing of video, we tackle the problem of automatically structuring motion pictures into meaningful story units, namely scenes. In our recent work, drawing guidance from film grammar, we proposed an algorithmic solution for extracting scenes in motion pictures based on a shot neighborhood color coherence measure. In this paper, we extend our work by presenting various refinement mechanisms, inspired by the knowledge of film devices that are brought to bear while crafting scenes, to further improve the results of the scene detection algorithm. We apply the enhanced algorithm to ten motion pictures and demonstrate the resulting improvements in performance.

Article Share on Media semantics: who needs it and why? Authors: Chitra Dorai Organizers OrganizersView Profile , Andreas Mauthe View Profile , Frank Nack Organizers OrganizersView Profile , Lloyd Rutledge View Profile , Thomas Sikora View Profile , Herbert Zettl View Profile Authors Info & Claims MULTIMEDIA '02: Proceedings of the tenth ACM international conference on MultimediaDecember 2002 Pages 580–583https://doi.org/10.1145/641007.641123Online:01 December 2002Publication History 9citation676DownloadsMetricsTotal Citations9Total Downloads676Last 12 Months8Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Automatically partitioning instructional videos into topic sections is a challenging problem in e-learning environments for efficient content management and cataloging. This paper addresses this problem by proposing a novel density function to delineate sections underscored by changes in topics in instructional and training videos. The content density function draws guidance from the observation that topic boundaries coincide with the ebb and flow of the 'density' of content shown in these videos. Based on this function, we propose two methods for high-level segmentation by determining topic boundaries. We study the performance of the two methods on eight training videos, and our experimental results demonstrate the effectiveness and robustness of the two proposed high-level segmentation algorithms for learning media.

Using film grammar as the underpinning, we study the extraction of structures in video based on color using a wide configuration of clustering methods combined with existing and new similarity measures. We study the visualisation of these structures, which we call Scene-Cluster Temporal Charts and show how it can bring out the interweaving of different themes and settings in a film. We also extract color events that filmmakers use to draw/force a viewer's attention to a shot/scene. This is done by first extracting a set of colors used rarely in film, and then building a probabilistic model for color event detection. We demonstrate with experimental results from ten movies that our algorithms are effective in the extraction of both scene-cluster temporal charts and color events.

The paper addresses the challenge of bridging the semantic gap that exists between the simplicity of features that can be currently computed in automated content indexing systems and the richness of semantics in user queries posed for media search and retrieval. It proposes a unique computational approach to extraction of expressive elements of motion pictures for deriving high-level semantics of stories portrayed, thus enabling rich video annotation and interpretation. This approach, motivated and directed by the existing cinematic conventions known as film grammar, as a first step toward demonstrating its effectiveness, uses the attributes of motion and shot length to define and compute a novel measure of tempo of a movie. Tempo flow plots are defined and derived for a number of full-length movies and edge analysis is performed leading to the extraction of dramatic story sections and events signaled by their unique tempo. The results confirm tempo as a useful high-level semantic construct in its own right and a promising component of others such as rhythm, tone or mood of a film. In addition to the development of this computable tempo measure, a study is conducted as to the usefulness of biasing it toward either of its constituents, namely motion or shot length. Finally, a refinement is made to the shot length normalizing mechanism, driven by the peculiar characteristics of shot length distribution exhibited by movies. Results of these additional studies, and possible applications and limitations are discussed.

Videotext refers to text superimposed on video frames. A videotext based multimedia description scheme has been adopted in the MPEG-7 standard. A study of published work in the area of videotext extraction and recognition reveals that, despite previous interest, a reliable general purpose video character recognition (VCR) system is yet to be developed. In our research and development of a character recognition algorithm designed specifically for the low resolution output from automatic videotext extractors, we observed that raw VCR accuracies obtained using various classifiers including kernel space methods such as SVMs, are inadequate for accurate video annotation and browsing. Intelligent post-processing mechanisms that are supported by general data characteristics of the domain are hence, required for performance improvement. We describe one such method, referred to as the font context analysis, which works independently of the raw character recognition technique. As a result, it can be easily implemented in conjunction with other VCR algorithms being developed elsewhere, and offer the same performance gains. Experimental results on various video streams show notable improvements in recognition rates with our system incorporating a SVM-based character recognition mechanism and font context analysis.

This paper examines film rhythm, an important expressive element in motion pictures, based on our ongoing study to exploit film grammar as a broad computational framework for the task of automated film and video understanding.Of the many, more or less elusive, narrative devices contributing to film rhythm, this paper discusses motion characteristics that form the basis of our analysis, and presents novel computational models for extracting rhythmic patterns induced through a perception of motion.In our rhythm model, motion behaviour is classified as being either nonexistent, fluid or staccato for a given shot.Shot neighbourhoods in movies are then grouped by proportional makeup of these motion behavioural classes to yield seven high-level rhythmic arrangements that prove to be adept at indicating likely scene content (e.g.dialogue or chase sequence) in our experiments.Underlying causes for this level of codification in our approach are postulated from film grammar, and are accompanied by detailed demonstration from real movies for the purposes of clarification.

Videotext refers to text superimposed on still images and video frames, and a videotext based Multimedia Description Scheme has recently been adopted into the MPEG-7 standard as one of the normative media content description interfaces. While much of the previous work including ours concentrates on the task of locating and extracting text from the video frames automatically, very little research has focused on reliably recognizing segmented text. The low resolution of videotext, unconstrained font styles and sizes, poor separation of characters often resulting from video compression and decoding, all pose severe problems even to commercial OCRs in recognizing videotext accurately. This paper describes novel end-to-end video character recognition system featuring new character attributes emphasizing macro shapes, a Support Vector Machine-based character classifier, videotext object synthesis, font context analysis, and temporal contiguity analysis, to successfully address the issues confounding accurate videotext recognition. We present results from our experiments with real video data that demonstrate the strengths of this system.

In this paper, we study the sound tracks in films and their indexical semiotic usage by developing a classification system that detects complex sound scenes and their constituent sound events in cinema. We investigate two main issues in this paper: Determination of what constitutes the presence of a high level sound scene and inferences about the thematic content of the scene that can be drawn from this presence, and classification of environmental sounds in the audio track of the scene, to assist in the automatic detection of the high level scene. Experiments with our classification system on pure sounds resulted in a correct event classification rate of 88.9%. When the audio content of a number of film scenes was examined, though a lower accuracy resulted with sound event detection due to the presence of mixed sounds, the film audio samples were generally classified with the correct high-level sound scene label, enabling correct inferences about the story content of the scenes.

E-learning or learning via the Web is now changing the way universities and coprporations offer education and training. Rather than using conventional means for remotely distributing and delivering courses and classroom lectures, universities and enterprises are moving towards Web-based academic and training offerings. While content acquisition and distribution can be automated in a systematic fashion, research challenges remain in the area of personalized content delivery, access, and augmented interaction. Automated content indexing and annotation for easy and efficient access and navigation of various media elements associated with the course material become key takss in flexible content delivery and use. E-Learning Media Navigator (ELM-N) from IBM Research is a system with which a user can access and interact with online courses presented as audiovisual material, along with related slides and white broad contents in a time-synchronized fashion. In this demonstration, the system capabilities and research technologies behind them will be presented.

We develop an algorithm for the detection and classification of affective sound events underscored by specific patterns of sound energy dynamics. We relate the portrayal of these events to proposed high level affect or emotional coloring of the events. In this paper, four possible characteristic sound energy events are identified that convey well established meanings through their dynamics to portray and deliver certain affect, sentiment related to the horror film genre. Our algorithm is developed with the ultimate aim of automatically structuring sections of films that contain distinct shades of emotion related to horror themes for nonlinear media access and navigation. An average of 82% of the energy events, obtained from the analysis of the audio tracks of sections of four sample films corresponded correctly to the proposed affect. While the discrimination between certain sound energy event types was low, the algotithm correctly detected 71% of the occurrences of the sound energy events within audio tracks of the films analyzed, and thus forms a useful basis for determining affective scenes characteristic of horror in movies.

Videotext refers to text superimposed on video frames and it enables automatic content annotation and indexing of large video and image collections. Its importance is underscored by the fact that a videotext-based multimedia description scheme has recently been adopted into the MPEG-7 standard. A study of published work in the area of automatic videotext extraction and recognition reveals that, despite recent interest, a reliable general purpose video character recognition (VCR) system is yet to be developed. In our development of a VCR system designed specifically to handle the low resolution output from videotext extractors, we observed that raw VCR accuracies obtained using various classifiers including kernel space methods such as SVM, are inadequate for accurate video annotation. We propose an intelligent postprocessing mechanism that is supported by general data characteristics of this domain for VCR performance improvement. We describe temporal contiguity analysis, which works independently of the raw character recognition technique and works well even for moving videotext. This novel mechanism can be easily implemented in conjunction with VCR algorithms being developed elsewhere to offer the same performance gains. Experimental results on various video streams show notable improvements in recognition rates with our system incorporating a SVM-based recognition engine and temporal contiguity analysis.

Distortions in fingerprint images arising from the elasticity of finger skin and the pressure and movement of fingers during image capture lead to great difficulties in establishing a match between multiple images acquired from a single finger. In a single fingerprint image depicting a finger at some given instant of time, it is difficult to get any distortion information. Further, static two-dimensional or three-dimensional (electronic) copies of fingerprints can be fabricated and used to spoof remote biometric security systems since the input required by the systems is not a function of time. This paper addresses these issues, by proposing the novel use of fingerprint video sequences to investigate and exploit dynamic behaviors manifested by fingers over time during image acquisition. In particular, we present a novel approach to detect and estimate distortion occurring in fingerprint video streams. Our approach directly works on MPEG-{1, 2} encoded fingerprint video bitstreams to estimate interfield flow without decompression, and uses flow characteristics to investigate temporal behaviour of the fingerprints. The joint temporal and motion analysis leads to a novel technique to detect and characterize distortion reliably. The proposed method has been tested on the NIST 24 database and the results are very promising.

This paper presents an original computational approach for the extraction of movie tempo for deriving story sections and events that convey high level semantics of stories portrayed in motion pictures, thus enabling better video annotation and interpretation systems. This approach, inspired by the existing cinematic conventions known as film grammar, uses the attributes of motion and shot length to define and compute a novel continuous measure of the tempo of a movie. Tempo flow plots are derived for several full-length motion pictures and edge detection is performed to extract dramatic story sections and events occurring in the movie, underlined by their unique tempo. The results confirm the reliable detection of actual distinct tempo changes and serve as a useful index for the dramatic development and narration of the story in motion pictures.

We present improved algorithms for automatic fade and dissolve detection in digital video analysis. We devise new two-step algorithms for fade and dissolve detection and introduce a method for eliminating false positives from a list of detected candidate transitions. In our detailed study of these gradual shot transitions, our objective has been to accurately classify the type of transitions (fade-in, fade-out, and dissolve) and to precisely locate the boundary of the transitions. This distinguishes our work from early work in scene change detection which focuses on identifying the existence of a transition rather than its precise temporal extent. We evaluate our algorithms against two other commonly used methods on a comprehensive data set, and demonstrate the improved performance due to our enhancements.

Presents a set of computational features originating from our study of editing effects, motion, and color used in videos, for the task of automatic video categorization. These features besides representing human understanding of typical attributes of different video genres, are also inspired by the techniques and rules used by many directors to endow specific characteristics to a genre-program which lead to certain emotional impact on viewers. We propose new features whilst also employing traditionally used ones for classification. This research, goes beyond the existing work with a systematic analysis of trends exhibited by each of our features in genres such as cartoons, commercials, music, news, and sports, and it enables an understanding of the similarities, dissimilarities, and also likely confusion between genres. Classification results from our experiments on several hours of video establish the usefulness of this feature set. We also explore the issue of video clip duration required to achieve reliable genre identification and demonstrate its impact on classification accuracy.

This work seeks to lay the framework of film grammar over the video to be analysed. We use the shot attributes of motion and shot length to produce a novel continuous measure of one of the aesthetic elements of films, namely the movie tempo. We refer to our previous work detailing the study of this construct and its automatic derivation, and also demonstrating its usefulness as an expressive element and as a sound basis for higher semantic descriptions such as dramatic events and story elements. Initial assessment of tempo was performed in our study on the basis that the relative importance of both shot length and motion in formulating the tempo function was the same. In this paper, we analyze their relative contributions to tempo, and demonstrate how these two factors can be manipulated to influence audience perception of movie time.

We present improved algorithms for cut, fade, and dissolve detection which are fundamental steps in digital video analysis. In particular, we propose a new adaptive threshold determination method that is shown to reduce artifacts created by noise and motion in scene cut detection. We also describe new two-step algorithms for fade and dissolve detection, and introduce a method for eliminating false positives from a list of detected candidate transitions. In our detailed study of these gradual shot transitions, our objective has been to accurately classify the type of transitions (fade-in, fade-out, and dissolve) and to precisely locate the boundary of the transitions. This distinguishes our work from other early work in scene change detection which tends to focus primarily on identifying the existence of a transition rather than its precise temporal extent. We evaluate our improved algorithms against two other commonly used shot detection techniques on a comprehensive data set, and demonstrate the improved performance due to our enhancements.

Region extraction and labeling is a fundamental process in building computer systems that automatically analyze and interpret images and image sequences of a scene. It involves extracting "meaningful" regions and their attributes from an image that are of interest in a given domain, e.g., a human face in face recognition and surveillance applications, a vehicle in intelligent highway monitoring, or an industrial tool in automated manufacturing and inspection applications. In this paper, we present a generalized and fast region labeling algorithm that can be applied to numerous image processing tasks: region extraction in binary images, region segmentation from gray scale or color images, region boundary extraction, chain code extraction, run-length coding of images, and image restoration from chain codes. The general applicability of our algorithm stems from its strengths in identifying the fundamental and common computational steps and providing robust and efficient procedures to accomplish these steps without requiring multiple scans of images. Our algorithm is shown to be more effective in terms of space and computational efficiency, in comparison with many existing approaches to region labeling and extraction. Results from our experiments with over 5,000 frames obtained from MPEG-1 video bitstreams demonstrate the good performance of our algorithm in terms of video text segmentation accuracy and computational efficiency.

Efficient content-based search and retrieval of video databases has become an important industrial and consumer application due to rapid proliferation of compressed digital video data on the Internet and corporate Intranets, and due to the launch of high definition television (HDTV) broadcasts in 1998. We have developed the first HDTV video content management system that automatically analyzes motion occurring in MPEG-2 encoded videos within the compressed domain itself. Our system produces descriptors characterizing the global motion in videos for content retrieval and repurposing applications in a digital television studio. These motion-direction and magnitude based video labels can be directly, incorporated as annotation indexes into a database for querying, or used to construct higher level event descriptions of videos. Results from our ongoing experiments with tens of thousands of frames obtained from several MPEG-1,2 video streams of various genres demonstrate the good performance of our annotation system in terms of motion identification accuracy and computational efficiency.

Efficient content-based retrieval of image and video databases is an important application due to rapid proliferation of digital video data on the Internet and corporate intranets. Text either embedded or superimposed within video frames is very useful for describing the contents of the frames, as it enables both keyword and free-text based search, automatic video logging, and video cataloging. We have developed a scheme for automatically extracting text from digital images and videos for content annotation and retrieval. We present our approach to robust text extraction from video frames, which can handle complex image backgrounds, deal with different font sizes, font styles, and font appearances such as normal and inverse video. Our algorithm results in segmented characters that can be directly processed by an OCR system to produce ASCII text. Results from our experiments with over 5000 frames obtained from twelve MPEG video streams demonstrate the good performance of our system in terms of text identification accuracy and computational efficiency.

Automatic 3D object model construction is important in applications ranging from manufacturing to entertainment, since CAD models of existing objects may be either unavailable or unusable. We describe a prototype system for automatically registering and integrating multiple views of objects from range data. The results can then be used to construct geometric models of the objects. New techniques for handling key problems such as robust estimation of transformations relating multiple views and seamless integration of registered data to form an unbroken surface have been proposed and implemented in the system. Experimental results on real surface data acquired using a digital interferometric sensor as well as a laser range scanner demonstrate the good performance of our system.

We address the problem of representing and recognizing 3D free-form objects when (1) the object viewpoint is arbitrary, (2) the objects may vary in shape and complexity, and (3) no restrictive assumptions are made about the types of surfaces on the object. We assume that a range image of a scene is available, containing a view of a rigid 3D object without occlusion. We propose a new and general surface representation scheme for recognizing objects with free-form (sculpted) surfaces. In this scheme, an object is described concisely in terms of maximal surface patches of constant shape index. The maximal patches that represent the object are mapped onto the unit sphere via their orientations, and aggregated via shape spectral functions. Properties such as surface area, curvedness, and connectivity, which are required to capture local and global information, are also built into the representation. The scheme yields a meaningful and rich description useful for object recognition. A novel concept, the shape spectrum of an object is also introduced within the framework of COSMOS for object view grouping and matching. We demonstrate the generality and the effectiveness of our scheme using real range images of complex objects.

We address the problem of constructing view aspects of 3D free-form objects for efficient matching during recognition. We introduce a novel view representation based on "shape spectrum" features, and propose a general and powerful technique for organizing multiple views of objects of complex shape and geometry into compact and homogeneous clusters. Our view grouping technique obviates the need for surface segmentation and edge detection. Experiments on 6,400 synthetically generated views of 20 free-form objects and 100 real range images of 10 sculpted objects demonstrate the good performance of our shape spectrum based model view selection technique.

This paper deals with robust registration of object views in the presence of uncertainties and noise in depth data. Errors in registration of multiple views of a 3D object severely affect view integration during automatic construction of object models. We derive a minimum variance estimator (MVE) for computing the view transformation parameters accurately from range data of two views of a 3D object. The results of our experiments show that view transformation estimates obtained using MVE are significantly more accurate than those computed with an unweighted error criterion for registration.

We address the problem of recognizing 3D rigid free-form objects using dense range data when the objects can be imaged from arbitrary viewpoints and the objects vary in shape and complexity. We propose a multi-level matching strategy that employs shape spectral analysis and features derived from the COSMOS representations of free-form objects for fast and efficient recognition. We demonstrate that with a large model database of object views, a small set of ranked candidate matches can be selected quickly using shape spectrum based matching for further verification. We propose a graph-based matching scheme for view hypothesis verification using COSMOS representations of object views to establish the correct identity and the pose of the sensed object. Preliminary experimental results on a database containing views often different objects are shown to demonstrate the effectiveness of the COSMOS-based 3D object recognition system.

Automatic 3D object model construction is important in applications ranging from manufacturing to entertainment industry, since CAD models of existing objects may be either unavailable or unusable. We describe a prototype system for automatically registering and integrating multiple views of objects using surface depth data to construct their geometric models. New techniques for handling key issues such as robust estimation of view transformations relating multiple views and seamless integration of registered data to form an unbroken surface have been proposed and implemented in our system. Our system has been tested on depth data that were obtained using digital interferometry techniques as well as a laser range scanner. Experimental results on real object surfaces demonstrate the good performance of our system.

We address the problem of representing and recognizing arbitrarily curved 3D rigid objects when: the objects may vary in shape and complexity, and no restrictive assumptions are made about the types of surfaces on the object. We propose a new and general surface representation scheme for recognizing objects with free form (sculpted) surfaces from range data. In this scheme, an object is described concisely in terms of maximal surface patches of constant shape index. These maximal patches are mapped onto the unit sphere via their orientations, and aggregated via shape spectral functions. Properties such as surface area, curvedness and connectivity that capture local and global information are also built into the representation. The scheme yields not only a meaningful and rich surface description useful for the recoverability of the object, but also a set of powerful indexing primitives for object matching. We demonstrate the generality and the effectiveness of our scheme using real range images of complex objects. We also present results on the categorization of object views based on a novel shape spectral matching technique.>

The concept of "shape spectrum" of an object view is introduced for constructing view aspects of free-form objects. The shape spectrum captures the shape characteristics of an object by aggregating the areas of the surfaces of the object at each value of the shape index. We derive a novel view representation based on shape spectral features and propose a general and powerful technique for organizing multiple views of objects of complex shape and geometry into compact and homogeneous clusters. Our view-grouping technique obviates the need for surface segmentation and edge detection. We demonstrate with experimental results on a database of 3,200 views of 10 objects that view aspects can be determined for sculpted objects easily and effectively.

Errors in registration of multiple views of an object based on estimated transformations between views can affect surface classification. The authors derive a minimum variance estimator (MVE) for computing the transformation parameters accurately from range data of two different views of a 3D object. The results of the authors' experiments show that the solution obtained using MVE is significantly more reliable than the estimate obtained with an unweighted distance criterion for registration.

A self-organizing neural network model for locus-addressable associative memory and binary pattern recognition is presented. The net may be used for either autoassociative or heteroassociative tasks. Locus addressability is suggested as a possible mechanism for retrieval of memories without any external cues in the form of partial or corrupted exemplar patterns. The architecture, which uses competitive dynamics, embodies a parallel search scheme which updates itself adaptively as the learning progresses. A thresholding mechanism ensures the learning of new exemplars. Upon saturation of the memory capacity, the net thereafter responds to new patterns by recalling exemplars in its memory that are nearest in Hamming distance to the presented input. The stability-plasticity problem is overcome by fast learning and irreversibility of connection-weight changes. This architecture overcomes the orthogonality and linear independence constraints that limit other models