Blindspot is a 2D web-based game that tells a story of a Chinese student newly arrived at an American university who experiences microaggressions in her college life. A prologue gives background information of the protagonist, followed by three chapters in different campus settings complete with puzzle games whose mechanics emphasize the microaggression topic of the chapter. Based on player dialogue choices, strategies for dealing with microaggressions may be followed or not, leading to different puzzles, with a hopeful ending presented to all. The narrative and game mechanics foster empathy with the protagonist, providing benefits for players in both marginalized and non-marginalized groups.

Given the deluge of multimedia content that is becoming available over the Internet, it is increasingly important to be able to effectively examine and organize these large stores of information in ways that go beyond browsing or collaborative filtering. In this paper we review previous work on audio and video processing, and define the task of Topic-Oriented Multimedia Summarization (TOMS) using natural language generation: given a set of automatically extracted features from a video (such as visual concepts and ASR transcripts) a TOMS system will automatically generate a paragraph of natural language ("a recounting"), which summarizes the important information in a video belonging to a certain topic area, and provides explanations for why a video was matched and retrieved. We see this as a first step towards systems that will be able to discriminate visually similar, but semantically different videos, compare two videos and provide textual output or summarize a large number of videos at once. In this paper, we introduce our approach of solving the TOMS problem. We extract visual concept features and ASR transcription features from a given video, and develop a template-based natural language generation system to produce a textual recounting based on the extracted features. We also propose possible experimental designs for continuously evaluating and improving TOMS systems, and present results of a pilot evaluation of our initial system.

A digital video library of over 900 hours of video and 18000 stories from The HistoryMakers was used by 266 students, faculty, librarians, and life-long learners interacting with a system providing multiple search and viewing capabilities over a trial period of several months. User demographics and actions were logged with this multimedia collection, providing quantitative and qualitative metrics on system use. These transaction logs were complemented with heuristic evaluation, interviews, and contextual inquiry with representative users. Collectively, these mixed methods informed the development of the next generation web-based interface for the HistoryMakers video oral histories to improve access to and dissemination of this rich cultural resource. In particular, the feature of a synchronized text transcript in the video player for the narratives merited further investigation. Such an interface has not seen widespread use in digital video players available on the web, yet was valued highly by oral history archive viewers. A user study with 27 participants measured the utility of the HistoryMakers web interface incorporating the synchronized transcript video player for stated fact-finding and open-ended tasks. For life oral histories, an aligned text transcript is valued for both tasks, with the video rated significantly more useful for open-ended tasks over fact-finding. These results suggest a task-dependent role of modality in presentation of oral histories, with synchronized transcripts rated highly across tasks.

This paper highlights the iterative development of a dynamic web environment for exploring physics pedagogy: the Physics Teaching Web Advisory, Pathway. The formative evaluation of the system, with high school physics teachers as users, uncovered a number of shortcomings in the synthetic interview presentation of expert teachers offering advice and instructional strategies. The problems and their solutions are discussed, with implications for the design of multimedia conversational interfaces for teachers helping teachers. The paper presents strategies for dealing with video assets to help teachers find relevant material quickly in addressing the pedagogy and full range of content taught in high school physics, while also promoting additional investigation and exploration of the synthetic interviews.

An approach using many intermediate semantic concepts is proposed with the potential to bridge the semantic gap between what a color, shape, and texture-based ldquolow-levelrdquo image analysis can extract from video and what users really want to find, most likely using text descriptions of their information needs. Semantic concepts such as cars, planes, roads, people, animals, and different types of scenes (outdoor, night time, etc.) can be automatically detected in the video with reasonable accuracy. This leads us to ask how can they be used automatically and how does a user (or a retrieval system) translate the user's information need into a selection of related concepts that would help find the relevant video clips, from the large list of available concepts. We illustrate how semantic concept retrieval can be automatically exploited by mapping queries into query classes and through pseudo-relevance feedback. We also provide evidence how a semantic concept can be utilized by users in interactive retrieval, through interfaces that provide affordances of explicit concept selection and search, concept filtering, and relevance feedback. How many concepts we actually need and how accurately they need to be detected and linked through various relationships is specified in the ontology structure.

TRECVID participants have enjoyed consistent success using storyboard interfaces for shot-based retrieval, as measured by TRECVID interactive search mean average precision (MAP). However, much is lost by only looking at MAP, and especially by neglecting to bring in representatives of the target user communities to conduct such tasks. This paper reports on the use of within-subjects experiments to reduce subject variability and emphasize the examination of specific video search interface features for their effectiveness in interactive retrieval and user satisfaction. A series of experiments is surveyed to illustrate the gradual realization of getting non-experts to utilize non-textual query features through interface adjustments. Notably, the paper explores the use of the search system by government intelligence analysts, concluding that a variety of search methods are useful for news video retrieval and lead to improved satisfaction. This community, dominated by text search system expertise but still new to video and image search, performed better with and favored a system with image and concept query capabilities over an exclusive text-search system. The user study also found that sports topics mean nothing for this user community and tens of relevant shots collected into the answer set are considered enough to satisfy the information need. Lessons learned from these user interactions are reported, with recommendations on both interface improvements for video retrieval systems and enhancing the ecological validity of video retrieval interface evaluations.

A number of researchers have been building high-level semantic concept detectors such as outdoors, face, building, to help with semantic video retrieval. Our goal is to examine how many concepts would be needed, and how they should be selected and used. Simulating performance of video retrieval under different assumptions of concept detection accuracy, we find that good retrieval can be achieved even when detection accuracy is low, if sufficiently many concepts are combined. We also derive suggestions regarding the types of concepts that would be most helpful for a large concept lexicon. Since our user study finds that people cannot predict which concepts will help their query, we also suggest ways to find the best concepts to use. Ultimately, this paper concludes that "concept-based" video retrieval with fewer than 5000 concepts, detected with a minimal accuracy of 10% mean average precision is likely to provide high accuracy results in broadcast news retrieval.

The authors developed an extensible system for video exploitation that puts the user in control to better accommodate novel situations and source material. Visually dense displays of thumbnail imagery in storyboard views are used for shot-based video exploration and retrieval. The user can identify a need for a class of audiovisual detection, adeptly and fluently supply training material for that class, and iteratively evaluate and improve the resulting automatic classification produced via multiple modality active learning and SVM. By iteratively reviewing the output of the classifier and updating the positive and negative training samples with less effort than typical for relevance feedback systems, the user can play an active role in directing the classification process while still needing to truth only a very small percentage of the multimedia data set. Examples are given illustrating the iterative creation of a classifier for a concept of interest to be included in subsequent investigations, and for a concept typically deemed irrelevant to be weeded out in follow-up queries. Filtering and browsing tools making use of existing and iteratively added concepts put the user further in control of the multimedia browsing and retrieval process.

Aural and visual cues can be automatically extracted from video and used to index its contents. The paper explores the relative merits of the cues extracted from the different modalities for locating relevant shots in video, specifically reporting on the indexing and interface strategies used to retrieve information from the Video TREC 2002 and 2003 data sets, and the evaluation of the interactive search runs. For the documentary and news material in these sets, automated speech recognition produces rich textual descriptions derived from the narrative, with visual descriptions and depictions offering additional browsing functionality. Through speech and visual processing, storyboard interfaces with query-based filtering provide an effective interactive retrieval interface. Examples drawn from the Video TREC 2002 and 2003 search topics and results using these topics illustrate the utility of multiple-document storyboards and other interfaces incorporating the results of multimodal processing.

The authors developed a system in which visually dense displays of thumbnail imagery in storyboard views are used for shot-based video retrieval. The views allow for effective retrieval, as evidenced by the success achieved by expert users with the system in interactive query for NIST TRECVID 2002 and 2003. This paper demonstrates that novice users also achieve comparatively high retrieval performance with these views using the TRECVID 2003 benchmarks. Through an analysis of the user interaction logs, heuristic evaluation, and think-aloud protocol, the usability of the video information retrieval system is appraised with respect to shot-based retrieval. Design implications are presented based on these TRECVID usability evaluations regarding efficient, effective information retrieval interfaces to locate visual information from video corpora.

This paper reviews successful approaches in evaluations of video retrieval over the last three years. The task involves the search and retrieval of shots from MPEG digitized video recordings using a combination of automatic speech, image and video analysis and information retrieval technologies. The search evaluations are grouped into interactive (with a human in the loop) and noninteractive (where the human merely enters the query into the system) submissions. Most non-interactive search approaches have relied extensively on text retrieval, and only recently have image-based features contributed reliably to improved search performance. Interactive approaches have substantially outperformed all non-interactive approaches, with most systems relying heavily on the user’s ability to refine queries and reject spurious answers. We will examine both the successful automatic search approaches and the user interface techniques that have enabled high performance video retrieval.

This paper investigates the level of metadata accuracy required for image filters to be valuable to users. Access to large digital image and video collections is hampered by ambiguous and incomplete metadata attributed to imagery. Though improvements are constantly made in the automatic derivation of semantic feature concepts such as indoor, outdoor, face, and cityscape, it is unclear how good these improvements should be and under what circumstances they are effective. This paper explores the relationship between metadata accuracy and effectiveness of retrieval using an amateur photo collection, documentary video, and news video. The accuracy of the feature classification is varied from performance typical of automated classifications today to ideal performance taken from manually generated truth data. Results establish an accuracy threshold at which semantic features can be useful, and empirically quantify the collection size when filtering first shows its effectiveness.

An image-rich interface is presented, which emphasizes visual exploration of sets of images representing shots returned from a query of filter against a digital video corpus. This interface, a storyboard of keyframes for multiple video segments, maintains temporal layout, accommodates contextual cues and filtering, supports additional filtering through visual features, and provides a means of drilling down to synchronized points in the associated video. These features allow for effective information retrieval from a video collection, as evidenced by the success achieved in interactive query for the TREC 2002 Video Retrieval Track (TREC-V). This paper introduces TREC-V, discusses the design of the multi-segment storyboard interface, illustrates its use with respect to the TREC-V topics, and presents results and conclusions based on the TREC-V evaluation.

This paper introduces the video collage, a novel effective interface for browsing and interpreting video collections. The paper discusses how collages are automatically produced, illustrates their use, and evaluates their effectiveness as summaries across news stories. Collages are presentations of text and images derived from multiple video sources, which provide an interactive visualization for a set of video documents, summarizing their contents and providing a navigation aid for further exploration. The dynamic creation of collages is based on user context, e.g., an originating query, coupled with automatic processing to refine the candidate imagery. Named entity identification and common phrase extraction provides descriptive text. The dynamic manipulation of collages allows user-directed browsing and reveals additional detail. The utility of collages as summaries is examined with respect to other published news summaries.

A storyboard is a presentation scheme for abstracting information in a digital video clip based on imagery. This paper describes a series of storyboard interfaces with added transcript text features. These interfaces are used in a controlled experiment focusing on the utility of transcript text in storyboards for news video navigation. We wished to explore whether such text resulted in improvements in video navigation, and, if so, whether the amount of text and its synchronization with video imagery affected the navigation task. The text-augmented storyboards performed significantly better than storyboards with no text. Full transcript text produced benefits when presented as a block, whereas reduced contextual text descriptions produced benefits when aligned with storyboard image rows.

Scalable Vector Graphics (SVG) is a language for describing two-dimensional graphics in XML, specifically vector graphic shapes, images, and text. SVG is a new World Wide Web Consortium (W3C) Candidate Recommendation as of November 2000, and this paper describes how SVG provides an ideal framework for presenting manipulable, interactive summarizations into a multimedia information repository. Specifically, we present VIBE and map SVG interfaces into a digital news video library for delivery through web browsers. Pan-and-zoom visualizations of video through SVG are discussed.

The Informedia Digital Video Library contains over 2000 hours of video, growing at a rate of 15 hours per week. A good query engine is not sufficient for information retrieval because often the candidate result sets grow in number as the library grows. Video digests summarize sets of stories from the library, providing users with a visual mechanism for interactive browsing and query refinement. These digests are generated dynamically under the direction of the user based on automatically derived metadata from the video library. Three types of digests are discussed: VIBE digests emphasizing word relationships, timelines showing trends against time, and maps showing geographic correlations. Multiple digests can be combined into a single view or animated into a temporal presentation.

This paper reports two studies that measured the effects of ,different "video skim" techniques on comprehension, navigation, and user satisfaction.Video skims are compact, ,content-rich abstractions of longer videos, condensations that preserve frame rate while greatly reducing viewing time.Their characteristics depend on the image-and audioprocessing techniques used to create them.Results from the initial study helped refine video skims, which were then reassessed in the second experiment.Significant benefits were found for skims built from audio sequences meeting certain criteria.Keyworck Video abstraction, evaluation, digital video library, video browsing, video skim, empirical studies, multimedia ?NTRQDUCTiQN With increasing computational power and storage capacity, the potential for large digital video libraries is growing rapidly-The World Wide Web has seen an increased use of digital video, and digital video remains a key component of many educational and entertainment applications.As the size of accessible video collections grows to thousands of hours, potential viewers will need abstractions and technology to help them browse effectively and efficiently through this new wealth of information.A multimedia abstraction ideally preserves and communicates in a compact representation the essential content of a source video.Examples include brief titles and individual "thumbnail" images that, when selected appropriately, facilitate operations on the corresponding source 131.Another common approach presents an ordered set of representative thumbnail images simultaneously on a computer screen [3,5, 6,9,11,15,17, IS].Wbile these abstractions have proven useful in various contexts, their static nature ignores video's temporal dimension.In addition, they often concentrate exclusively on the image content and neglect de audio information carried in a video Pemkion to make digitaLbard copies ofall or pro% of this material for personal or classroom use is granted without fee provided that the copies are not made or diswiiuted for proM or commercial advantage, the copyright notice, the title of tile publication and ils date appear, and notice is given that copyright is by pcmksion of the ACh4.

No abstract available.

New digital video technologies provide a wide spectrum of multimedia interface capabilities for educational courses running on personal computers. A formal experiment was conducted using a digital video course on code inspection to determine the effects of such capabilities on recall performance and attitude. The findings suggest that the presentation of material as motion video rather than as a slide show within an interactive video course leads to better recall performance. In addition, the presence of motion video in the interfaces and the use of surrogate travel for navigation promote better student opinions toward the subject matter.