Visual search connects physical (offline) objects with (online) digital media. Using objects from the environment, like newspapers, magazines, books and posters, we can retrieve supplemental information from the online world. In this demonstration, we show a framework for delivering visual search services to users of mobile devices. We show how users can point a mobile device at any location in a document, magazine or book to view related, online material on the device. We describe client applications now being deployed for the iPhone and the server architecture used for recognition of scanned images.

Mobile applications are increasingly becoming part of our lives and our work. Moreover, visual capture capability on mobile devices is increasingly being used for linking the real world and the digital world. In this demonstration paper we present an application for accessing and browsing Web sites by pointing a mobile phone at a text book or a printout of a Web site, taking a picture of a small region, performing content recognition and automatically displaying a Web site that is linked to the content.

HotPaper enables users to electronically interact with paper documents by pointing their camera phones at a document. Electronic interaction is in the form of reading and/or writing images, audio and video clips, urls, notes, etc to the paper. HotPaper does not require any machine readable code, such as a barcode or watermark, to be present on the document. It utilizes our Brick Wall Coding document recognition algorithm [1] that uses a small document patch image to retrieve electronic versions of documents without performing OCR. The HotPaper demonstration runs on a Treo 700w 312 MHz mobile phone and performs document recognition at 4 frames per second using 176x144 video input.

In this paper, we describe a method for storing an entire video, animation sequence, or any other media type, on paper. The method is based on printing a key frame from a video on paper along with a barcode that encodes the motion information and other auxiliary information in MPEG-4 format. Unlike other video barcode systems in the prior art, a barcode in our system does not contain a link to the video clip; instead it contains motion information. A client device applies the motion information to an image of the video key frame to obtain full motion video. Therefore, the paper document is a self contained representation of a video clip and access to a server is not required. We modified an MPEG-4 [1] encoder and decoder to implement the video flip-book encoder and decoder. Experiments show that it is possible to encode several seconds of video on paper using our method. This is sufficient to create small animations for some printed materials such as video greeting cards and children's books.

A multimedia note taking system called MinuteAid is described that lets users of laptops include multimedia objects in notes they write during a presentation. The multimedia data is provided by devices that capture slide images, whiteboard images, audio and video clips. The MinuteAid system includes a software interface between these devices and a commonly available word processing application (Microsoft Word) and lets the user either insert copies of multimedia objects while they're captured or references to multimedia objects that are automatically retrieved and inserted in the document later. We describe the architecture, implementation and user experience with this system.

Our contextual inquiry into the practices of oral historians unearthed a curious incongruity. While oral historians consider interview recordings a central historical artifact, these recordings sit unused after a written transcript is produced. We hypothesized that this is largely because books are more usable than recordings. Therefore, we created Books with Voices: bar-code augmented paper transcripts enabling fast, random access to digital video interviews on a PDA. We present quantitative results of an evaluation of this tangible interface with 13 participants. They found this lightweight, structured access to original recordings to offer substantial benefits with minimal overhead. Oral historians found a level of emotion in the video not available in the printed transcript. The video also helped readers clarify the text and observe nonverbal cues.

The components of a key frame selection algorithm for a paper-based multimedia browsing interface called Video Paper are described. Analysis of video image frames is combined with the results of processing the closed caption to select key frames that are printed on a paper document together with the closed caption. Bar codes positioned near the key frames allow a user to play the video from the corresponding times. This paper describes several component techniques that are being investigated for key frame selection in the Video Paper system, including face detection and text recognition. The Video Paper system implementation is also discussed.

A paper-based interface for browsing video is proposed. A paper document shows key frames selected from a video, a transcript for the parallel audio track, and bar codes that, when scanned, invoke a multimedia player. The paper document provides a stand-alone representation for a video recording that lets a user both understand the content of the file and replay only selected parts of the multimedia that are necessary to gain a better understanding. This approach applies the two-dimensional display characteristics of a newspaper to multimedia retrieval. By so doing, the user's browsing and search efficiency is greatly improved. This poster describes an implementation of the video paper system using a pocket PC with a bar code reader as the remote control device and an archive of video recordings on the pocket PC or an external server.

Video Paper is a prototype system for multimedia browsing, analysis, and replay. Key frames extracted from a video recording are printed on paper together with bar codes that allow for random access and replay. A transcript for the audio track can also be shown so that users can read what was said, thus making the document a stand-alone representation for the contents of the multimedia recording. The Video Paper system has been used for several applications, including the analysis of recorded meetings, broadcast news, oral histories and personal recordings. This demonstration will show how the Video Paper system was applied to these domains and the various replay systems that were developed, including a self-contained portable implementation on a PDA and a fixed implementation on desktop PC.

The design and implementation of a portable meeting recorder is presented. Composed of an omni-directional video camera with four-channel audio capture, the system saves a view of all the activity in a meeting and the directions from which people spoke. Subsequent analysis computes metadata that includes video activity analysis of the compressed data stream and audio processing that helps locate events that occurred during the meeting. Automatic calculation of the room in which the meeting occurred allows for efficient navigation of a collection of recorded meetings. A user interface is populated from the metadata description to allow for simple browsing and location of significant events.

Over the last two centuries, reading styles have shifted away from the reading of documents from beginning to end and toward the skimming of documents in search of relevant information. This trend continues today where readers, often confronted with an insurmountable amount of text, seek more efficient methods of extracting relevant information from documents. In this paper, a new document reading environment is introduced called the Readers HelperTM, which supports the reading of electronic and paper documents. The Readers Helper analyzes documents and produces a relevance score for each of the readers topics of interest, thereby helping the reader decide whether the document is actually worth skimming or reading. Moreover, during the analysis process, topic of interest phrases are automatically annotated to help the reader quickly locate relevant information. A new information visualization tool, called the ThumbarTM, is used in conjunction with relevancy scoring and automatic annotation to portray a continuous, dynamic thumb-nail representation of the document. This further supports rapid navigation of the text.