Demonstration venue · read-only. Every page can be browsed; the buttons that would change it are switched off. Create an account to run TaxoReview on your own data. A simulation from public publication records: the people listed were not invited and did not agree to review.
? Tutorial for this page

Jonathan Foote

dblp:95/5723 · also J. T. Foote · DBLP profile ↗
← Back
54ranked-venue papers
16as first author
0since 2021 · last 2014
0000-0003-4411-1362ORCID · corroborated

Domains — the database's venue-derived domains; a paper can count in several

Graphics, computer vision, multimedia, augmented reality and games · 46 · 15 first-authorHuman-computer interaction and ubiquitous computing · 5Artificial intelligence and machine learning · 2 · 2 first-authorDatabases, data management, data science and information retrieval · 2Computer networks · 1Security and privacy · 1

Expertise — from the expertise taxonomy: the topics of the expert's papers under the CCF categories. A weight counts papers with recency: 1 for a paper about the topic, 0.3 when the topic is its context, halved every five years.

Computer graphics and multimedia
17 papers
Virtual and augmented reality · 47% Multimedia analysis and retrieval · 19% Image and video processing · 10%
Network and information security
1 paper
Systems and software security · 100%
Software engineering, system software, and programming languages
1 paper
Software testing · 100%
Human-computer interaction and pervasive computing
7 papers
Collaborative and social computing · 59% Interaction techniques and input · 19% Ubiquitous computing and smart environments · 13%

Topics — the 30 heaviest of 46, each with the papers that count most for it

TopicWeightPapersLastEvidence papers
Systems and software security › vulnerability discovery
fuzzing
0.212014
Optimizing Seed Selection for Fuzzing · USENIX Security Symposium 2014
Systems and software security
vulnerability discovery
0.212014
Optimizing Seed Selection for Fuzzing · USENIX Security Symposium 2014
Software testing
fuzzing
0.212014
Optimizing Seed Selection for Fuzzing · USENIX Security Symposium 2014
Software testing › fuzzing › seed scheduling
seed selection
0.212014
Optimizing Seed Selection for Fuzzing · USENIX Security Symposium 2014
Virtual and augmented reality › immersive video
360-degree video
0.142005
Region of interest extraction and virtual camera control based on panoramic video capturing · IEEE Trans. Multim. 2005
Panoramic video capturing and compressed domain virtual camera control · ACM Multimedia 2001
FlyAbout: spatially indexed panoramic video · ACM Multimedia 2001
Collaborative and social computing
remote collaboration
0.132010
Remote interactive graffiti · ACM Multimedia 2004
The virtual factory: Exploring 3D worlds as industrial collaboration and control environments · VR 2010
The virtual chocolate factory: mixed reality industrial collaboration and control · ACM Multimedia 2010
Virtual and augmented reality
mixed reality
0.112010
The virtual factory: Exploring 3D worlds as industrial collaboration and control environments · VR 2010
Virtual and augmented reality › immersive interaction › 3d interaction
virtual camera control
0.022005
Panoramic video capturing and compressed domain virtual camera control · ACM Multimedia 2001
Region of interest extraction and virtual camera control based on panoramic video capturing · IEEE Trans. Multim. 2005
Multimedia analysis and retrieval › multimedia analysis › multimedia collection analysis
photo collection organization
0.012003
Managing digital memories with the FXPAL photo application · ACM Multimedia 2003
Multimedia analysis and retrieval › multimedia analysis
visual content analysis
0.012003
Temporal event clustering for digital photo collections · ACM Multimedia 2003
Interaction techniques and input
gesture input
0.012003
Shared interactive video for teleconferencing · ACM Multimedia 2003
Collaborative and social computing › computer-mediated communication
teleconferencing
0.012003
Shared interactive video for teleconferencing · ACM Multimedia 2003
Image and video processing › motion analysis
camera motion analysis
0.022002
A semi-automatic approach to home video editing · UIST 2000
Creating music videos using automatic media analysis · ACM Multimedia 2002
Audio and music processing
audio analysis
0.012002
Creating music videos using automatic media analysis · ACM Multimedia 2002
Visual content generation and editing › video editing
automatic video editing
0.012002
Creating music videos using automatic media analysis · ACM Multimedia 2002
Visual content generation and editing › video generation
music video generation
0.012002
Creating music videos using automatic media analysis · ACM Multimedia 2002
Information retrieval
content-based retrieval
0.021996
Open-Vocabulary Speech Indexing for Voice and Video Mail Retrieval · ACM Multimedia 1996
Automatic Content-Based Retrieval of broadcast News · ACM Multimedia 1995
Multimedia analysis and retrieval
video analysis
0.012000
A semi-automatic approach to home video editing · UIST 2000
Visual content generation and editing
video editing
0.012000
A semi-automatic approach to home video editing · UIST 2000
Audio and music processing
music information retrieval
0.011999
Visualizing music and audio using self-similarity · ACM Multimedia (1) 1999
Audio and music processing
music structure analysis
0.011999
Visualizing music and audio using self-similarity · ACM Multimedia (1) 1999
Multimedia analysis and retrieval
video summarization
0.011999
Video Manga: generating semantically meaningful video summaries · ACM Multimedia (1) 1999
Multimedia analysis and retrieval
multimedia browsing
0.011998
An Intelligent Media Browser Using Automatic Multimodal Analysis · ACM Multimedia 1998
Information retrieval › document retrieval
spoken document retrieval
0.011996
Retrieving Spoken Documents by Combining Multiple Index Sources · SIGIR 1996
Audio and music processing › audio analysis › audio content analysis
audio indexing
0.011996
Open-Vocabulary Speech Indexing for Voice and Video Mail Retrieval · ACM Multimedia 1996
Computational photography and imaging
projector-camera systems
0.012004
Remote interactive graffiti · ACM Multimedia 2004
Multimedia analysis and retrieval › video retrieval
news video retrieval
0.011995
Automatic Content-Based Retrieval of broadcast News · ACM Multimedia 1995
Transport protocols and congestion control › real-time communication
video conferencing
0.012003
Shared interactive video for teleconferencing · ACM Multimedia 2003
Image and video processing
video segmentation
0.012002
Creating music videos using automatic media analysis · ACM Multimedia 2002
Human-robot interaction
remote control
0.012002
FlySPEC: a multi-user video camera system with hybrid human and automatic control · ACM Multimedia 2002

Methods — techniques the papers use, named apart from their topics

optimization · 0.4virtual world mirroring · 0.2mobile and web-based display · 0.2data integration · 0.23d modeling · 0.2image alignment · 0.1camera distortion correction · 0.1kalman filter · 0.1video window interaction · 0.1gesture-based interaction · 0.1automated organization · 0.1motion analysis · 0.1compressed-domain analysis · 0.1pan/tilt/zoom camera control · 0.0pan-tilt-zoom camera control · 0.0statistical text retrieval · 0.0word spotting · 0.0continuous speech recognition · 0.0
YearPublicationVenuePosition
2014 Optimizing Seed Selection for Fuzzing
Alexandre Rebert, Sang Kil Cha, Thanassis Avgerinos, Jonathan Foote, David Warren, Gustavo Grieco, David Brumley
USENIX Security Symposium4
2010 The virtual chocolate factory: Building a real world mixed-reality system for industrial collaboration and control
abstract
Virtual, mobile, and mixed reality systems have diverse uses for data visualization and remote collaboration in industrial settings, especially factories. We report our experiences in designing complex mixed-reality collaboration, control, and display systems for a real-world factory, for delivering real-time factory information to multiple types of users. In collaboration with TCHO, a chocolate maker in San Francisco, our research group is building a virtual "mirror" world of a real-world chocolate factory and its processes. Sensor data is imported into the multi-user 3D environment from hundreds of sensors on the factory floor. The resulting "virtual factory" is designed for simulation, visualization, and collaboration, using a set of interlinked, real-time layers of information about the factory and its processes. We are also looking at appropriate industrial uses for mobile devices such as cell phones and tablet computers, and how they intersect with virtual worlds and mixed realities. For example, an experimental iPhone web app provides mobile laboratory monitoring and control. The mobile system is integrated with the database underlying the virtual factory world. These systems were deployed at the real-world factory and lab in 2009, and are now in beta development. Through this mashup of mobile, social, mixed and virtual technologies, we hope to create industrial systems for enhanced collaboration between physically remote people and places - for example, factories in China with managers in Japan or the US.
Maribeth Back, Don Kimber, Eleanor Gilbert Rieffel, Anthony Dunnigan, Bee Liew, Sagar Gattepally, Jonathan Foote, Jun Shingu, Jim Vaughan
ICME7
2010 The virtual chocolate factory: mixed reality industrial collaboration and control
abstract
We show several aspects of a complex mixed reality system that we have built and deployed in a real-world factory setting. In our system, virtual worlds, augmented realities, and social and mobile applications are all fed from the same infrastructure. In collaboration with TCHO[1], a chocolate maker in San Francisco, we built a virtual "mirror" world of a real-world chocolate factory and its processes. Sensor data is imported into the multi-user 3D environment from hundreds of sensors on the factory floor. The resulting virtual factory is used for simulation, visualization, and collaboration, using a set of interlinked, real-time layers of information. Another part of our infrastructure is designed to support appropriate industrial uses for mobile devices such as cell phones and tablet computers. We deployed this system at the real-world factory in 2009, and it is now is daily use there. By simultaneously developing mobile, virtual, and web-based display and collaboration environments, we aimed to create an infrastructure that did not skew toward one type of application but that could serve many at once, interchangeably. Through this mixture of mobile, social, mixed and virtual technologies, we hope to create systems for enhanced collaboration in industrial settings between physically remote people and places, such as factories in China with managers in the US.
Maribeth Back, Don Kimber, Eleanor Gilbert Rieffel, Anthony Dunnigan, Bee Liew, Sagar Gattepally, Jonathan Foote, Jun Shingu, Jim Vaughan
ACM Multimedia7
2010 The virtual factory: Exploring 3D worlds as industrial collaboration and control environments
abstract
This project investigates practical uses of virtual, mobile, and mixed reality systems in industrial settings, in particular control and collaboration applications for factories. In collaboration with TCHO, a chocolate maker start-up in San Francisco, we have built virtual mirror-world representations of a real-world chocolate factory and are importing its data and modeling its processes. The system integrates mobile devices such as cell phones and tablet computers. The resulting "virtual factory" is a cross-reality environment designed for simulation, visualization, and collaboration, using a set of interlinked, real-time 3D and 2D layers of information about the factory and its processes.
Maribeth Back, Timothy Childs, Anthony Dunnigan, Jonathan Foote, Sagar Gattepally, Bee Liew, Jun Shingu, Jim Vaughan
VR4
2005 Discriminative techniques for keyframe selection
abstract
A convenient representation of a video segment is a single "keyframe". Keyframes are widely used in applications such as non-linear browsing and video editing. With existing methods of keyframe selection, similar video segments result in very similar keyframes, with the drawback that actual differences between the segments may be obscured. We present methods for keyframe selection based on two criteria: capturing the similarity to the represented segment, and preserving the differences from other segment keyframes, so that different segments will have visually distinct representations. We present two discriminative keyframe selection methods, and an example of experimental results.
Matthew Cooper 0002, Jonathan Foote
ICME2
2005 Mediated Meeting Interaction for Teleconferencing
abstract
A common problem with teleconferences is awkward turn-taking – particularly ‘ collisions,’ whereby multiple parties inadvertently speak over each other due to communication delays. We propose a model for teleconference discussions including the effects of delays, and describe tools that can improve the quality of those interactions. We describe an interface to gently provide latency awareness, and to give advanced notice of ‘ incoming speech’ to help participants avoid collisions. This is possible when codec latencies are significant, or when a low bandwidth side channel or out-of-band signaling is available with lower latency than the primary video channel. We report on results of simulations, and of experiments carried out with transpacific meetings, that demonstrate these tools can improve the quality of teleconference discussions.
Kazumasa Murai, Don Kimber, Jonathan Foote, Qiong Liu 0003, John Doherty
ICME3
2005 Region of interest extraction and virtual camera control based on panoramic video capturing
abstract
We present a system for automatically extracting the region of interest (ROI) and controlling virtual cameras' control based on panoramic video. It targets applications such as classroom lectures and video conferencing. For capturing panoramic video, we use the FlyCam system that produces high resolution, wide-angle video by stitching video images from multiple stationary cameras. To generate conventional video, a region of interest can be cropped from the panoramic video. We propose methods for ROI detection, tracking, and virtual camera control that work in both the uncompressed and compressed domains. The ROI is located from motion and color information in the uncompressed domain and macroblock information in the compressed domain, and tracked using a Kalman filter. This results in virtual camera control that simulates human controlled video recording. The system has no physical camera motion and the virtual camera parameters are readily available for video indexing.
Xinding Sun, Jonathan Foote, Don Kimber, B. S. Manjunath
IEEE Trans. Multim.2
2005 Temporal event clustering for digital photo collections
abstract
Organizing digital photograph collections according to events such as holiday gatherings or vacations is a common practice among photographers. To support photographers in this task, we present similarity-based methods to cluster digital photos by time and image content. The approach is general and unsupervised, and makes minimal assumptions regarding the structure or statistics of the photo collection. We present several variants of an automatic unsupervised algorithm to partition a collection of digital photographs based either on temporal similarity alone, or on temporal and content-based similarity. First, interphoto similarity is quantified at multiple temporal scales to identify likely event clusters. Second, the final clusters are determined according to one of three clustering goodness criteria. The clustering criteria trade off computational complexity and performance. We also describe a supervised clustering method based on learning vector quantization. Finally, we review the results of an experimental evaluation of the proposed algorithms and existing approaches on two test collections.
Matthew Cooper 0002, Jonathan Foote, Andreas Girgensohn, Lynn Wilcox
ACM Trans. Multim. Comput. Commun. Appl.2
2004 Remote interactive graffiti
abstract
We present an installation that allows distributed internet participants to "draw" on a public scene using light. The iLight system is a camera/projector system designed for remote collaboration. Using a familiar digital drawing interface, remote users "draw" on a live video image of a real-life object or scene. Graics drawn by the user are then projected onto the scene, where they are visible in the camera image. Because camera distortions are corrected and the video is aligned with the image canvas, drawn graics appear exactly where desired. Thus the remote users may harmlessly mark a ysical object to serve their own their artistic and/or expressive needs. We also describe how local participants may interact with remote users through the projected images. Besides the intrinsic "neat factor" of action at a distance, this installation serves as an experiment in how multiple users from different locales and cultures can create a social space that interacts with a ysical one, as well as raising issues of free expression in a non-destructive context.
Jonathan Foote, Don Kimber
ACM Multimedia1
2003 Automatically organizing digital photographs using time and content
abstract
We present similarity-based methods to cluster digital photos by time and image content. This approach is general, unsupervised, and makes minimal assumptions regarding the structure or statistics of the photo collection. We describe versions of the algorithm using temporal similarity with and without content-based similarity, and compare the algorithms with existing techniques, measured against ground-truth clusters created by humans.
Matthew Cooper 0002, Jonathan Foote, Andreas Girgensohn
ICIP (3)2
2003 Time base modulation: a new approach to watermarking audio
abstract
A novel method is presented for inaudibly hiding information in an audio signal by subtly applying time-scale modification to segments of the signal. The sequence, duration, and degree of the time-scale modifications are the parameters which encode information in the altered signal. By comparing the altered signal with a reference copy, compressed and expanded regions can be identified and the hidden data recovered. This approach is novel and has several advantages over other methods: it is theoretically noiseless, it introduces no spectral distortion, and it is robust to all known methods of reproduction, compression, and transmission.
Jonathan Foote, John Adcock, Andreas Girgensohn
ICME1
2003 Multichannel video/audio acquisition for immersive conferencing
abstract
This paper presents an information-driven audiovisual signal acquisition approach. This approach has several advantages: users are encouraged to assist in signal acquisition; available sensors are managed based on both signal characteristics and users' suggestions. The problem formulation is consistent with many well-known empirical approaches widely used in previous systems and may provide analytical explanations to these approaches. We demonstrate the use of this approach to pan/tilt/zoom (PTZ) camera management with field data.
Qiong Liu 0003, Don Kimber, Jonathan Foote, Chunyuan Liao
ICME3
2003 Manipulating and Annotating Slides in a Multi-Display Environment
Patrick Chiu, Qiong Liu 0003, John S. Boreczky, Jonathan Foote, Don Kimber, Surapong Lertsithichai, Chunyuan Liao
INTERACT4
2003 Simplifying the Management of Large Photo Collections
Andreas Girgensohn, John Adcock, Matthew Cooper 0002, Jonathan Foote, Lynn Wilcox
INTERACT4
2003 Managing digital memories with the FXPAL photo application
abstract
The FXPAL Photo Application is designed to faciliate the organization of digital images from digital cameras and other sources through automated organization and intuitive user interfaces.
John Adcock, Matthew Cooper 0002, John Doherty, Jonathan Foote, Andreas Girgensohn, Lynn Wilcox
ACM Multimedia4
2003 Temporal event clustering for digital photo collections
abstract
We present similarity-based methods to cluster digital photos by time and image content. The approach is general, unsupervised, and makes minimal assumptions regarding the structure or statistics of the photo collection. We present results for the algorithm based solely on temporal similarity, and jointly on temporal and content-based similarity. We also describe a supervised algorithm based on learning vector quantization. Finally, we include experimental results for the proposed algorithms and several competing approaches on two test collections.
Matthew Cooper 0002, Jonathan Foote, Andreas Girgensohn, Lynn Wilcox
ACM Multimedia2
2003 Shared interactive video for teleconferencing
abstract
We present a system that allows remote and local participants to control devices in a meeting environment using mouse or pen based gestures "through" video windows. Unlike state-of-the-art device control interfaces that require interaction with text commands, buttons, or other artificial symbols, our approach allows users to interact with devices through live video of the environment. This naturally extends our video supported pan/tilt/zoom (PTZ) camera control system, by allowing gestures in video windows to control not only PTZ cameras, but also other devices visible in video images. For example, an authorized meeting participant can show a presentation on a screen by dragging the file on a personal laptop and dropping it on the video image of the presentation screen. This paper presents the system architecture, implementation tradeoffs, and various meeting control scenarios.
Chunyuan Liao, Qiong Liu 0003, Don Kimber, Patrick Chiu, Jonathan Foote, Lynn Wilcox
ACM Multimedia5
2002 Enhanced video browsing using automatically extracted audio excerpts
abstract
We present a method for rapidly and robustly extracting audio excerpts without the overhead of speech recognition or speaker segmentation. An immediate application is to automatically augment keyframe-based video summaries with informative audio excerpts associated with the video segments represented by the keyframes. Short audio clips combined with keyframes comprise an extremely lightweight and Web-browsable interface for auditioning video or similar media, without using bandwidth-intensive streaming video or audio.
Jonathan Foote, Matthew Cooper 0002, Lynn Wilcox
ICME (2)1
2002 Managing a camera system to serve different video requests
abstract
This paper presents a camera system called FlySPEC. In contrast to a traditional camera system that provides the same video stream to every user, FlySPEC can simultaneously serve different video-viewing requests. This flexibility allows users to conveniently participate in a seminar or meeting at their own pace. Meanwhile, the FlySPEC system provides a seamless blend of manual control and automation. With this control mix, users can easily make tradeoffs between video capture effort and video quality. The FlySPEC camera is constructed by installing a set of pan/tilt/zoom (PTZ) cameras near a high-resolution panoramic camera. While the panoramic camera provides the basic functionality of serving different viewing requests, the PTZ camera is managed by our algorithm to improve the overall video quality that may affect users watching details. The video resolution improvements from using different camera management strategies are compared in the experimental section.
Qiong Liu 0003, Don Kimber, Lynn Wilcox, Matthew Cooper 0002, Jonathan Foote, John S. Boreczky
ICME (2)5
2002 Detecting path intersections in panoramic video
abstract
Given panoramic video taken along a self-intersecting path, we present a method for detecting the intersection points. This allows "virtual tours" to be synthesized by splicing the panoramic video at the intersection points. Spatial intersections are detected by finding the best-matching panoramic images from a number of nearby candidates. Each panoramic image is segmented into horizontal strips. Each strip is averaged in the vertical direction. The Fourier coefficients of the resulting 1-D data capture the rotation-invariant horizontal texture of each panoramic image. The distance between two panoramic images is calculated as the sum of the distances between their strip texture pairs at the same row positions. The intersection is chosen as the two candidate panoramic images that have the minimum distance.
Xinding Sun, Don Kimber, Jonathan Foote, B. S. Manjunath
ICME (2)3
2002 Creating music videos using automatic media analysis
abstract
We present methods for automatic and semi-automatic creation of music videos, given an arbitrary audio soundtrack and source video. Significant audio changes are automatically detected; similarly, the source video is automatically segmented and analyzed for suitability based on camera motion and exposure. Video with excessive camera motion or poor contrast is penalized with a high unsuitability score, and is more likely to be discarded in the final edit. High quality video clips are then automatically selected and aligned in time with significant audio changes. Video clips are adjusted to match the audio segments by selecting the most suitable region of the desired length. Besides a fully automated solution, our system can also start with clips manually selected and ordered using a graphical interface. The video is then created by truncating the selected clips (preserving the high quality portions) to produce a video digest that is synchronized with the soundtrack music, thus enhancing the impact of both.
Jonathan Foote, Matthew Cooper 0002, Andreas Girgensohn
ACM Multimedia1
2002 FlySPEC: a multi-user video camera system with hybrid human and automatic control
abstract
FlySPEC is a video camera system designed for real-time remote operation. A hybrid design combines the high resolution of an optomechanical video camera with the wide field of view always available from a panoramic camera. The control system integrates requests from multiple users so that each controls a virtual camera. The control system seamlessly integrates manual and fully automatic control. It supports a range of options from untended automatic to full manual control. The system can also learn control strategies from user requests. Additionally, the panoramic view is always available for an intuitive interface, and objects are never out of view regardless of the zoom factor. We present the system architecture, an information-theoretic approach to combining panoramic and zoomed images to optimally satisfy user requests, and experimental results that show the FlySPEC system significantly assists users in a remote inspection tasks.
Qiong Liu 0003, Don Kimber, Jonathan Foote, Lynn Wilcox, John S. Boreczky
ACM Multimedia3
2001 Scene boundary detection via video self-similarity analysis
abstract
We present a novel framework for analyzing video using self-similarity. Video scenes are located by analyzing inter-frame similarity matrices. The approach is flexible to the choice of similarity measure and is robust and data-independent because the data is used to model itself. We present the approach and its application to scene boundary detection. This is shown to dramatically outperform a conventional scene-boundary detector that uses a histogram-based measure of frame difference.
Matthew Cooper 0002, Jonathan Foote
ICIP (3)2
2001 Recording the region of interest from FlyCam panoramic video
abstract
A novel method for region of interest tracking and recording video is presented. The proposed method is based on the FlyCam system (Foote et al., 2000), which produces high-resolution and wide-angle video sequences by stitching the video frames from multiple stationary cameras. The method integrates tracking and recording processes, and targets applications such as classroom lectures and videoconferencing. First, the region of interest (which typically covers the speaker) is tracked using a Kalman filter. Then, the Kalman filter estimation results are used for virtual camera control and to record the video. The system has no physical camera motion and the virtual camera parameters are readily available for video indexing. The proposed system has been implemented for real-time recording of lectures and presentations.
Xinding Sun, Jonathan Foote, Don Kimber, B. S. Manjunath
ICIP (1)2
2001 The Beat Spectrum: A New Approach To Rhythm Analysis
abstract
We introduce the beat spectrum, a new method of automatically characterizing the rhythm and tempo of music and audio. The beat spectrum is a measure of acoustic self-similarity as a function of time lag. Highly structured or repetitive music will have strong beat spectrum peaks at the repetition times. This reveals both tempo and the relative strength of particular beats, and therefore can distinguish between different kinds of rhythms at the same tempo. We also introduce the beat spectrogram which graphically illustrates rhythm variation over time. Unlike previous approaches to tempo analysis, the beat spectrum does not depend on particular attributes such as energy or frequency, and thus will work for any music or audio in any genre. We present tempo estimation results which are accurate to within 1 % for a variety of musical genres. This approach has a variety of applications, including music retrieval by similarity and automatically generating music videos. 1.
Jonathan Foote, Shingo Uchihashi
ICME1
2001 FlyAbout: spatially indexed panoramic video
abstract
We describe a system called FlyAbout which uses spatially indexed panoramic video for virtual reality applications. Panoramic video is captured by moving a [email protected] camera along continuous paths. Users can interactively replay the video with the ability to view any interesting object or choose a particular direction. Spatially indexed video gives the ability to travel along paths or roads with a map-like interface. At junctions, or intersection points, users can chose which path to follow as well as which direction to look, allowing interaction not available with conventional video. Combining the spatial index with a spatial databsdde of maps or objects allows users to navigate to specific locations or interactively inspect particular objects.
Don Kimber, Jonathan Foote, Surapong Lertsithichai
ACM Multimedia2
2001 Panoramic video capturing and compressed domain virtual camera control
abstract
A system for capturing panoramic video and a novel method for corresponding compressed domain virtual camera control is presented. It targets applications such as classroom lectures and video conferencing. The proposed method is based on the FlyCam panoramic video system that is designed to produce high resolution and wide-angle video sequences by stitching the video pictures from multiple stationary cameras. The panoramic video sequence is compressed into an MPEG-2 stream for delivery. The proposed method integrates region of Interest (ROI) detection, tracking, and virtual camera control, and works on compressed domain information only. It first detects the ROI in the P (predictive coded) picture using only the macroblock type information, It then up-samples this detection result to obtain the ROI of the whole video stream. The ROI is tracked using a Kalman filter. The Kalman filter estimation results are used for virtual camera control that simulates human controlled video recording. The system has no physical camera motion and the virtual camera parameters are readily available for video indexing. The proposed system has been implemented for real time processing.
Xinding Sun, Jonathan Foote, Don Kimber, B. S. Manjunath
ACM Multimedia2
2000 FlyCam: practical panoramic video
Jonathan Foote, Don Kimber
ACM Multimedia1
2000 A semi-automatic approach to home video editing
abstract
Hitchcock is a system that allows users to easily create custom videos from raw video shot with a standard video camera.In contrast to other video editing systems, Hitchcock uses automatic analysis to determine the suitability of portions of the raw video.Unsuitable video typically has fast or erratic camera motion.Hitchcock first analyzes video to identify the type and amount of camera motion: fast pan, slow zoom, etc.Based on this analysis, a numerical "unsuitability" score is computed for each frame of the video.Combined with standard editing rules, this score is used to identify clips for inclusion in the final video and to select their start and end points.To create a custom video, the user drags keyframes corresponding to the desired clips into a storyboard.Users can lengthen or shorten the clip without specifying the start and end frames explicitly.Clip lengths are balanced automatically using a spring-based algorithm.
Andreas Girgensohn, John S. Boreczky, Patrick Chiu, John Doherty, Jonathan Foote, Gene Golovchinsky, Shingo Uchihashi, Lynn Wilcox
UIST5
1999 Finding presentations in recorded meetings using audio and video features
abstract
This paper describes a method for finding segments in video-recorded meetings that correspond to presentations. These segments serve as indexes into the recorded meeting. The system automatically detects intervals of video that correspond to presentation slides. We assume that only one person speaks during an interval when slides are detected. Thus these intervals can be used as training data for a speaker spotting system. An HMM is automatically constructed and trained on the audio data from each slide interval. A Viterbi alignment then resegments the audio according to speaker. Since the same speaker may talk across multiple slide intervals the acoustic data from these intervals is clustered to yield an estimate of the number of distinct speakers and their order. This allows the individual presentations in the video to be identified from the location of each presenter's speech. Results are presented for a corpus of six meeting videos.
Jonathan Foote, John S. Boreczky, Lynn Wilcox
ICASSP1
1999 Video classification using transform coefficients
abstract
This paper describes techniques for classifying video frames using statistical models of reduced DCT or Hadamard transform coefficients. When decimated in time and reduced using truncation or principal component analysis, transform coefficients taken across an entire frame image allow rapid modeling, segmentation and similarity calculation. Unlike color-histogram metrics, this approach models image composition and works on grayscale images. Modeling the statistics of the transformed video frame images gives a likelihood measure that allows video to be segmented, classified, and ranked by similarity for retrieval. Experiments are presented that show an 87% correct classification rate for different classes. Applications are presented including a content-aware video browser.
Andreas Girgensohn, Jonathan Foote
ICASSP2
1999 Summarizing video using a shot importance measure and a frame-packing algorithm
abstract
This paper presents methods of generating compact pictorial summarizations of video. By calculating a measure of shot importance video can be summarized by de-emphasizing or discarding less important information, such as repeated or common scenes. In contrast to other approaches that present keyframes for each shot, this measure allows summarization by presenting only the most important shots. Selected keyframes can also be resized depending on their relative importance. We present an efficient packing algorithm that constructs a pictorial representation from differently-sized keyframes. This results in a compact and visually pleasing summary reminiscent of a comic book.
Shingo Uchihashi, Jonathan Foote
ICASSP2
1999 Facilitating Video Access by Visualizing Automatic Analysis
Andreas Girgensohn, John S. Boreczky, Lynn Wilcox, Jonathan Foote
INTERACT4
1999 Multimedia access and retrieval: the state of the art and future directions (panel session)
abstract
Several years have passed since the research topic of content based multimedia retrieval emerged.We have witnessed the burgeoning research activities into a plenitude of new indexing, retrieval, and filtering tools for images, video, audio, music, graphics, and their combinations with text-based information.Exciting research opportunities arise when integrating knowledge from multiple disciplines, such as media content processing, database, information retrieval, and machine user interface.In the commercial domain, we have also witnessed several impressive efforts moving technologies into practical arenas.This panel includes experts from industry, research labs, and academia.The panel will assess the state of the art and articulate the important future directions in the general field of multimedia access and retrieval.
Shih-Fu Chang, Gwendal Auffret, Jonathan Foote, Chung-Sheng Li, Behzad Shahraray, Tanveer F. Syeda-Mahmood, HongJiang Zhang
ACM Multimedia (1)3
1999 Visualizing music and audio using self-similarity
abstract
This paper presents a novel approach to visualizing the time structure of music and audio. The acoustic similarity between any two instants of an audio recording is displayed in a 2D representation, allowing identification of structural and rhythmic characteristics. Examples are presented for classical and popular music. Applications include content-based analysis and segmentation, as well as tempo and structure extraction.
Jonathan Foote
ACM Multimedia (1)1
1999 Video Manga: generating semantically meaningful video summaries
abstract
This paper presents methods for automatically creating pictorial video summaries that resemble comic books. The relative importance of video segments is computed from their length and novelty. Image and audio analysis is used to automatically detect and emphasize meaningful events. Based on this importance measure, we choose relevant keyframes. Selected keyframes are sized by importance, and then efficiently packed into a pictorial summary. We present a quantitative measure of how well a summary captures the salient events in a video, and show how it can be used to improve our summaries. The result is a compact and visually pleasing summary that captures semantically important events, and is suitable for printing or Web access. Such a summary can be further enhanced by including text captions derived from OCR or other methods. We describe how the automatically generated summaries are used to simplify access to a large collection of videos.
Shingo Uchihashi, Jonathan Foote, Andreas Girgensohn, John S. Boreczky
ACM Multimedia (1)2
1999 MBase: indexing, browsing, and playback of media at FXPAL
abstract
No abstract available.
Lynn Wilcox, Shingo Uchihashi, Andreas Girgensohn, Jonathan Foote, John S. Boreczky
ACM Multimedia (2)4
1999 An Overview of Audio Information Retrieval
Jonathan Foote
Multim. Syst.1
1998 An Intelligent Media Browser Using Automatic Multimodal Analysis
abstract
Many techniques can extract information from an multimedia stream, such as speaker identity or shot boundaries. We present a browser that uses this information to navigate through stored media. Because automatically-derived information is not wholly reliable, it is transformed into a time-dependent “confidence score. ” When presented graphically, confidence scores enable users to make informed decisions about regions of interest in the media, so that non-interesting areas may be skipped. Additionally, index points may be determined automatically for easy navigation, selection, editing, and annotation and will support analysis types other than the speaker identification and shot detection used here. 1.1 Keywords Content-based retrieval, video, speaker identification, automatic analysis, visualization, skimming 2.
Jonathan Foote, John S. Boreczky, Andreas Girgensohn, Lynn Wilcox
ACM Multimedia1
1997 Acoustic indexing for multimedia retrieval and browsing
abstract
This paper reviews the Video Mail Retrieval (VMR) project at Cambridge University and ORL. The VMR project began in September 1993 with the aim of developing methods for retrieving video documents by scanning the audio soundtrack for keywords. The project has shown, both experimentally and through the construction of a working prototype, that speech recognition can be combined with information retrieval methods to locate multimedia documents by content. The final version of the VMR system uses pre-computed phone lattices to allow extremely rapid word spotting and audio indexing, and statistical information retrieval (IR) methods to mitigate the effects of spotting errors. The net result is a retrieval system that is open-vocabulary and speaker-independent, and which can search audio orders of magnitude faster than real time.
Steve J. Young, Martin G. Brown, Jonathan Foote, Gareth J. F. Jones, Karen Spärck Jones
ICASSP3
1997 Unconstrained keyword spotting using phone lattices with application to spoken document retrieval
Jonathan Foote, Steve J. Young, Gareth J. F. Jones, Karen Spärck Jones
Comput. Speech Lang.1
1996 Robust talker-independent audio document retrieval
abstract
The goal of the video mail retrieval (VMR) project is to integrate state-of-the-art document retrieval methods with speech recognition to yield a robust and efficient retrieval system. The work presented extends VMR towards an open-vocabulary, talker-independent system for retrieving spontaneously-spoken audio and video messages. We present results showing successful retrieval using a standard large-vocabulary (LV) recogniser, despite the lack of a matched language model and vocabulary. We further show that integrating a LV recogniser with conventional word spotting (WS) gives more robust retrieval performance than either method alone. This paper gives details of the message archive used, the speech recognition methodologies, the information retrieval methods, and experimental results.
Gareth J. F. Jones, Jonathan Foote, Karen Spärck Jones, Steve J. Young
ICASSP2
1996 Open-Vocabulary Speech Indexing for Voice and Video Mail Retrieval
abstract
This paper presents recentwork on amultimedia retrieval project at Cambridge University and Olivetti Research Limited (ORL). We present novel techniques that allow extremely rapid audio indexing, at rates approaching several thousand times real time. Unlike other methods, these techniques do not depend on a fixed vocabulary recognition system or on keywords that must be known well in advance. Using statistical methods developed for text, these indexing techniques allow rapid and efficient retrieval and browsing of audio and video documents. This paper presents the project background, the indexing and retrieval techniques, and a video mail retrieval application incorporating content-based audio indexing, retrieval, and browsing.
Martin G. Brown, Jonathan Foote, Gareth J. F. Jones, Karen Spärck Jones, Steve J. Young
ACM Multimedia2
1996 Retrieving Spoken Documents by Combining Multiple Index Sources
abstract
This paper presents domain-independent methods of spoken document retrieval.Both a continuous-speech large vocabulary recognition system, and a phone-lattice word spotter,
Gareth J. F. Jones, Jonathan Foote, Karen Spärck Jones, Steve J. Young
SIGIR2
1996 Experiments in Spoken Document Retrieval
Karen Spärck Jones, Gareth J. F. Jones, Jonathan Foote, Steve J. Young
Inf. Process. Manag.3
1995 Discrete MMI probability models for HMM speech recognition
abstract
This paper presents a method of non-parametrically modeling HMM output probabilities. Discrete output probabilities are estimated from a tree-based maximum mutual information (MMI) partition of the feature space, rather than the usual vector quantization. One advantage of a decision-tree method is that very high-dimensional spaces can be partitioned. Time variation can then be explicitly modeled by concatenating time-adjacent vectors, which is shown to improve recognition performance. Though the model is discrete, it provides recognition performance better than i-component Gaussian mixture HMMs on the ARPA Resource Management (RM) task. This method is not without drawbacks: because of its non-parametric nature, a large number of parameters are needed for a good model and the available RM training data is probably not sufficient. Besides the computational advantages of a discrete model, this method has promising applications in talker identification, adaptation, and clustering.
Jonathan Foote
ICASSP1
1995 Video mail retrieval: the effect of word spotting accuracy on precision
abstract
The goal of the video mail retrieval project is to integrate state-of-the-art document retrieval methods with high accuracy word spotting to yield a robust and efficient retrieval system. This paper describes a preliminary study to determine the extent to which retrieval precision is affected by word spotting performance. It includes a description of the database design, the word spotting algorithm, and the information retrieval method used. Results are presented which show audio retrieval performance very close to that of text.
Gareth J. F. Jones, Jonathan Foote, Karen Spärck Jones, Steve J. Young
ICASSP2
1995 WSJCAMO: a British English speech corpus for large vocabulary continuous speech recognition
abstract
A significant new speech corpus of British English has been recorded at Cambridge University. Derived from the Wall Street Journal text corpus, WSJCAMO constitutes one of the largest corpora of spoken British English currently in existence. It has been specifically designed for the construction and evaluation of speaker-independent speech recognition systems. The database consists of 140 speakers each speaking about 110 utterances. This paper describes the motivation for the corpus, the processes undertaken in its construction and the utilities needed as support tools. All utterance transcriptions have been verified and a phonetic dictionary has been developed to cover the training data and evaluation tasks. Two evaluation tasks have been defined using standard 5000 word bigram and 20000 word trigram language models. The paper concludes with comparative results on these tasks for British and American English.
Tony Robinson, Jeroen Fransen, David Pye, Jonathan Foote, Steve Renals
ICASSP4
1995 Talker-independent keyword spotting for information retrieval
abstract
The goal of the Video Mail Retrieval (VMR) project is to integTate state-of-the-art information retrieval (IR) methods with high-accuracy word spotting to yield a robust and efficient multimedia retrieval system. This paper concerns open-talker and arbitrary-keyword retrieval based on talker-independent subword models. Because talker-independent subword models can not be expected to work as well as the talker-dependent whole-keyword models used in previous VMR experiments, speaker adaptation is investigated as a means of improving perfor- mance (especially for talkers with non-British accents). Both standard FOM word spotting measures and actual retrieval results are computed. The results show that the FOM is not necessarily a good indicator of retrieval performance, and that talker adaptation can substantially improve both spotting and retrieval results.
Jonathan Foote, Gareth J. F. Jones, Karen Spärck Jones, Steve J. Young
EUROSPEECH1
1995 Automatic Content-Based Retrieval of broadcast News
abstract
This paper presents current work on a video retrieval project at Cambridge University and Olivetti Research Limited (ORL). We show that statistical methods developed for text retrieval are also effective for retrieving and browsing multimedia documents. These methods allow rapid retrieval of news broadcasts by information content determined from teletext subtitles. Information retrieval results for experiments performed on a large archive of news broadcasts are presented. This is made possible by the ORL Medusa system, which allows practical recording, storage, and playback of tens of gigabytes of multimedia data. This work is a step towards practical retrieval of multimedia documents, where the information content is determined from speech recognition performed on the audio soundtrack. We describe the project background, the ORL Medusa multimedia system, and retrieval application, as well as the news broadcast corpus and methods of browsing the retrieved news stories.
Martin G. Brown, Jonathan Foote, Gareth J. F. Jones, Karen Spärck Jones, Steve J. Young
ACM Multimedia2
1994 A model distance measure for talker clustering and identification
abstract
This paper describes methods of talker clustering and identification based on a "distance" metric between discrete HMM output probabilities. Output probabilities are derived on a tree-based MMI partition of the feature space, rather than the usual vector quantization. The information divergence (relative entropy) between speaker-dependent models is used as a quantitative measure of how much a given talker differs from another talker. An immediate application is talker identification: an unknown speaker may be identified by finding the closest speaker-dependent reference model to a model trained on the unknown speaker's data. Another application is to cluster similar talkers into a group; these may be used to train a HMM model that represents that talker better than a more general model. It is shown that using the model "nearest" a novel talker enhances the performance of a talker-independent speech recognition system.>
Jonathan Foote, Harvey F. Silverman
ICASSP (1)1
1993 Stop classification using DESA-1 high resolution formant tracking
Jonathan Foote, Daniel J. Mashao, Harvey F. Silverman
ICASSP (2)1
1992 Distributed hidden Markov model training on loosely-coupled multiprocessor networks
abstract
An explicit-duration hidden Markov model (HMM) algorithm for speech recognition has been proposed that potentially provides a more precise and versatile duration model than the implicit models ordinarily used, but at the cost of increased computation. The authors address the computational issues involved in conventional and explicit-duration HMM training by providing an analysis of the algorithm, suggesting serial enhancements and two efficient parallel implementations, and presenting experimental results on both common network workstations as well as a parallel system.>
Jonathan Foote, Mike Hochberg, Peter M. Athanas, Aaron Smith, Michael E. Wazlowski, Harvey F. Silverman
ICASSP1
1991 Hidden Markov model/neural network training techniques for connected alphadigit speech recognition
abstract
A neural network formulation for an HMM (hidden Markov model) is presented, and training using maximum likelihood, maximum mutual information, minimum mean-squared-error (MMSE), and unconstrained MMSE is described. Recognition results are presented for the variously trained models evaluated on a speaker-independent, connected alphadigit speech recognition task. It is concluded that viewing neural networks as HMMs provides a framework for building temporally dependent neural networks, while viewing HMMs as neural networks broadens the class of natural training methods. Despite several drawbacks, performance results indicate that models trained with error-correcting criteria on sufficient amounts of data may do better at discriminating similar sounds.>
Mike Hochberg, Les T. Niles, Jonathan Foote, Harvey F. Silverman
ICASSP3