Demo Data from EDBT 2026 / reviewers in the wild / expert
Jonathan Foote
dblp:95/5723 · also J. T. Foote
· DBLP profile ↗
54ranked-venue papers
16as first author
0since 2021 · last 2014
0000-0003-4411-1362ORCID · corroborated
Domains — the database's venue-derived domains; a paper can count in several
Graphics, computer vision, multimedia, augmented reality and games · 46 · 15 first-authorHuman-computer interaction and ubiquitous computing · 5Artificial intelligence and machine learning · 2 · 2 first-authorDatabases, data management, data science and information retrieval · 2Computer networks · 1Security and privacy · 1
Expertise — from the expertise taxonomy: the topics of the expert's papers under the CCF categories. A weight counts papers with recency: 1 for a paper about the topic, 0.3 when the topic is its context, halved every five years.
| Computer graphics and multimedia
17 papers |
Virtual and augmented reality · 47% Multimedia analysis and retrieval · 19% Image and video processing · 10% | |
| Network and information security
1 paper |
Systems and software security · 100% | |
| Software engineering, system software, and programming languages
1 paper |
Software testing · 100% | |
| Human-computer interaction and pervasive computing
7 papers |
Collaborative and social computing · 59% Interaction techniques and input · 19% Ubiquitous computing and smart environments · 13% |
Topics — the 30 heaviest of 46, each with the papers that count most for it
| Topic | Weight | Papers | Last | Evidence papers |
|---|---|---|---|---|
Systems and software security › vulnerability discovery
fuzzing |
0.2 | 1 | 2014 | Optimizing Seed Selection for Fuzzing · USENIX Security Symposium 2014 |
Systems and software security
vulnerability discovery |
0.2 | 1 | 2014 | Optimizing Seed Selection for Fuzzing · USENIX Security Symposium 2014 |
Software testing
fuzzing |
0.2 | 1 | 2014 | Optimizing Seed Selection for Fuzzing · USENIX Security Symposium 2014 |
Software testing › fuzzing › seed scheduling
seed selection |
0.2 | 1 | 2014 | Optimizing Seed Selection for Fuzzing · USENIX Security Symposium 2014 |
Virtual and augmented reality › immersive video
360-degree video |
0.1 | 4 | 2005 | Region of interest extraction and virtual camera control based on panoramic video capturing · IEEE Trans. Multim. 2005 Panoramic video capturing and compressed domain virtual camera control · ACM Multimedia 2001 FlyAbout: spatially indexed panoramic video · ACM Multimedia 2001 |
Collaborative and social computing
remote collaboration |
0.1 | 3 | 2010 | Remote interactive graffiti · ACM Multimedia 2004 The virtual factory: Exploring 3D worlds as industrial collaboration and control environments · VR 2010 The virtual chocolate factory: mixed reality industrial collaboration and control · ACM Multimedia 2010 |
Virtual and augmented reality
mixed reality |
0.1 | 1 | 2010 | The virtual factory: Exploring 3D worlds as industrial collaboration and control environments · VR 2010 |
Virtual and augmented reality › immersive interaction › 3d interaction
virtual camera control |
0.0 | 2 | 2005 | Panoramic video capturing and compressed domain virtual camera control · ACM Multimedia 2001 Region of interest extraction and virtual camera control based on panoramic video capturing · IEEE Trans. Multim. 2005 |
Multimedia analysis and retrieval › multimedia analysis › multimedia collection analysis
photo collection organization |
0.0 | 1 | 2003 | Managing digital memories with the FXPAL photo application · ACM Multimedia 2003 |
Multimedia analysis and retrieval › multimedia analysis
visual content analysis |
0.0 | 1 | 2003 | Temporal event clustering for digital photo collections · ACM Multimedia 2003 |
Interaction techniques and input
gesture input |
0.0 | 1 | 2003 | Shared interactive video for teleconferencing · ACM Multimedia 2003 |
Collaborative and social computing › computer-mediated communication
teleconferencing |
0.0 | 1 | 2003 | Shared interactive video for teleconferencing · ACM Multimedia 2003 |
Image and video processing › motion analysis
camera motion analysis |
0.0 | 2 | 2002 | A semi-automatic approach to home video editing · UIST 2000 Creating music videos using automatic media analysis · ACM Multimedia 2002 |
Audio and music processing
audio analysis |
0.0 | 1 | 2002 | Creating music videos using automatic media analysis · ACM Multimedia 2002 |
Visual content generation and editing › video editing
automatic video editing |
0.0 | 1 | 2002 | Creating music videos using automatic media analysis · ACM Multimedia 2002 |
Visual content generation and editing › video generation
music video generation |
0.0 | 1 | 2002 | Creating music videos using automatic media analysis · ACM Multimedia 2002 |
Information retrieval
content-based retrieval |
0.0 | 2 | 1996 | Open-Vocabulary Speech Indexing for Voice and Video Mail Retrieval · ACM Multimedia 1996 Automatic Content-Based Retrieval of broadcast News · ACM Multimedia 1995 |
Multimedia analysis and retrieval
video analysis |
0.0 | 1 | 2000 | A semi-automatic approach to home video editing · UIST 2000 |
Visual content generation and editing
video editing |
0.0 | 1 | 2000 | A semi-automatic approach to home video editing · UIST 2000 |
Audio and music processing
music information retrieval |
0.0 | 1 | 1999 | Visualizing music and audio using self-similarity · ACM Multimedia (1) 1999 |
Audio and music processing
music structure analysis |
0.0 | 1 | 1999 | Visualizing music and audio using self-similarity · ACM Multimedia (1) 1999 |
Multimedia analysis and retrieval
video summarization |
0.0 | 1 | 1999 | Video Manga: generating semantically meaningful video summaries · ACM Multimedia (1) 1999 |
Multimedia analysis and retrieval
multimedia browsing |
0.0 | 1 | 1998 | An Intelligent Media Browser Using Automatic Multimodal Analysis · ACM Multimedia 1998 |
Information retrieval › document retrieval
spoken document retrieval |
0.0 | 1 | 1996 | Retrieving Spoken Documents by Combining Multiple Index Sources · SIGIR 1996 |
Audio and music processing › audio analysis › audio content analysis
audio indexing |
0.0 | 1 | 1996 | Open-Vocabulary Speech Indexing for Voice and Video Mail Retrieval · ACM Multimedia 1996 |
Computational photography and imaging
projector-camera systems |
0.0 | 1 | 2004 | Remote interactive graffiti · ACM Multimedia 2004 |
Multimedia analysis and retrieval › video retrieval
news video retrieval |
0.0 | 1 | 1995 | Automatic Content-Based Retrieval of broadcast News · ACM Multimedia 1995 |
Transport protocols and congestion control › real-time communication
video conferencing |
0.0 | 1 | 2003 | Shared interactive video for teleconferencing · ACM Multimedia 2003 |
Image and video processing
video segmentation |
0.0 | 1 | 2002 | Creating music videos using automatic media analysis · ACM Multimedia 2002 |
Human-robot interaction
remote control |
0.0 | 1 | 2002 | FlySPEC: a multi-user video camera system with hybrid human and automatic control · ACM Multimedia 2002 |
Methods — techniques the papers use, named apart from their topics
optimization · 0.4virtual world mirroring · 0.2mobile and web-based display · 0.2data integration · 0.23d modeling · 0.2image alignment · 0.1camera distortion correction · 0.1kalman filter · 0.1video window interaction · 0.1gesture-based interaction · 0.1automated organization · 0.1motion analysis · 0.1compressed-domain analysis · 0.1pan/tilt/zoom camera control · 0.0pan-tilt-zoom camera control · 0.0statistical text retrieval · 0.0word spotting · 0.0continuous speech recognition · 0.0
| Year | Publication | Venue | Position |
|---|---|---|---|
| 2014 | Optimizing Seed Selection for Fuzzing
Alexandre Rebert, Sang Kil Cha, Thanassis Avgerinos, Jonathan Foote, David Warren, Gustavo Grieco, David Brumley |
USENIX Security Symposium | 4 |
| 2010 | The virtual chocolate factory: Building a real world mixed-reality system for industrial collaboration and controlabstractVirtual, mobile, and mixed reality systems have diverse uses for data visualization and remote collaboration in industrial settings, especially factories. We report our experiences in designing complex mixed-reality collaboration, control, and display systems for a real-world factory, for delivering real-time factory information to multiple types of users. In collaboration with TCHO, a chocolate maker in San Francisco, our research group is building a virtual "mirror" world of a real-world chocolate factory and its processes. Sensor data is imported into the multi-user 3D environment from hundreds of sensors on the factory floor. The resulting "virtual factory" is designed for simulation, visualization, and collaboration, using a set of interlinked, real-time layers of information about the factory and its processes. We are also looking at appropriate industrial uses for mobile devices such as cell phones and tablet computers, and how they intersect with virtual worlds and mixed realities. For example, an experimental iPhone web app provides mobile laboratory monitoring and control. The mobile system is integrated with the database underlying the virtual factory world. These systems were deployed at the real-world factory and lab in 2009, and are now in beta development. Through this mashup of mobile, social, mixed and virtual technologies, we hope to create industrial systems for enhanced collaboration between physically remote people and places - for example, factories in China with managers in Japan or the US. Maribeth Back, Don Kimber, Eleanor Gilbert Rieffel, Anthony Dunnigan, Bee Liew, Sagar Gattepally, Jonathan Foote, Jun Shingu, Jim Vaughan |
ICME | 7 |
| 2010 | The virtual chocolate factory: mixed reality industrial collaboration and controlabstractWe show several aspects of a complex mixed reality system that we have built and deployed in a real-world factory setting. In our system, virtual worlds, augmented realities, and social and mobile applications are all fed from the same infrastructure. In collaboration with TCHO[1], a chocolate maker in San Francisco, we built a virtual "mirror" world of a real-world chocolate factory and its processes. Sensor data is imported into the multi-user 3D environment from hundreds of sensors on the factory floor. The resulting virtual factory is used for simulation, visualization, and collaboration, using a set of interlinked, real-time layers of information. Another part of our infrastructure is designed to support appropriate industrial uses for mobile devices such as cell phones and tablet computers. We deployed this system at the real-world factory in 2009, and it is now is daily use there. By simultaneously developing mobile, virtual, and web-based display and collaboration environments, we aimed to create an infrastructure that did not skew toward one type of application but that could serve many at once, interchangeably. Through this mixture of mobile, social, mixed and virtual technologies, we hope to create systems for enhanced collaboration in industrial settings between physically remote people and places, such as factories in China with managers in the US. Maribeth Back, Don Kimber, Eleanor Gilbert Rieffel, Anthony Dunnigan, Bee Liew, Sagar Gattepally, Jonathan Foote, Jun Shingu, Jim Vaughan |
ACM Multimedia | 7 |
| 2010 | The virtual factory: Exploring 3D worlds as industrial collaboration and control environmentsabstractThis project investigates practical uses of virtual, mobile, and mixed reality systems in industrial settings, in particular control and collaboration applications for factories. In collaboration with TCHO, a chocolate maker start-up in San Francisco, we have built virtual mirror-world representations of a real-world chocolate factory and are importing its data and modeling its processes. The system integrates mobile devices such as cell phones and tablet computers. The resulting "virtual factory" is a cross-reality environment designed for simulation, visualization, and collaboration, using a set of interlinked, real-time 3D and 2D layers of information about the factory and its processes. Maribeth Back, Timothy Childs, Anthony Dunnigan, Jonathan Foote, Sagar Gattepally, Bee Liew, Jun Shingu, Jim Vaughan |
VR | 4 |
| 2005 | Discriminative techniques for keyframe selectionabstractA convenient representation of a video segment is a single "keyframe". Keyframes are widely used in applications such as non-linear browsing and video editing. With existing methods of keyframe selection, similar video segments result in very similar keyframes, with the drawback that actual differences between the segments may be obscured. We present methods for keyframe selection based on two criteria: capturing the similarity to the represented segment, and preserving the differences from other segment keyframes, so that different segments will have visually distinct representations. We present two discriminative keyframe selection methods, and an example of experimental results. Matthew Cooper 0002, Jonathan Foote |
ICME | 2 |
| 2005 | Mediated Meeting Interaction for TeleconferencingabstractA common problem with teleconferences is awkward turn-taking – particularly ‘ collisions,’ whereby multiple parties inadvertently speak over each other due to communication delays. We propose a model for teleconference discussions including the effects of delays, and describe tools that can improve the quality of those interactions. We describe an interface to gently provide latency awareness, and to give advanced notice of ‘ incoming speech’ to help participants avoid collisions. This is possible when codec latencies are significant, or when a low bandwidth side channel or out-of-band signaling is available with lower latency than the primary video channel. We report on results of simulations, and of experiments carried out with transpacific meetings, that demonstrate these tools can improve the quality of teleconference discussions. Kazumasa Murai, Don Kimber, Jonathan Foote, Qiong Liu 0003, John Doherty |
ICME | 3 |
| 2005 | Region of interest extraction and virtual camera control based on panoramic video capturingabstractWe present a system for automatically extracting the region of interest (ROI) and controlling virtual cameras' control based on panoramic video. It targets applications such as classroom lectures and video conferencing. For capturing panoramic video, we use the FlyCam system that produces high resolution, wide-angle video by stitching video images from multiple stationary cameras. To generate conventional video, a region of interest can be cropped from the panoramic video. We propose methods for ROI detection, tracking, and virtual camera control that work in both the uncompressed and compressed domains. The ROI is located from motion and color information in the uncompressed domain and macroblock information in the compressed domain, and tracked using a Kalman filter. This results in virtual camera control that simulates human controlled video recording. The system has no physical camera motion and the virtual camera parameters are readily available for video indexing. Xinding Sun, Jonathan Foote, Don Kimber, B. S. Manjunath |
IEEE Trans. Multim. | 2 |
| 2005 | Temporal event clustering for digital photo collectionsabstractOrganizing digital photograph collections according to events such as holiday gatherings or vacations is a common practice among photographers. To support photographers in this task, we present similarity-based methods to cluster digital photos by time and image content. The approach is general and unsupervised, and makes minimal assumptions regarding the structure or statistics of the photo collection. We present several variants of an automatic unsupervised algorithm to partition a collection of digital photographs based either on temporal similarity alone, or on temporal and content-based similarity. First, interphoto similarity is quantified at multiple temporal scales to identify likely event clusters. Second, the final clusters are determined according to one of three clustering goodness criteria. The clustering criteria trade off computational complexity and performance. We also describe a supervised clustering method based on learning vector quantization. Finally, we review the results of an experimental evaluation of the proposed algorithms and existing approaches on two test collections. Matthew Cooper 0002, Jonathan Foote, Andreas Girgensohn, Lynn Wilcox |
ACM Trans. Multim. Comput. Commun. Appl. | 2 |
| 2004 | Remote interactive graffitiabstractWe present an installation that allows distributed internet participants to "draw" on a public scene using light. The iLight system is a camera/projector system designed for remote collaboration. Using a familiar digital drawing interface, remote users "draw" on a live video image of a real-life object or scene. Graics drawn by the user are then projected onto the scene, where they are visible in the camera image. Because camera distortions are corrected and the video is aligned with the image canvas, drawn graics appear exactly where desired. Thus the remote users may harmlessly mark a ysical object to serve their own their artistic and/or expressive needs. We also describe how local participants may interact with remote users through the projected images. Besides the intrinsic "neat factor" of action at a distance, this installation serves as an experiment in how multiple users from different locales and cultures can create a social space that interacts with a ysical one, as well as raising issues of free expression in a non-destructive context. Jonathan Foote, Don Kimber |
ACM Multimedia | 1 |
| 2003 | Automatically organizing digital photographs using time and contentabstractWe present similarity-based methods to cluster digital photos by time and image content. This approach is general, unsupervised, and makes minimal assumptions regarding the structure or statistics of the photo collection. We describe versions of the algorithm using temporal similarity with and without content-based similarity, and compare the algorithms with existing techniques, measured against ground-truth clusters created by humans. Matthew Cooper 0002, Jonathan Foote, Andreas Girgensohn |
ICIP (3) | 2 |
| 2003 | Time base modulation: a new approach to watermarking audioabstractA novel method is presented for inaudibly hiding information in an audio signal by subtly applying time-scale modification to segments of the signal. The sequence, duration, and degree of the time-scale modifications are the parameters which encode information in the altered signal. By comparing the altered signal with a reference copy, compressed and expanded regions can be identified and the hidden data recovered. This approach is novel and has several advantages over other methods: it is theoretically noiseless, it introduces no spectral distortion, and it is robust to all known methods of reproduction, compression, and transmission. Jonathan Foote, John Adcock, Andreas Girgensohn |
ICME | 1 |
| 2003 | Multichannel video/audio acquisition for immersive conferencingabstractThis paper presents an information-driven audiovisual signal acquisition approach. This approach has several advantages: users are encouraged to assist in signal acquisition; available sensors are managed based on both signal characteristics and users' suggestions. The problem formulation is consistent with many well-known empirical approaches widely used in previous systems and may provide analytical explanations to these approaches. We demonstrate the use of this approach to pan/tilt/zoom (PTZ) camera management with field data. Qiong Liu 0003, Don Kimber, Jonathan Foote, Chunyuan Liao |
ICME | 3 |
| 2003 | Manipulating and Annotating Slides in a Multi-Display Environment
Patrick Chiu, Qiong Liu 0003, John S. Boreczky, Jonathan Foote, Don Kimber, Surapong Lertsithichai, Chunyuan Liao |
INTERACT | 4 |
| 2003 | Simplifying the Management of Large Photo Collections
Andreas Girgensohn, John Adcock, Matthew Cooper 0002, Jonathan Foote, Lynn Wilcox |
INTERACT | 4 |
| 2003 | Managing digital memories with the FXPAL photo applicationabstractThe FXPAL Photo Application is designed to faciliate the organization of digital images from digital cameras and other sources through automated organization and intuitive user interfaces. John Adcock, Matthew Cooper 0002, John Doherty, Jonathan Foote, Andreas Girgensohn, Lynn Wilcox |
ACM Multimedia | 4 |
| 2003 | Temporal event clustering for digital photo collectionsabstractWe present similarity-based methods to cluster digital photos by time and image content. The approach is general, unsupervised, and makes minimal assumptions regarding the structure or statistics of the photo collection. We present results for the algorithm based solely on temporal similarity, and jointly on temporal and content-based similarity. We also describe a supervised algorithm based on learning vector quantization. Finally, we include experimental results for the proposed algorithms and several competing approaches on two test collections. Matthew Cooper 0002, Jonathan Foote, Andreas Girgensohn, Lynn Wilcox |
ACM Multimedia | 2 |
| 2003 | Shared interactive video for teleconferencingabstractWe present a system that allows remote and local participants to control devices in a meeting environment using mouse or pen based gestures "through" video windows. Unlike state-of-the-art device control interfaces that require interaction with text commands, buttons, or other artificial symbols, our approach allows users to interact with devices through live video of the environment. This naturally extends our video supported pan/tilt/zoom (PTZ) camera control system, by allowing gestures in video windows to control not only PTZ cameras, but also other devices visible in video images. For example, an authorized meeting participant can show a presentation on a screen by dragging the file on a personal laptop and dropping it on the video image of the presentation screen. This paper presents the system architecture, implementation tradeoffs, and various meeting control scenarios. Chunyuan Liao, Qiong Liu 0003, Don Kimber, Patrick Chiu, Jonathan Foote, Lynn Wilcox |
ACM Multimedia | 5 |
| 2002 | Enhanced video browsing using automatically extracted audio excerptsabstractWe present a method for rapidly and robustly extracting audio excerpts without the overhead of speech recognition or speaker segmentation. An immediate application is to automatically augment keyframe-based video summaries with informative audio excerpts associated with the video segments represented by the keyframes. Short audio clips combined with keyframes comprise an extremely lightweight and Web-browsable interface for auditioning video or similar media, without using bandwidth-intensive streaming video or audio. Jonathan Foote, Matthew Cooper 0002, Lynn Wilcox |
ICME (2) | 1 |
| 2002 | Managing a camera system to serve different video requestsabstractThis paper presents a camera system called FlySPEC. In contrast to a traditional camera system that provides the same video stream to every user, FlySPEC can simultaneously serve different video-viewing requests. This flexibility allows users to conveniently participate in a seminar or meeting at their own pace. Meanwhile, the FlySPEC system provides a seamless blend of manual control and automation. With this control mix, users can easily make tradeoffs between video capture effort and video quality. The FlySPEC camera is constructed by installing a set of pan/tilt/zoom (PTZ) cameras near a high-resolution panoramic camera. While the panoramic camera provides the basic functionality of serving different viewing requests, the PTZ camera is managed by our algorithm to improve the overall video quality that may affect users watching details. The video resolution improvements from using different camera management strategies are compared in the experimental section. Qiong Liu 0003, Don Kimber, Lynn Wilcox, Matthew Cooper 0002, Jonathan Foote, John S. Boreczky |
ICME (2) | 5 |
| 2002 | Detecting path intersections in panoramic videoabstractGiven panoramic video taken along a self-intersecting path, we present a method for detecting the intersection points. This allows "virtual tours" to be synthesized by splicing the panoramic video at the intersection points. Spatial intersections are detected by finding the best-matching panoramic images from a number of nearby candidates. Each panoramic image is segmented into horizontal strips. Each strip is averaged in the vertical direction. The Fourier coefficients of the resulting 1-D data capture the rotation-invariant horizontal texture of each panoramic image. The distance between two panoramic images is calculated as the sum of the distances between their strip texture pairs at the same row positions. The intersection is chosen as the two candidate panoramic images that have the minimum distance. Xinding Sun, Don Kimber, Jonathan Foote, B. S. Manjunath |
ICME (2) | 3 |
| 2002 | Creating music videos using automatic media analysisabstractWe present methods for automatic and semi-automatic creation of music videos, given an arbitrary audio soundtrack and source video. Significant audio changes are automatically detected; similarly, the source video is automatically segmented and analyzed for suitability based on camera motion and exposure. Video with excessive camera motion or poor contrast is penalized with a high unsuitability score, and is more likely to be discarded in the final edit. High quality video clips are then automatically selected and aligned in time with significant audio changes. Video clips are adjusted to match the audio segments by selecting the most suitable region of the desired length. Besides a fully automated solution, our system can also start with clips manually selected and ordered using a graphical interface. The video is then created by truncating the selected clips (preserving the high quality portions) to produce a video digest that is synchronized with the soundtrack music, thus enhancing the impact of both. Jonathan Foote, Matthew Cooper 0002, Andreas Girgensohn |
ACM Multimedia | 1 |
| 2002 | FlySPEC: a multi-user video camera system with hybrid human and automatic controlabstractFlySPEC is a video camera system designed for real-time remote operation. A hybrid design combines the high resolution of an optomechanical video camera with the wide field of view always available from a panoramic camera. The control system integrates requests from multiple users so that each controls a virtual camera. The control system seamlessly integrates manual and fully automatic control. It supports a range of options from untended automatic to full manual control. The system can also learn control strategies from user requests. Additionally, the panoramic view is always available for an intuitive interface, and objects are never out of view regardless of the zoom factor. We present the system architecture, an information-theoretic approach to combining panoramic and zoomed images to optimally satisfy user requests, and experimental results that show the FlySPEC system significantly assists users in a remote inspection tasks. Qiong Liu 0003, Don Kimber, Jonathan Foote, Lynn Wilcox, John S. Boreczky |
ACM Multimedia | 3 |
| 2001 | Scene boundary detection via video self-similarity analysisabstractWe present a novel framework for analyzing video using self-similarity. Video scenes are located by analyzing inter-frame similarity matrices. The approach is flexible to the choice of similarity measure and is robust and data-independent because the data is used to model itself. We present the approach and its application to scene boundary detection. This is shown to dramatically outperform a conventional scene-boundary detector that uses a histogram-based measure of frame difference. Matthew Cooper 0002, Jonathan Foote |
ICIP (3) | 2 |
| 2001 | Recording the region of interest from FlyCam panoramic videoabstractA novel method for region of interest tracking and recording video is presented. The proposed method is based on the FlyCam system (Foote et al., 2000), which produces high-resolution and wide-angle video sequences by stitching the video frames from multiple stationary cameras. The method integrates tracking and recording processes, and targets applications such as classroom lectures and videoconferencing. First, the region of interest (which typically covers the speaker) is tracked using a Kalman filter. Then, the Kalman filter estimation results are used for virtual camera control and to record the video. The system has no physical camera motion and the virtual camera parameters are readily available for video indexing. The proposed system has been implemented for real-time recording of lectures and presentations. Xinding Sun, Jonathan Foote, Don Kimber, B. S. Manjunath |
ICIP (1) | 2 |
| 2001 | The Beat Spectrum: A New Approach To Rhythm AnalysisabstractWe introduce the beat spectrum, a new method of automatically characterizing the rhythm and tempo of music and audio. The beat spectrum is a measure of acoustic self-similarity as a function of time lag. Highly structured or repetitive music will have strong beat spectrum peaks at the repetition times. This reveals both tempo and the relative strength of particular beats, and therefore can distinguish between different kinds of rhythms at the same tempo. We also introduce the beat spectrogram which graphically illustrates rhythm variation over time. Unlike previous approaches to tempo analysis, the beat spectrum does not depend on particular attributes such as energy or frequency, and thus will work for any music or audio in any genre. We present tempo estimation results which are accurate to within 1 % for a variety of musical genres. This approach has a variety of applications, including music retrieval by similarity and automatically generating music videos. 1. Jonathan Foote, Shingo Uchihashi |
ICME | 1 |
| 2001 | FlyAbout: spatially indexed panoramic videoabstractWe describe a system called FlyAbout which uses spatially indexed panoramic video for virtual reality applications. Panoramic video is captured by moving a [email protected] camera along continuous paths. Users can interactively replay the video with the ability to view any interesting object or choose a particular direction. Spatially indexed video gives the ability to travel along paths or roads with a map-like interface. At junctions, or intersection points, users can chose which path to follow as well as which direction to look, allowing interaction not available with conventional video. Combining the spatial index with a spatial databsdde of maps or objects allows users to navigate to specific locations or interactively inspect particular objects. Don Kimber, Jonathan Foote, Surapong Lertsithichai |
ACM Multimedia | 2 |
| 2001 | Panoramic video capturing and compressed domain virtual camera controlabstractA system for capturing panoramic video and a novel method for corresponding compressed domain virtual camera control is presented. It targets applications such as classroom lectures and video conferencing. The proposed method is based on the FlyCam panoramic video system that is designed to produce high resolution and wide-angle video sequences by stitching the video pictures from multiple stationary cameras. The panoramic video sequence is compressed into an MPEG-2 stream for delivery. The proposed method integrates region of Interest (ROI) detection, tracking, and virtual camera control, and works on compressed domain information only. It first detects the ROI in the P (predictive coded) picture using only the macroblock type information, It then up-samples this detection result to obtain the ROI of the whole video stream. The ROI is tracked using a Kalman filter. The Kalman filter estimation results are used for virtual camera control that simulates human controlled video recording. The system has no physical camera motion and the virtual camera parameters are readily available for video indexing. The proposed system has been implemented for real time processing. Xinding Sun, Jonathan Foote, Don Kimber, B. S. Manjunath |
ACM Multimedia | 2 |
| 2000 | FlyCam: practical panoramic video
Jonathan Foote, Don Kimber |
ACM Multimedia | 1 |
| 2000 | A semi-automatic approach to home video editingabstractHitchcock is a system that allows users to easily create custom videos from raw video shot with a standard video camera.In contrast to other video editing systems, Hitchcock uses automatic analysis to determine the suitability of portions of the raw video.Unsuitable video typically has fast or erratic camera motion.Hitchcock first analyzes video to identify the type and amount of camera motion: fast pan, slow zoom, etc.Based on this analysis, a numerical "unsuitability" score is computed for each frame of the video.Combined with standard editing rules, this score is used to identify clips for inclusion in the final video and to select their start and end points.To create a custom video, the user drags keyframes corresponding to the desired clips into a storyboard.Users can lengthen or shorten the clip without specifying the start and end frames explicitly.Clip lengths are balanced automatically using a spring-based algorithm. Andreas Girgensohn, John S. Boreczky, Patrick Chiu, John Doherty, Jonathan Foote, Gene Golovchinsky, Shingo Uchihashi, Lynn Wilcox |
UIST | 5 |
| 1999 | Finding presentations in recorded meetings using audio and video featuresabstractThis paper describes a method for finding segments in video-recorded meetings that correspond to presentations. These segments serve as indexes into the recorded meeting. The system automatically detects intervals of video that correspond to presentation slides. We assume that only one person speaks during an interval when slides are detected. Thus these intervals can be used as training data for a speaker spotting system. An HMM is automatically constructed and trained on the audio data from each slide interval. A Viterbi alignment then resegments the audio according to speaker. Since the same speaker may talk across multiple slide intervals the acoustic data from these intervals is clustered to yield an estimate of the number of distinct speakers and their order. This allows the individual presentations in the video to be identified from the location of each presenter's speech. Results are presented for a corpus of six meeting videos. Jonathan Foote, John S. Boreczky, Lynn Wilcox |
ICASSP | 1 |
| 1999 | Video classification using transform coefficientsabstractThis paper describes techniques for classifying video frames using statistical models of reduced DCT or Hadamard transform coefficients. When decimated in time and reduced using truncation or principal component analysis, transform coefficients taken across an entire frame image allow rapid modeling, segmentation and similarity calculation. Unlike color-histogram metrics, this approach models image composition and works on grayscale images. Modeling the statistics of the transformed video frame images gives a likelihood measure that allows video to be segmented, classified, and ranked by similarity for retrieval. Experiments are presented that show an 87% correct classification rate for different classes. Applications are presented including a content-aware video browser. Andreas Girgensohn, Jonathan Foote |
ICASSP | 2 |
| 1999 | Summarizing video using a shot importance measure and a frame-packing algorithmabstractThis paper presents methods of generating compact pictorial summarizations of video. By calculating a measure of shot importance video can be summarized by de-emphasizing or discarding less important information, such as repeated or common scenes. In contrast to other approaches that present keyframes for each shot, this measure allows summarization by presenting only the most important shots. Selected keyframes can also be resized depending on their relative importance. We present an efficient packing algorithm that constructs a pictorial representation from differently-sized keyframes. This results in a compact and visually pleasing summary reminiscent of a comic book. Shingo Uchihashi, Jonathan Foote |
ICASSP | 2 |
| 1999 | Facilitating Video Access by Visualizing Automatic Analysis
Andreas Girgensohn, John S. Boreczky, Lynn Wilcox, Jonathan Foote |
INTERACT | 4 |
| 1999 | Multimedia access and retrieval: the state of the art and future directions (panel session)abstractSeveral years have passed since the research topic of content based multimedia retrieval emerged.We have witnessed the burgeoning research activities into a plenitude of new indexing, retrieval, and filtering tools for images, video, audio, music, graphics, and their combinations with text-based information.Exciting research opportunities arise when integrating knowledge from multiple disciplines, such as media content processing, database, information retrieval, and machine user interface.In the commercial domain, we have also witnessed several impressive efforts moving technologies into practical arenas.This panel includes experts from industry, research labs, and academia.The panel will assess the state of the art and articulate the important future directions in the general field of multimedia access and retrieval. Shih-Fu Chang, Gwendal Auffret, Jonathan Foote, Chung-Sheng Li, Behzad Shahraray, Tanveer F. Syeda-Mahmood, HongJiang Zhang |
ACM Multimedia (1) | 3 |
| 1999 | Visualizing music and audio using self-similarityabstractThis paper presents a novel approach to visualizing the time structure of music and audio. The acoustic similarity between any two instants of an audio recording is displayed in a 2D representation, allowing identification of structural and rhythmic characteristics. Examples are presented for classical and popular music. Applications include content-based analysis and segmentation, as well as tempo and structure extraction. Jonathan Foote |
ACM Multimedia (1) | 1 |
| 1999 | Video Manga: generating semantically meaningful video summariesabstractThis paper presents methods for automatically creating pictorial video summaries that resemble comic books. The relative importance of video segments is computed from their length and novelty. Image and audio analysis is used to automatically detect and emphasize meaningful events. Based on this importance measure, we choose relevant keyframes. Selected keyframes are sized by importance, and then efficiently packed into a pictorial summary. We present a quantitative measure of how well a summary captures the salient events in a video, and show how it can be used to improve our summaries. The result is a compact and visually pleasing summary that captures semantically important events, and is suitable for printing or Web access. Such a summary can be further enhanced by including text captions derived from OCR or other methods. We describe how the automatically generated summaries are used to simplify access to a large collection of videos. Shingo Uchihashi, Jonathan Foote, Andreas Girgensohn, John S. Boreczky |
ACM Multimedia (1) | 2 |
| 1999 | MBase: indexing, browsing, and playback of media at FXPALabstractNo abstract available. Lynn Wilcox, Shingo Uchihashi, Andreas Girgensohn, Jonathan Foote, John S. Boreczky |
ACM Multimedia (2) | 4 |
| 1999 | An Overview of Audio Information Retrieval
Jonathan Foote |
Multim. Syst. | 1 |
| 1998 | An Intelligent Media Browser Using Automatic Multimodal AnalysisabstractMany techniques can extract information from an multimedia stream, such as speaker identity or shot boundaries. We present a browser that uses this information to navigate through stored media. Because automatically-derived information is not wholly reliable, it is transformed into a time-dependent “confidence score. ” When presented graphically, confidence scores enable users to make informed decisions about regions of interest in the media, so that non-interesting areas may be skipped. Additionally, index points may be determined automatically for easy navigation, selection, editing, and annotation and will support analysis types other than the speaker identification and shot detection used here. 1.1 Keywords Content-based retrieval, video, speaker identification, automatic analysis, visualization, skimming 2. Jonathan Foote, John S. Boreczky, Andreas Girgensohn, Lynn Wilcox |
ACM Multimedia | 1 |
| 1997 | Acoustic indexing for multimedia retrieval and browsingabstractThis paper reviews the Video Mail Retrieval (VMR) project at Cambridge University and ORL. The VMR project began in September 1993 with the aim of developing methods for retrieving video documents by scanning the audio soundtrack for keywords. The project has shown, both experimentally and through the construction of a working prototype, that speech recognition can be combined with information retrieval methods to locate multimedia documents by content. The final version of the VMR system uses pre-computed phone lattices to allow extremely rapid word spotting and audio indexing, and statistical information retrieval (IR) methods to mitigate the effects of spotting errors. The net result is a retrieval system that is open-vocabulary and speaker-independent, and which can search audio orders of magnitude faster than real time. Steve J. Young, Martin G. Brown, Jonathan Foote, Gareth J. F. Jones, Karen Spärck Jones |
ICASSP | 3 |
| 1997 | Unconstrained keyword spotting using phone lattices with application to spoken document retrieval
Jonathan Foote, Steve J. Young, Gareth J. F. Jones, Karen Spärck Jones |
Comput. Speech Lang. | 1 |
| 1996 | Robust talker-independent audio document retrievalabstractThe goal of the video mail retrieval (VMR) project is to integrate state-of-the-art document retrieval methods with speech recognition to yield a robust and efficient retrieval system. The work presented extends VMR towards an open-vocabulary, talker-independent system for retrieving spontaneously-spoken audio and video messages. We present results showing successful retrieval using a standard large-vocabulary (LV) recogniser, despite the lack of a matched language model and vocabulary. We further show that integrating a LV recogniser with conventional word spotting (WS) gives more robust retrieval performance than either method alone. This paper gives details of the message archive used, the speech recognition methodologies, the information retrieval methods, and experimental results. Gareth J. F. Jones, Jonathan Foote, Karen Spärck Jones, Steve J. Young |
ICASSP | 2 |
| 1996 | Open-Vocabulary Speech Indexing for Voice and Video Mail RetrievalabstractThis paper presents recentwork on amultimedia retrieval project at Cambridge University and Olivetti Research Limited (ORL). We present novel techniques that allow extremely rapid audio indexing, at rates approaching several thousand times real time. Unlike other methods, these techniques do not depend on a fixed vocabulary recognition system or on keywords that must be known well in advance. Using statistical methods developed for text, these indexing techniques allow rapid and efficient retrieval and browsing of audio and video documents. This paper presents the project background, the indexing and retrieval techniques, and a video mail retrieval application incorporating content-based audio indexing, retrieval, and browsing. Martin G. Brown, Jonathan Foote, Gareth J. F. Jones, Karen Spärck Jones, Steve J. Young |
ACM Multimedia | 2 |
| 1996 | Retrieving Spoken Documents by Combining Multiple Index SourcesabstractThis paper presents domain-independent methods of spoken document retrieval.Both a continuous-speech large vocabulary recognition system, and a phone-lattice word spotter, Gareth J. F. Jones, Jonathan Foote, Karen Spärck Jones, Steve J. Young |
SIGIR | 2 |
| 1996 | Experiments in Spoken Document Retrieval
Karen Spärck Jones, Gareth J. F. Jones, Jonathan Foote, Steve J. Young |
Inf. Process. Manag. | 3 |
| 1995 | Discrete MMI probability models for HMM speech recognitionabstractThis paper presents a method of non-parametrically modeling HMM output probabilities. Discrete output probabilities are estimated from a tree-based maximum mutual information (MMI) partition of the feature space, rather than the usual vector quantization. One advantage of a decision-tree method is that very high-dimensional spaces can be partitioned. Time variation can then be explicitly modeled by concatenating time-adjacent vectors, which is shown to improve recognition performance. Though the model is discrete, it provides recognition performance better than i-component Gaussian mixture HMMs on the ARPA Resource Management (RM) task. This method is not without drawbacks: because of its non-parametric nature, a large number of parameters are needed for a good model and the available RM training data is probably not sufficient. Besides the computational advantages of a discrete model, this method has promising applications in talker identification, adaptation, and clustering. Jonathan Foote |
ICASSP | 1 |
| 1995 | Video mail retrieval: the effect of word spotting accuracy on precisionabstractThe goal of the video mail retrieval project is to integrate state-of-the-art document retrieval methods with high accuracy word spotting to yield a robust and efficient retrieval system. This paper describes a preliminary study to determine the extent to which retrieval precision is affected by word spotting performance. It includes a description of the database design, the word spotting algorithm, and the information retrieval method used. Results are presented which show audio retrieval performance very close to that of text. Gareth J. F. Jones, Jonathan Foote, Karen Spärck Jones, Steve J. Young |
ICASSP | 2 |
| 1995 | WSJCAMO: a British English speech corpus for large vocabulary continuous speech recognitionabstractA significant new speech corpus of British English has been recorded at Cambridge University. Derived from the Wall Street Journal text corpus, WSJCAMO constitutes one of the largest corpora of spoken British English currently in existence. It has been specifically designed for the construction and evaluation of speaker-independent speech recognition systems. The database consists of 140 speakers each speaking about 110 utterances. This paper describes the motivation for the corpus, the processes undertaken in its construction and the utilities needed as support tools. All utterance transcriptions have been verified and a phonetic dictionary has been developed to cover the training data and evaluation tasks. Two evaluation tasks have been defined using standard 5000 word bigram and 20000 word trigram language models. The paper concludes with comparative results on these tasks for British and American English. Tony Robinson, Jeroen Fransen, David Pye, Jonathan Foote, Steve Renals |
ICASSP | 4 |
| 1995 | Talker-independent keyword spotting for information retrievalabstractThe goal of the Video Mail Retrieval (VMR) project is to integTate state-of-the-art information retrieval (IR) methods with high-accuracy word spotting to yield a robust and efficient multimedia retrieval system. This paper concerns open-talker and arbitrary-keyword retrieval based on talker-independent subword models. Because talker-independent subword models can not be expected to work as well as the talker-dependent whole-keyword models used in previous VMR experiments, speaker adaptation is investigated as a means of improving perfor- mance (especially for talkers with non-British accents). Both standard FOM word spotting measures and actual retrieval results are computed. The results show that the FOM is not necessarily a good indicator of retrieval performance, and that talker adaptation can substantially improve both spotting and retrieval results. Jonathan Foote, Gareth J. F. Jones, Karen Spärck Jones, Steve J. Young |
EUROSPEECH | 1 |
| 1995 | Automatic Content-Based Retrieval of broadcast NewsabstractThis paper presents current work on a video retrieval project at Cambridge University and Olivetti Research Limited (ORL). We show that statistical methods developed for text retrieval are also effective for retrieving and browsing multimedia documents. These methods allow rapid retrieval of news broadcasts by information content determined from teletext subtitles. Information retrieval results for experiments performed on a large archive of news broadcasts are presented. This is made possible by the ORL Medusa system, which allows practical recording, storage, and playback of tens of gigabytes of multimedia data. This work is a step towards practical retrieval of multimedia documents, where the information content is determined from speech recognition performed on the audio soundtrack. We describe the project background, the ORL Medusa multimedia system, and retrieval application, as well as the news broadcast corpus and methods of browsing the retrieved news stories. Martin G. Brown, Jonathan Foote, Gareth J. F. Jones, Karen Spärck Jones, Steve J. Young |
ACM Multimedia | 2 |
| 1994 | A model distance measure for talker clustering and identificationabstractThis paper describes methods of talker clustering and identification based on a "distance" metric between discrete HMM output probabilities. Output probabilities are derived on a tree-based MMI partition of the feature space, rather than the usual vector quantization. The information divergence (relative entropy) between speaker-dependent models is used as a quantitative measure of how much a given talker differs from another talker. An immediate application is talker identification: an unknown speaker may be identified by finding the closest speaker-dependent reference model to a model trained on the unknown speaker's data. Another application is to cluster similar talkers into a group; these may be used to train a HMM model that represents that talker better than a more general model. It is shown that using the model "nearest" a novel talker enhances the performance of a talker-independent speech recognition system.> Jonathan Foote, Harvey F. Silverman |
ICASSP (1) | 1 |
| 1993 | Stop classification using DESA-1 high resolution formant tracking
Jonathan Foote, Daniel J. Mashao, Harvey F. Silverman |
ICASSP (2) | 1 |
| 1992 | Distributed hidden Markov model training on loosely-coupled multiprocessor networksabstractAn explicit-duration hidden Markov model (HMM) algorithm for speech recognition has been proposed that potentially provides a more precise and versatile duration model than the implicit models ordinarily used, but at the cost of increased computation. The authors address the computational issues involved in conventional and explicit-duration HMM training by providing an analysis of the algorithm, suggesting serial enhancements and two efficient parallel implementations, and presenting experimental results on both common network workstations as well as a parallel system.> Jonathan Foote, Mike Hochberg, Peter M. Athanas, Aaron Smith, Michael E. Wazlowski, Harvey F. Silverman |
ICASSP | 1 |
| 1991 | Hidden Markov model/neural network training techniques for connected alphadigit speech recognitionabstractA neural network formulation for an HMM (hidden Markov model) is presented, and training using maximum likelihood, maximum mutual information, minimum mean-squared-error (MMSE), and unconstrained MMSE is described. Recognition results are presented for the variously trained models evaluated on a speaker-independent, connected alphadigit speech recognition task. It is concluded that viewing neural networks as HMMs provides a framework for building temporally dependent neural networks, while viewing HMMs as neural networks broadens the class of natural training methods. Despite several drawbacks, performance results indicate that models trained with error-correcting criteria on sufficient amounts of data may do better at discriminating similar sounds.> Mike Hochberg, Les T. Niles, Jonathan Foote, Harvey F. Silverman |
ICASSP | 3 |