Due to the rapid increase in video capture technology, more and more tourist videos are captured every day, creating a challenge for organization and association with metadata. In this paper, we present a novel system for annotating and navigating tourist videos. Placing annotations in a video is difficult because of the need to track the movement of the camera. Navigation of a regular video is also challenging due to the sequential nature of the media. To overcome these challenges, we introduce a system for registering videos to geo-referenced 3D models and analyzing the video contents. We also introduce a novel scheduling algorithm for showing annotations in video. We show results in automatically annotated videos and in a map-based application for browsing videos. Our user study indicates the system is very useful.

Systems such as Google Street View and Bing Maps Streetside enable users to virtually visit cities by navigating between immersive 360° panoramas, or bubbles. The discrete moves from bubble to bubble enabled in these systems do not provide a good visual sense of a larger aggregate such as a whole city block. Multi-perspective "strip" panoramas can provide a visual summary of a city street but lack the full realism of immersive panoramas. We present Street Slide, which combines the best aspects of the immersive nature of bubbles with the overview provided by multi-perspective strip panoramas. We demonstrate a seamless transition between bubbles and multi-perspective panoramas. We also present a dynamic construction of the panoramas which overcomes many of the limitations of previous systems. As the user slides sideways, the multi-perspective panorama is constructed and rendered dynamically to simulate either a perspective or hyper-perspective view. This provides a strong sense of parallax, which adds to the immersion. We call this form of sliding sideways while looking at a street façade a street slide. Finally we integrate annotations and a mini-map within the user interface to provide geographic information as well additional affordances for navigation. We demonstrate our Street Slide system on a series of intersecting streets in an urban setting. We report the results of a user study, which shows that visual searching is greatly enhanced with the Street Slide interface over existing systems from Google and Bing.

While onboard navigation systems are gaining in importance, maps are still the medium of choice for laying out a route to a destination and for way finding. However, even with a map, one is almost always more comfortable navigating a route the second time due to the visual memory of the route. To make the first time navigating a route feel more familiar, we present a system that integrates a map with a video automatically constructed from panoramic imagery captured at close intervals along the route. The routing information is used to create a variable speed video depicting the route. During playback of the video, the frame and field of view are dynamically modulated to highlight salient features along the route and connect them back to the map. A user interface is demonstrated to allow exploration of the combined map, video, and textual driving directions. We discuss the construction of the hybrid map and video interface. Finally, we report the results of a study that provides evidence of the effectiveness of such a system for route following. ACM Classification: H5.2 [Information interfaces and presentation]: User Interfaces.- Graphical user interfaces. General terms:

In this paper we present novel input devices that combine the standard capabilities of a computer mouse with multi-touch sensing. Our goal is to enrich traditional pointer-based desktop interactions with touch and gestures. To chart the design space, we present five different multi-touch mouse implementations. Each explores a different touch sensing strategy, which leads to differing form-factors and hence interactive possibilities. In addition to the detailed description of hardware and software implementations of our prototypes, we discuss the relative strengths, limitations and affordances of these novel input devices as informed by the results of a preliminary user study.

In this paper, we introduce a novel system for browsing, enhancing, and manipulating casual outdoor photographs by combining them with already existing georeferenced digital terrain and urban models. A simple interactive registration process is used to align a photograph with such a model. Once the photograph and the model have been registered, an abundance of information, such as depth, texture, and GIS data, becomes immediately available to our system. This information, in turn, enables a variety of operations, ranging from dehazing and relighting the photograph, to novel view synthesis, and overlaying with geographic information. We describe the implementation of a number of these applications and discuss possible extensions. Our results show that augmenting photographs with already available 3D models of the world supports a wide variety of new ways for us to experience and interact with our everyday snapshots.

Light fields can be used to represent an object's appearance with a high degree of realism. However, unlike their geometric counterparts, these image-based representations lack user control for manipulating them. We present a system that allows a user to interactively manipulate, composite and render multiple light fields. LightShop is a modular system consisting of three parts: 1) a set of functions that allow a user to model a scene containing multiple light fields, 2) a ray-shading language that describes how an image should be constructed from a set of light fields, and 3) a real-time light field rendering system in OpenGL that can plug into existing 3D engines as a GLSL shader.

We present a software pipeline that enables an animator to deform light fields. The pipeline can be used to deform complex objects, such as furry toys, while maintaining photo-realistic quality. Our pipeline consists of three stages. First, we split the light field into sub-light fields. To facilitate splitting of complex objects, we employ a novel technique based on projected light patterns. Second, we deform each sub-light field. To do this, we provide the animator with controls similar to volumetric free-form deformation. Third, we recombine and render each sub-light field. Our rendering technique properly handles visibility changes due to occlusion among sub-light fields. To ensure consistent illumination of objects after they have been deformed, our light fields are captured with the light source fixed to the camera, rather than being fixed to the object. We demonstrate our deformation pipeline using synthetic and photographically acquired light fields. Potential applications include animation, interior design, and interactive gaming.

We present a novel photographic technique called dual photography, which exploits Helmholtz reciprocity to interchange the lights and cameras in a scene. With a video projector providing structured illumination, reciprocity permits us to generate pictures from the viewpoint of the projector, even though no camera was present at that location. The technique is completely image-based, requiring no knowledge of scene geometry or surface properties, and by its nature automatically includes all transport paths, including shadows, inter-reflections and caustics. In its simplest form, the technique can be used to take photographs without a camera; we demonstrate this by capturing a photograph using a projector and a photo-resistor. If the photo-resistor is replaced by a camera, we can produce a 4D dataset that allows for relighting with 2D incident illumination. Using an array of cameras we can produce a 6D slice of the 8D reflectance field that allows for relighting with arbitrary light fields. Since an array of cameras can operate in parallel without interference, whereas an array of light sources cannot, dual photography is fundamentally a more efficient way to capture such a 6D dataset than a system based on multiple projectors and one camera. As an example, we show how dual photography can be used to capture and relight scenes.

Confocal microscopy is a family of imaging techniques that employ focused patterned illumination and synchronized imaging to create cross-sectional views of 3D biological specimens. In this paper, we adapt confocal imaging to large-scale scenes by replacing the optical apertures used in microscopy with arrays of real or virtual video projectors and cameras. Our prototype implementation uses a video projector, a camera, and an array of mirrors. Using this implementation, we explore confocal imaging of partially occluded environments, such as foliage, and weakly scattering environments, such as murky water. We demonstrate the ability to selectively image any plane in a partially occluded environment, and to see further through murky water than is otherwise possible. By thresholding the confocal images, we extract mattes that can be used to selectively illuminate any plane in the scene.

We consider collaborative control systems, where multiple sources share control of a single robot. These sources could come from multiple sensors (sensor fusion), multiple control processes (subsumption), or multiple human operators. Reports suggest that such systems are highly fault tolerant, even with large numbers of sources. We develop a formal model, modeling sources with finite automata. A collaborative ensemble of sources generates a single stream of incremental steps to control the motion of a point robot moving in the plane. We first analyze system performance with a uniform ensemble of well-behaved deterministic sources. We then model malfunctioning sources that go silent or generate inverted control signals. We discover that performance initially improves in the presence of malfunctioning sources and remains robust even when a sizeable fraction of sources malfunction. Initial tests suggest similar results with non-deterministic (random) sources. The formal model may also provide insight into how humans can share control of an online robot.

We describe a system that allows a distributed group of users to simultaneously teleoperate an industrial robot arm via the Internet. A Java applet at each client streams mouse motion vectors from up to 3D users; a sewer aggregates these inputs to produce a single control stream for the robot. Users receive visual feedback from a digital camera mounted above the robot arm. To our knowledge, this is the first collaboratively controlled robot on the Internet.