The Google Cultural Institute Platform1is a large-scale system for ingesting, archiving, organizing, and interacting with digital assets of cultural material. This paper explains the components through which the platform contextualizes individual assets in order to enable storytelling. Contextualization is an inverse problem: given assets that are instances of cultural material, infer their precise context and use that as a way to support the storytelling process. The approach is based on three components: extraction, knowledge, and scale. Extraction is the inference of context from two sources of information: explicitly provided metadata, and automatically extracted features. Knowledge is the use of a large reference fact database for further contextualizing an asset based on its descriptors. And scale, achieved through global self-serve, enables massively expanded coverage of the knowledge database and crowdsource potential for metadata refinement. Together these components sustain a storytelling framework and a compelling user experience that has the potential to become the largest repository of cultural information and coherent narrative in history.

Traditional minimally invasive surgeries use a view port provided by an endoscope or laparoscope. We argue that a useful addition to typical endoscopic imagery would be a global 3-D view providing a wider field of view with explicit depth information for both the exterior and interior of target anatomy. One technical challenge of implementing such a view is finding efficient and accurate means of registering texture images from the laparoscope on prebuilt 3-D surface models of target anatomy derived from magnetic resonance (MR) or computed tomography (CT) images. This paper presents a novel method for addressing this challenge that differs from previous approaches, which depend on tracking the position of the laparoscope. We take advantage of the fact that neighboring frames within a video sequence usually contain enough coherence to allow a 2-D-2-D registration, which is a much more tractable problem. The texturing process can be bootstrapped by an initial 2-D-3-D user-assisted registration of the first video frame followed by mostly-automatic texturing of subsequent frames. We perform experiments on phantom and real data, validate the algorithm against the ground truth, and compare it with the traditional tracking method by simulations. Experiments show that our method improves registration performance compared to the traditional tracking approach.

We report on a display system to support laparoscopy by loosely coupling input sensors (e.g., cameras) with outputs (e.g., image displays). A low-latency and general-purpose computing platform is inserted between the traditionally tightly-coupled inputs and outputs of a laparoscopic surgical environment. The architecture of the proposed system provides flexibility in the fusion and augmentation of multiple input data sources and in the display of the data on a scalable large display. The performance of the described system is measured. The result indicates that our system has great potential to improve the performance of laparoscopic surgeons.

Creating uniform lighting for archival-quality document acquisition remains a non-trivial problem. We propose a novel method for automatic photometric correction of non-planar documents by estimating a single, point light-source using a simple light probe. By adding a simple piece of folded white paper with a known 3D surface to a scene, we are able to extract the 3D position of a light source, automatically perform white balance correction, and determine areas of poor illumination. Furthermore, this method is designed with the purpose of adding it to an already implemented document digitization pipeline. To justify our claims, we provide an accuracy analysis of our correction technique using simulated ground-truth data which allows individual sources of error to be determined and compared. These techniques are then applied on real documents that have been acquired using a 3D scanner.

This paper presents a framework to restore the 2D content printed on documents in the presence of geometric distortion and non-uniform illumination. Compared with textbased document imaging approaches that correct distortion to a level necessary to obtain sufficiently readable text or to facilitate optical character recognition (OCR), our work targets nontextual documents where the original printed content is desired. To achieve this goal, our framework acquires a 3D scan of the document's surface together with a high-resolution image. Conformal mapping is used to rectify geometric distortion by mapping the 3D surface back to a plane while minimizing angular distortion. This conformal "deskewing" assumes no parametric model of the document's surface and is suitable for arbitrary distortions. Illumination correction is performed by using the 3D shape to distinguish content gradient edges from illumination gradient edges in the high-resolution image. Integration is performed using only the content edges to obtain a reflectance image with significantly less illumination artifacts. This approach makes no assumptions about light sources and their positions. The results from the geometric and photometric correction are combined to produce the final output.

This paper introduces a method for building a readable image of an opaque, rolled or folded text from a volumetric, penetrating scan. The problem is framed by localizing, constructing, and manipulating an image induced by a surface embedded in a 3D voxel space. There are two central contributions that lead to the demonstrated results. First is an energy-based texture formation algorithm, which is a function of voxel intensities and the geometry of the embedded surface. Second is a regularization algorithm based on a constrained mapping of the embedded surface to a regularized image plane. The mapping preserves angles and lengths, which minimizes the distortion of text in the image. The experimental results show readable images derived from custom, high resolution (X-ray-based) CT scans of rolled papyrus and ink samples. These methods are significant for scholars seeking to study inaccessible texts, and may lead to viable techniques for scanning everyday opaque objects (books) without opening them.

We present a system to restore the 2D content printed on distorted documents. Our system works by acquiring a 3D scan of the document's surface together with a high-resolution image. Using the 3D surface information and the 2D image, we can ameliorate unwanted surface distortion and effects from non-uniform illumination. Our system can process arbitrary geometric distortions, not requiring any pre-assumed parametric models for the document's geometry. The illumination correction uses the 3D shape to distinguish content edges from illumination edges to recover the 2D content's reflectance image while making no assumptions about light sources and their positions. Results are shown for real objects, demonstrating a complete framework capable of restoring geometric and photometric artifacts on distorted documents

We present a framework for acquiring and restoring images of warped documents. The purpose of our restoration is to create a planar representation of a once planar document that has undergone an arbitrary and unknown rigid deformation. To accomplish this restoration, our framework acquires and flattens the 3D shape of a warped document to determine a nonlinear image transform that can correct for image distortion caused by the document's shape. Our framework is designed for use in library and museum digitization efforts where very old and badly damaged manuscripts are imaged.

Computer graphics and high-resolution digital imagery are becoming increasingly pervasive in communities not traditionally associated with graphics. Commodity graphics cards and digital cameras, along with powerful modeling software, allow organizations such as libraries, museums, and small businesses to produce expressive and realistic computer generated imagery, far surpassing the capabilities of current desktop monitors. What remains elusive to these groups is access to affordable and easy to use large format display technology. We present a practical system for deploying flexible projector based tiled displays. Our framework integrates two key components: (1) self-calibrating display geometry with real-time geometric correction and (2) PC-based distributed rendering that supports an established graphics API. Our system's display geometry is easy to configure and reconfigure, accommodates casually tiled projectors and arbitrary display surfaces, and can be operational in a matter of minutes. In addition, the underlying distributed rendering architecture (WireGL) is transparent to existing OpenGL applications, requiring no custom APIs or re-compilation of existing OpenGL executables. In short, we present a practical and flexible low-cost tiled display system that is simple to deploy and easy to operate.

We present a framework for restoring arbitrarily warped and deformed documents to their original planar shape. The impetus for this work is the need for tools and techniques to help digitally preserve and restore fragile manuscripts. Current digitization is performed under the assumption that the documents are flat, with subsequent image-processing and restoration algorithms either relying on this assumption or attempting to overcome it without shape information. Although most manuscripts were originally flat, many become deformed from damage and deterioration. Physical flattening is not possible without risking further, possibly irreversible, damage. Our framework addresses this restoration problem with two primary contributions. First, we present a working 3D digitization setup that acquires a 3D model with accurate shape-to-texture registration under multiple lighting conditions. Second, we show how the 3D model and a mass-spring particle system can be used together as a framework for digital flattening. We show that this restoration process can correct document deformations and can significantly improve subsequent document analysis.

The Rich Photo-Realistic Content (RPC) image-based model defined and used by ArchVision represents a photo-realistic scene as an ordered set of images. A desired view from the represented viewing directions can quickly be extracted for rendering and display. The benefit of the RPC is its photo-realistic quality. One weakness, however, is that there is no support for view interpolation between closely related basis views. This paper presents results from a tool for creating a new RPC-like image-based model, which detects and discards captured views that are determined to be faithfully and automatically interpolated from the remaining images in the RPC. We exploit the geometric constraints of the acquisition system and provide an automated matching algorithm, and interpolation scheme, and an error metric displayed in a graphical tool for monitoring the quality impact of discarding particular frames.

Front-projection display environments suffer from a fundamental problem: users and other objects in the environment can easily and inadvertently block projectors, creating shadows on the displayed image. We introduce a technique that detects and corrects transient shadows in a multi-projector display. Our approach is to minimize the difference between predicted (generated) and observed (camera) images by continuous modification of the projected image values for each display device. We speculate that the general predictive monitoring framework introduced here is capable of addressing more general radiometric consistency problems. Using an automatically-derived relative position of cameras and projectors in the display environment and a straightforward color correction scheme, the system renders an expected image for each camera location. Cameras observe the displayed image, which is compared with the expected image to detect shadowed regions. These regions are transformed to the appropriate projector frames, where corresponding pixel values are increased. In display regions where more than one projector contributes to the image, shadow regions are eliminated. We demonstrate an implementation of the technique in a multiprojector system.

This paper presents a new algorithm for immersive teleconferencing, which addresses the problem of registering and blending multiple images together to create a single seamless panorama. In the immersive teleconference paradigm, one frame of the teleconference is a panorama that is constructed from a compound-image sensing device. These frames are rendered at the remote site on a projection surface that surrounds the user, creating an immersive feeling of presence and participation in the teleconference. Our algorithm efficiently creates panoramic frames for a teleconference session that are both geometrically registered and intensity blended. We demonstrate a prototype that is able to capture images from a compound-image sensor, register them into a seamless panoramic frame, and render those panoramic frames on a projection surface at 30 frames per second.

Current camera-monitor teleconferencing applications produce unrealistic imagery and break any sense of presence for the participants. Other capture/display technologies can be used to provide more compelling teleconferencing. However, complex geometries in capture/display systems make producing geometrically correct imagery difficult. It is usually impractical to detect, model and compensate for all effects introduced by the capture/display system. Most applications simply ignore these issues and rely on the user acceptance of the camera-monitor paradigm.

Conventional projector-based display systems are typically designed around precise and regular configurations of projectors and display surfaces. While this results in rendering simplicity and speed, it also means painstaking construction and ongoing maintenance. In previously published work, we introduced a vision of projector-based displays constructed from a collection of casually-arranged projectors and display surfaces. In this paper, we present flexible yet practical methods for realizing this vision, enabling low-cost mega-pixel display systems with large physical dimensions, higher resolution, or both. The techniques afford new opportunities to build personal 3D visualization systems in offices, conference rooms, theaters, or even your living room. As a demonstration of the simplicity and effectiveness of the methods that we continue to perfect, we show in the included video that a 10-year old child can construct and calibrate a two-camera, two-projector, head-tracked display system, all in about 15 minutes.

A viewer-centered approach is presented to modeling the geometry of the visible occluding contour of solid 3-D shape. The rim appearance representation models the exact appearance of the occluding contour formed by the edges of a polyhedron. An algorithm is presented for constructing the rim appearance representation. Bounds on space and time are given, and implementation results show that the rim appearance representation is significantly smaller than both the aspect graph and the asp representation.>