Abstract When creating a High‐Dynamic‐Range (HDR) image from a sequence of differently exposed Low‐Dynamic‐Range (LDR) images, the set of LDR images is usually generated by sampling the space of exposure times with a geometric progression and without explicitly accounting for the distribution of irradiance values of the scene. We argue that this choice can produce sub‐optimal results both in terms of the number of acquired pictures and the quality of the resulting HDR image. This paper presents a method to estimate the full irradiance histogram of a scene, and a strategy to select the set of exposures that need to be acquired. Our selection usually requires a smaller or equal set of LDRs, yet produces higher quality HDR images.

All-in-focus imaging is a computational photography technique that produces images free of defocus blur by capturing a stack of images focused at different distances and merging them into a single sharp result. Current approaches assume that images have been captured offline, and that a reasonably powerful computer is available to process them. In contrast, we focus on the problem of how to capture such input stacks in an efficient and scene-adaptive fashion. Inspired by passive autofocus techniques, which select a single best plane of focus in the scene, we propose a method to automatically select a minimal set of images, focused at different depths, such that all objects in a given scene are in focus in at least one image. We aim to minimize both the amount of time spent metering the scene and capturing the images, and the total amount of high-resolution data that is captured. The algorithm first analyzes a set of low-resolution sharpness measurements of the scene while continuously varying the focus distance of the lens. From these measurements, we estimate the final lens positions required to capture all objects in the scene in acceptable focus. We demonstrate the use of our technique in a mobile computational photography scenario, where it is essential to minimize image capture time (as the camera is typically handheld) and processing time (as the computation and energy resources are limited).

The widespread availability of digital cameras and ubiquitous Internet access have facilitated the creation of massive image collections. These collections can be highly interconnected through implicit links between image pairs viewing the same or similar objects. We propose building graphs called Image Webs to represent such connections. While earlier efforts studied local neighborhoods of such graphs, we are interested in understanding global structure and exploiting connectivity at larger scales. We show how to efficiently construct Image Webs that capture the connectivity in an image collection using spectral graph theory. Our technique can link together tens of thousands of images in a few minutes using a computer cluster. We also demonstrate applications for exploring collections based on global topological analysis.

We present a novel approach to HDR (high-dynamic-range) image fusion that copes with image blur degradation often present in long-exposed images. The proposed approach can deal with both camera and object motion blur in a computationally efficient manner suitable for implementation on a mobile device. The main idea is to exploit the differences between the image degradations that affect images captured for HDR fusion. Short-exposed images are mainly affected by sensor noise and less affected by motion blur, whereas longer exposed images are less noisy but potentially blurry due to motion during their exposure. Our approach consists of two steps. First we calculate an HDR representation of the scene by applying a typical HDR fusion approach. This representation could be blurry if some of the input images with long exposure time are blurry. We then fuse the HDR result with a photometrically modified version of the image with the shortest exposure time, which allows us to retain the sharpness of the short-exposed image and the noise-free characteristics of the HDR-fused image. The method does not assume an invariant blur PSF over any of the input images, and hence it can solve both for local and global blur, due to object or camera motion, respectively. We demonstrate the algorithm through a series of experiments and comparisons on natural images.

Many natural scenes have a dynamic range that is larger than the dynamic range of a camera's image sensor. A popular approach to producing an image without under- and over-exposed areas is to capture several input images with varying exposure settings, and later merge them into a single high-quality result using offline image processing software. In this paper, we describe a system for creating images of high-dynamic-range (HDR) scenes that operates entirely on a mobile camera. Our system consists of an automatic HDR metering algorithm that determines which exposures to capture, a video-rate viewfinder preview algorithm that allows the user to verify the dynamic range that will be recorded, and a light-weight image merging algorithm that computes a high-quality result directly on the camera. By using our system, a photographer can capture, view, and share images of HDR scenes directly on the camera, without using offline image processing software.

Many natural scenes have a dynamic range that is larger than the dynamic range of a camera's image sensor. A popular approach to producing an image without under- and over-exposed areas is to capture several input images with varying exposure settings, and later merge them into a single high-quality result using offline image processing software. We present a system for creating images of high-dynamic-range (HDR) scenes that operates entirely on a mobile camera. Our system consists of an automatic HDR metering algorithm that determines which exposures to capture, a video-rate viewfinder preview algorithm that allows the user to verify the dynamic range that will be recorded, and a light-weight image merging algorithm that computes a high-quality result directly on the camera. By using our system, a photographer can capture, view, and share images of HDR scenes directly on camera, without using offline image processing software

Abstract Recent proliferation of camera phones, photo sharing and social network services has significantly changed how we process our photos. Instead of going through the traditional download‐edit‐share cycle using desktop editors, an increasing number of photos are taken with camera phones and published through cellular networks. The immediacy of the sharing process means that on‐device image editing, if needed, should be quick and intuitive. However, due to the limited computational resources and vastly different user interaction model on small screens, most traditional local selection methods can not be directly adapted to mobile devices. To address this issue, we present TouchTone, a new method for edge‐aware image adjustment using simple finger gestures. Our method enables users to select regions within the image and adjust their corresponding photographic attributes simultaneously through a simple point‐and‐swipe interaction. To enable fast interaction, we develop a memory‐ and computation‐efficient algorithm which samples a collection of 1D paths from the image, computes the adjustment solution along these paths, and interpolates the solutions to entire image through bilateral filtering. Our system is intuitive to use, and can support several local editing tasks, such as brightness, contrast, and color balance adjustments, within a minute on a mobile device.

Although there has been much interest in computational photography within the research and photography communities, progress has been hampered by the lack of a portable, programmable camera with sufficient image quality and computing power. To address this problem, we have designed and implemented an open architecture and API for such cameras: the Frankencamera. It consists of a base hardware specification, a software stack based on Linux, and an API for C++. Our architecture permits control and synchronization of the sensor and image processing pipeline at the microsecond time scale, as well as the ability to incorporate and synchronize external hardware like lenses and flashes. This paper specifies our architecture and API, and it describes two reference implementations we have built. Using these implementations we demonstrate six computational photography applications: HDR viewfinding and capture, low-light viewfinding and capture, automated acquisition of extended dynamic range panoramas, foveal imaging, IMU-based hand shake detection, and rephotography. Our goal is to standardize the architecture and distribute Frankencameras to researchers and students, as a step towards creating a community of photographer-programmers who develop algorithms, applications, and hardware for computational cameras.

We present an efficient algorithm for continuous image recognition and feature descriptor tracking in video which operates by reducing the search space of possible interest points inside of the scale space image pyramid. Instead of performing tracking in 2D images, we search and match candidate features in local neighborhoods inside the 3D image pyramid without computing their feature descriptors. The candidates are further validated by fitting to a motion model. The resulting tracked interest points are more repeatable and resilient to noise, and descriptor computation becomes much more efficient because only those areas of the image pyramid that contain features are searched. We demonstrate our method on real-time object recognition and label augmentation running on a mobile device.

We present a novel data-driven algorithm that leverages online image repositories such as Flickr for automatically generating tourist maps. Our hypothesis is that, given a large enough dataset of images with geo-based metadata, clusters of matching images from that dataset tend to provide reliable cues as to what the popular tourist spots may be. Our algorithm takes the geographical area of interest as input and retrieves geotagged photos from online photo collections. By clustering the photos based on their locations and identifying the popular tags for each cluster, our algorithm generates a set of points of interest (POIs) for the area. After retrieving additional photos based on these discovered POI tags, we use image matching to find the most representative landmark view for each POI. Finally, we remove clutter from the representative image and apply tooning to generate a map icon for each landmark.

We propose a method for accelerating a broad class of non-linear filters that includes the bilateral, non-local means, and other related filters. These filters can all be expressed in a similar way: First, assign each value to be filtered a position in some vector space. Then, replace every value with a weighted linear combination of all values, with weights determined by a Gaussian function of distance between the positions. If the values are pixel colors and the positions are ( x, y ) coordinates, this describes a Gaussian blur. If the positions are instead ( x, y, r, g, b ) coordinates in a five-dimensional space-color volume, this describes a bilateral filter. If we instead set the positions to local patches of color around the associated pixel, this describes non-local means. We describe a Monte-Carlo kd-tree sampling algorithm that efficiently computes any filter that can be expressed in this way, along with a GPU implementation of this technique. We use this algorithm to implement an accelerated bilateral filter that respects full 3D color distance; accelerated non-local means on single images, volumes, and unaligned bursts of images for denoising; and a fast adaptation of non-local means to geometry. If we have n values to filter, and each is assigned a position in a d -dimensional space, then our space complexity is O(dn ) and our time complexity is O(dn log n ), whereas existing methods are typically either exponential in d or quadratic in n .

Mobile phones are an attractive platform for landmark-based pedestrian navigation systems. To be practical, such a system must be able to automatically generate lightweight directions that can be displayed on these mobile devices. We present a system that leverages an online collection of geotagged photographs to automatically generate navigational instructions. These are presented to the user as a sequence of images of landmarks augmented with directional instructions. Both the landmark selection and image augmentation are done automatically. We present a user study that indicates these generated directions are beneficial to users and suggest areas for future improvement.

Abstract The viewfinder of a digital camera has traditionally been used for one purpose: to display to the user a preview of what is seen through the camera's lens. High quality cameras are now available on devices such as mobile phones and PDAs, which provide a platform where the camera is a programmable device, enabling applications such as online computational photography, computer vision‐based interactive gaming, and augmented reality. For such online applications, the camera viewfinder provides the user's main interaction with the environment. In this paper, we describe an algorithm for aligning successive viewfinder frames. First, an estimate of inter‐frame translation is computed by aligning integral projections of edges in two images. The estimate is then refined to compute a full 2D similarity transformation by aligning point features. Our algorithm is robust to noise, never requires storing more than one viewfinder frame in memory, and runs at 30 frames per second on standard smartphone hardware. We use viewfinder alignment for panorama capture, low‐light photography, and a camera‐based game controller.

Recent convergence of imaging sensors and general purpose processors on mobile phones creates an opportunity for a new class of augmented reality applications. Robust image feature extraction is a crucial enabler of this type of systems. In this article, we discuss an efficient mobile phone implementation of a state-of-the-art algorithm for computing robust image features called SURF. We implement several improvements to the basic algorithm that significantly improve its performance and reduce its memory footprint making the use of this algorithm on the mobile phone practical. Our prototype implementation has been applied to several practical applications such as image search, object recognition and augmented reality applications.

We present a system for automatic reassembly of broken 3D solids. Given as input 3D digital models of the broken fragments, we analyze the geometry of the fracture surfaces to find a globally consistent reconstruction of the original object. Our reconstruction pipeline consists of a graph-cuts based segmentation algorithm for identifying potential fracture surfaces, feature-based robust global registration for pairwise matching of fragments, and simultaneous constrained local registration of multiple fragments. We develop several new techniques in the area of geometry processing, including the novel integral invariants for computing multi-scale surface characteristics, registration based on forward search techniques and surface consistency, and a non-penetrating iterated closest point algorithm. We illustrate the performance of our algorithms on a number of real-world examples.

We propose a framework for pairwise registration of shapes represented by point cloud data (PCD). We assume that the points are sampled from a surface and formulate the problem of aligning two PCDs as a minimization of the squared distance between the underlying surfaces. Local quadratic approximants of the squared distance function are used to develop a linear system whose solution gives the best aligning rigid transform for the given pair of point clouds. The rigid transform is applied and the linear system corresponding to the new orientation is build. This process is iterated until it converges. The point-to-point and the point-to-plane Iterated Closest Point (ICP) algorithms can be treated as special cases in this framework. Our algorithm can align PCDs even when they are placed far apart, and is experimentally found to be more stable than point-to-plane ICP. We analyze the convergence behavior of our algorithm and of point-to-point and point-to-plane ICP under our proposed framework, and derive bounds on their rate of convergence. We compare the stability and convergence properties of our algorithm with other registration algorithms on a variety of scanned data.

In this paper we present an approach for teaching the Freshman-Sophomore introduction to data structures course (CS2) in a way that provides an introduction to object-oriented software engineering patterns in addition to the theory of data structures. We survey in this paper several design patterns and describe how they can be naturally integrated in the CS2 curriculum.