Novel view synthesis is a challenging and ill-posed inverse rendering problem. Neural rendering techniques have recently achieved photorealistic image quality for this task. State-of-the-art (SOTA) neural volume rendering approaches, however, are slow to train and require minutes of inference (i.e., rendering) time for high image resolutions. We adopt high-capacity neural scene representations with periodic activations for jointly optimizing an implicit surface and a radiance field of a scene supervised exclusively with posed 2D images. Our neural rendering pipeline accelerates SOTA neural volume rendering by about two orders of magnitude and our implicit surface representation is unique in allowing us to export a mesh with view-dependent texture information. Thus, like other implicit surface representations, ours is compatible with traditional graphics pipelines, enabling real-time rendering rates, while achieving unprecedented image quality compared to other surface methods. We assess the quality of our approach using existing datasets as well as high-quality 3D face data captured with a custom multi-camera rig.

Abstract Collaborative filtering collects similar patches, jointly filters them and scatters the output back to input patches; each pixel gets a contribution from each patch that overlaps with it, allowing signal reconstruction from highly corrupted data. Exploiting self‐similarity, however, requires finding matching image patches, which is an expensive operation. We propose a GPU‐friendly approximated‐nearest‐neighbour(ANN) algorithm that produces high‐quality results for any type of collaborative filter. We evaluate our ANN search against state‐of‐the‐art ANN algorithms in several application domains. Our method is orders of magnitudes faster, yet provides similar or higher quality results than the previous work.

The human visual system can operate in a wide range of illumination levels due to several adaptation processes working in concert. For the most part, these adaptation mechanisms are transparent, leaving the observer unaware of his or her absolute adaptation state. At extreme illumination levels, however, some of these mechanisms produce perceivable secondary effects, or epiphenomena. In bright light, these include bleaching afterimages and adaptation afterimages, while in dark conditions these include desaturation, loss of acuity, mesopic hue shift, and the Purkinje effect. In this work we examine whether displaying these effects explicitly can be used to extend the apparent dynamic range of a conventional computer display. We present phenomenological models for each effect, describe efficient computer graphics methods for rendering our models, and propose a gaze-adaptive display that injects the effects into imagery on a standard computer monitor. Finally, we report the results of psychophysical experiments which reveal that, while mesopic epiphenomena are a strong cue that a stimulus is very dark, afterimages have little impact on the perception that a stimulus is very bright.

Obtaining a good baseline between different video frames is one of the key elements in vision-based monocular SLAM systems. However, if the video frames contain only a few 2D feature correspondences with a good baseline, or the camera only rotates without sufficient translation in the beginning, tracking and mapping becomes unstable. We introduce a real-time visual SLAM system that incrementally tracks individual 2D features, and estimates camera pose by using matched 2D features, regardless of the length of the baseline. Triangulating 2D features into 3D points is deferred until key frames with sufficient baseline for the features are available. Our method can also deal with pure rotational motions, and fuse the two types of measurements in a bundle adjustment step. Adaptive criteria for key frame selection are also introduced for efficient optimization and dealing with multiple maps. We demonstrate that our SLAM system improves camera pose estimates and robustness, even with purely rotational motions.

Abstract Conventional depth video compression uses video codecs designed for color images. Given the performance of current encoding standards, this solution seems efficient. However, such an approach suffers from many issues stemming from discrepancies between depth and light perception. To exploit the inherent limitations of human depth perception, we propose a novel depth compression method that employs a disparity perception model. In contrast to previous methods, we account for disparity masking, and model a distinct relation between depth perception and contrast in luminance. Our solution is a natural extension to the H.264 codec and can easily be integrated into existing decoders. It significantly improves both the compression efficiency without sacrificing visual quality of depth of rendered content, and the output of depth‐reconstruction algorithms or depth cameras.

Conventional pipelines for capturing, displaying, and storing images are usually defined as a series of cascaded modules, each responsible for addressing a particular problem. While this divide-and-conquer approach offers many benefits, it also introduces a cumulative error, as each step in the pipeline only considers the output of the previous step, not the original sensor data. We propose an end-to-end system that is aware of the camera and image model, enforces natural-image priors, while jointly accounting for common image processing steps like demosaicking, denoising, deconvolution, and so forth, all directly in a given output representation (e.g., YUV, DCT). Our system is flexible and we demonstrate it on regular Bayer images as well as images from custom sensors. In all cases, we achieve large improvements in image quality and signal reconstruction compared to state-of-the-art techniques. Finally, we show that our approach is capable of very efficiently handling high-resolution images, making even mobile implementations feasible.

We demonstrate that layered spatial light modulators (SLMs), subject to fixed lateral displacements and refreshed at staggered intervals, can synthesize images with greater spatiotemporal resolution than that afforded by any single SLM used in their construction. Dubbed cascaded displays , such architectures enable superresolution flat panel displays (e.g., using thin stacks of liquid crystal displays (LCDs)) and digital projectors (e.g., relaying the image of one SLM onto another). We introduce a comprehensive optimization framework, leveraging non-negative matrix and tensor factorization, that decomposes target images and videos into multi-layered, time-multiplexed attenuation patterns---offering a flexible trade-off between apparent image brightness, spatial resolution, and refresh rate. Through this analysis, we develop a real-time dual-layer factorization method that quadruples spatial resolution and doubles refresh rate. Compared to prior superresolution displays, cascaded displays place fewer restrictions on the hardware, offering thin designs without moving parts or the necessity of temporal multiplexing. Furthermore, cascaded displays are the first use of multi-layer displays to increase apparent temporal resolution. We validate these concepts using two custom-built prototypes: a dual-layer LCD and a dual-modulation liquid crystal on silicon (LCoS) projector, with the former emphasizing head-mounted display (HMD) applications.

We present a novel method for aligning images in an HDR (high-dynamic-range) image stack to produce a new exposure stack where all the images are aligned and appear as if they were taken simultaneously, even in the case of highly dynamic scenes. Our method produces plausible results even where the image used as a reference is either too dark or bright to allow for an accurate registration.

We introduce an energy efficient time-sharing pyramid pipeline architecture designed for multi-resolution image analysis in mobile computer vision. The time-sharing pipeline efficiently reduces the off-chip memory traffic by re-organizing the data storage and processing order of an image pyramid. We build a parameterized image pyramid hardware generator and successfully evaluate the overall pyramid design space. Our results demonstrate that the time-sharing pyramid pipeline achieves about 50% of hardware savings in terms of area and power consumption compared to the traditional linear pipeline implementation. We also implement the multi-resolution Lucas-Kanade optical flow algorithm on the time-sharing pipeline, and demonstrate an order of magnitude savings in off-chip memory traffic and system energy consumption.

Digital cameras with electronic viewfinders provide a relatively faithful depiction of the final image, providing a WYSIWYG experience. If, however, the image is created from a burst of differently captured images, or non-linear interactive edits significantly alter the final outcome, then the photographer cannot directly see the results, but instead must imagine the post-processing effects. This paper explores the notion of viewfinder editing , which makes the viewfinder more accurately reflect the final image the user intends to create. We allow the user to alter the local or global appearance (tone, color, saturation, or focus) via stroke-based input, and propagate the edits spatiotemporally. The system then delivers a real-time visualization of these modifications to the user, and drives the camera control routines to select better capture parameters.

We propose a new stereo technique using a pair of flash and no-flash stereo images that is both efficient and robust in handling occlusion boundaries. Our work is motivated by the observation that the brightness variations introduced by the flash can provide a robust cue for establishing stereo matches at occlusion boundaries. This photometric cue is computed per pixel, and though on its own is not robust to reliably resolve depth, it can provide a new discriminant to support patch-based stereo matching algorithms. Our experiments using a hand-held Fujifilm W3 3D camera show satisfying stereo performance over a variety of scenes, including several outdoor scenes.

Abstract When creating a High‐Dynamic‐Range (HDR) image from a sequence of differently exposed Low‐Dynamic‐Range (LDR) images, the set of LDR images is usually generated by sampling the space of exposure times with a geometric progression and without explicitly accounting for the distribution of irradiance values of the scene. We argue that this choice can produce sub‐optimal results both in terms of the number of acquired pictures and the quality of the resulting HDR image. This paper presents a method to estimate the full irradiance histogram of a scene, and a strategy to select the set of exposures that need to be acquired. Our selection usually requires a smaller or equal set of LDRs, yet produces higher quality HDR images.

This course is designed to demonstrate the use of the FCam, an Application Programming Interface (API) which allows for easy and precise control of camera systems. It lets programmers implement advanced computational photography applications. Such applications even allow the modification of basic camera routines such as metering, which are hidden inside black boxes in most camera systems. FCam is also an excellent tool for teaching and research on interactive mobile computational photography. The API has been open-sourced, and is currently implemented for Nokia N900 and an experimental Frankencamera (the F2). The Frankencamera architecture for computational cameras was presented at SIGGRAPH 2010. By end-2011, the FCam will hopefully be available on Nokia™ Linux phones, NVIDIA™ Tegra 3 development boards, and F3, the next version of the Stanford experimental camera. The F3 will be distributed to research universities within the US along with courseware in a National Science Foundation (NSF)-sponsored project. NVIDIA Tegras and Nokia™ Linux phones are available globally.

The popularity of mobile interfaces and application development is increasing along with the rapid expansion of the mobile electronics market and its migration from text-based applications to various multimedia applications. Real-time graphics and web applications are becoming one of the most attractive applications in mobile terminals due to their benefits for enterprise, gaming, and social media. This hands-on course will cover a comprehensive set of topics for developing mobile visual interfaces, including an overview of the mobile market, a comparison of mobile and desktop applications, and a survey of mobile development environments. We will also undertake a detailed discussion of UI development for mobiles, and graphics development for mobiles. During the course, various smartphones will be loaned to attendees for trying out several in-class exercises.

All-in-focus imaging is a computational photography technique that produces images free of defocus blur by capturing a stack of images focused at different distances and merging them into a single sharp result. Current approaches assume that images have been captured offline, and that a reasonably powerful computer is available to process them. In contrast, we focus on the problem of how to capture such input stacks in an efficient and scene-adaptive fashion. Inspired by passive autofocus techniques, which select a single best plane of focus in the scene, we propose a method to automatically select a minimal set of images, focused at different depths, such that all objects in a given scene are in focus in at least one image. We aim to minimize both the amount of time spent metering the scene and capturing the images, and the total amount of high-resolution data that is captured. The algorithm first analyzes a set of low-resolution sharpness measurements of the scene while continuously varying the focus distance of the lens. From these measurements, we estimate the final lens positions required to capture all objects in the scene in acceptable focus. We demonstrate the use of our technique in a mobile computational photography scenario, where it is essential to minimize image capture time (as the camera is typically handheld) and processing time (as the computation and energy resources are limited).

The 17 papers in this special issue focus on resource-limited systems.

We present a fast and high-quality image stitching approach for creating high-resolution and high-quality panoramic images on mobile devices. In the approach, a sequential panorama stitching procedure is created with color and luminance compensation for reducing differences between source images, fast image labeling for optimal seam finding, and gradient domain image blending for transition smoothing. The color and luminance compensation is performed in the gamma-corrected sRGB color space to avoid pixel saturation problems, optimal seams between adjacent source images are found by dynamic programming optimization for fast speed and less memory, and the source images are blended together by Poisson blending for a high blending quality. We compare results and performance with other approaches to demonstrate the advantages of our approach in mobile panorama applications. The main contribution of this work is studying how applying color and luminance compensation before Poisson blending improves the blending quality and processing speed of Poisson blending in long image sequences where colors and luminance vary between source images, and how it also improves the quality of image labeling. The integration of color correction, fast labeling, and Poisson blending into a sequential panorama stitching procedure allows creation of high-resolution panoramic images with large source images more quickly and using less memory. These are very important aspects for mobile devices that have limited computational power and memory resources.

We propose an effective image segmentation approach with a novel automatic boundary refinement procedure that requires little user interaction and makes the object cutout process more robust and convenient. It achieves these goals by the following three steps. First, merge over-segmented regions according to the maximal similarity rule using a few marking strokes as input. Second, detect possible erroneous low-contrast object boundaries by analyzing image content. Third, automatically refine those boundary regions using both local and global information. Experimental results are good even on very complex images.

We present a novel approach to HDR (high-dynamic-range) image fusion that copes with image blur degradation often present in long-exposed images. The proposed approach can deal with both camera and object motion blur in a computationally efficient manner suitable for implementation on a mobile device. The main idea is to exploit the differences between the image degradations that affect images captured for HDR fusion. Short-exposed images are mainly affected by sensor noise and less affected by motion blur, whereas longer exposed images are less noisy but potentially blurry due to motion during their exposure. Our approach consists of two steps. First we calculate an HDR representation of the scene by applying a typical HDR fusion approach. This representation could be blurry if some of the input images with long exposure time are blurry. We then fuse the HDR result with a photometrically modified version of the image with the shortest exposure time, which allows us to retain the sharpness of the short-exposed image and the noise-free characteristics of the HDR-fused image. The method does not assume an invariant blur PSF over any of the input images, and hence it can solve both for local and global blur, due to object or camera motion, respectively. We demonstrate the algorithm through a series of experiments and comparisons on natural images.

Many natural scenes have a dynamic range that is larger than the dynamic range of a camera's image sensor. A popular approach to producing an image without under- and over-exposed areas is to capture several input images with varying exposure settings, and later merge them into a single high-quality result using offline image processing software. In this paper, we describe a system for creating images of high-dynamic-range (HDR) scenes that operates entirely on a mobile camera. Our system consists of an automatic HDR metering algorithm that determines which exposures to capture, a video-rate viewfinder preview algorithm that allows the user to verify the dynamic range that will be recorded, and a light-weight image merging algorithm that computes a high-quality result directly on the camera. By using our system, a photographer can capture, view, and share images of HDR scenes directly on the camera, without using offline image processing software.

Many natural scenes have a dynamic range that is larger than the dynamic range of a camera's image sensor. A popular approach to producing an image without under- and over-exposed areas is to capture several input images with varying exposure settings, and later merge them into a single high-quality result using offline image processing software. We present a system for creating images of high-dynamic-range (HDR) scenes that operates entirely on a mobile camera. Our system consists of an automatic HDR metering algorithm that determines which exposures to capture, a video-rate viewfinder preview algorithm that allows the user to verify the dynamic range that will be recorded, and a light-weight image merging algorithm that computes a high-quality result directly on the camera. By using our system, a photographer can capture, view, and share images of HDR scenes directly on camera, without using offline image processing software

This paper addresses the problem of color and luminance compensation for sequences of overlapping images where the source images have very different colors and luminance. We apply the method for panoramic image construction on mobile phones. A simple approach is proposed that minimizes both the color differences of neighboring images and the overall color correction over the whole sequence. We compare several combinations of gamma correction and linear adjustment over different color representations, and select the method with best results: use YCbCr and apply gamma correction for the luminance component and linear correction for the chrominance components

We provide an efficient technique of color and luminance compensation for sequences of overlapping images. It can be used in construction of high-resolution and high-quality panoramic images even when the input images have very different colors and luminance. The technique uses color matching in the overlapping areas of source image pairs to balance colors and luminance in the whole image sequence. It performs gamma correction for the luminance component and linear correction for the chrominance components of source images in the image sequence. Compared to existing approaches, our technique is simple and efficient, yet it can avoid color saturation problems during color correction and perform color and luminance compensation globally in the whole image sequence. We apply the technique to create high-quality panoramic images on mobile phones

Although there has been much interest in computational photography within the research and photography communities, progress has been hampered by the lack of a portable, programmable camera with sufficient image quality and computing power. To address this problem, we have designed and implemented an open architecture and API for such cameras: the Frankencamera. It consists of a base hardware specification, a software stack based on Linux, and an API for C++. Our architecture permits control and synchronization of the sensor and image processing pipeline at the microsecond time scale, as well as the ability to incorporate and synchronize external hardware like lenses and flashes. This paper specifies our architecture and API, and it describes two reference implementations we have built. Using these implementations we demonstrate six computational photography applications: HDR viewfinding and capture, low-light viewfinding and capture, automated acquisition of extended dynamic range panoramas, foveal imaging, IMU-based hand shake detection, and rephotography. Our goal is to standardize the architecture and distribute Frankencameras to researchers and students, as a step towards creating a community of photographer-programmers who develop algorithms, applications, and hardware for computational cameras.

We present an efficient algorithm for continuous image recognition and feature descriptor tracking in video which operates by reducing the search space of possible interest points inside of the scale space image pyramid. Instead of performing tracking in 2D images, we search and match candidate features in local neighborhoods inside the 3D image pyramid without computing their feature descriptors. The candidates are further validated by fitting to a motion model. The resulting tracked interest points are more repeatable and resilient to noise, and descriptor computation becomes much more efficient because only those areas of the image pyramid that contain features are searched. We demonstrate our method on real-time object recognition and label augmentation running on a mobile device.

We present an image enhancement algorithm based on fusing the visual information present in two images of the same scene, captured with different exposure times. The main idea is to exploit the differences between the image degradations that affect the two images. On one hand the short-exposed image is less affected by motion blur, whereas the long-exposed image is less affected by noise. Different fusion rules are designed for the luminance and chrominance components such that to preserve the desirable properties from each input image. We also present a method for estimating the brightness transfer function between the input images for photometric calibration of the short-exposed image with respect to the long-exposed image. As no global blur PSF is assumed, our method can deal with blur from both camera and object motions. We demonstrate the algorithm by a series of experiments and simulations.

We present an image labeling approach for merging a set of aligned source images into a composite image by finding optimal seams in the overlapping areas of the source images quickly and using little memory. A minimal-cost path in the overlapping area of two images is found by dynamic programming and used as an optimal seam to label images. The overlapping images are cut along the seam and merged together. A sequential image stitching procedure is integrated with the fast image labeling for producing high-resolution and high-quality panoramic images using large source images under limited computational and memory resources. The approach presents several advantages: the use of dynamic programming optimization for finding the minimal-cost path over adjacent source images guarantees finding the optimal seam and allows images to be merged quickly; ghosting and blurring problems caused by moving objects and small registration errors can be avoided by the optimal seam finding process; the combination of the sequential image stitching procedure with the fast image labeling allows processing large source images for creating high-resolution panoramic images using little memory; the fast labeling process is easy to combine with intensive blending to produce high-quality panoramic images. The method is implemented in our mobile panorama system and runs with good performance on mobile devices.

Abstract The viewfinder of a digital camera has traditionally been used for one purpose: to display to the user a preview of what is seen through the camera's lens. High quality cameras are now available on devices such as mobile phones and PDAs, which provide a platform where the camera is a programmable device, enabling applications such as online computational photography, computer vision‐based interactive gaming, and augmented reality. For such online applications, the camera viewfinder provides the user's main interaction with the environment. In this paper, we describe an algorithm for aligning successive viewfinder frames. First, an estimate of inter‐frame translation is computed by aligning integral projections of edges in two images. The estimate is then refined to compute a full 2D similarity transformation by aligning point features. Our algorithm is robust to noise, never requires storing more than one viewfinder frame in memory, and runs at 30 frames per second on standard smartphone hardware. We use viewfinder alignment for panorama capture, low‐light photography, and a camera‐based game controller.

Enveloping, or the mapping of skeletal controls to the deformations of a surface, is key to driving realistic animated characters. Despite its widespread use, enveloping still relies on slow or inaccurate deformation methods. We propose a method that is both fast, accurate and example-based. Our technique introduces a rotational regression model that captures common skinning deformations such as muscle bulging, twisting, and challenging areas such as the shoulders. Our improved treatment of rotational quantities is made practical by model reduction that ensures real-time solution of least-squares problems, independent of the mesh size. Our method is significantly more accurate than linear blend skinning and almost as fast, suggesting its use as a replacement for linear blend skinning when examples are available.

The functionality of an information kiosk can be extended by allowing it to interact with a smartphone, as demonstrated by the Kimono system, and the user interface can be greatly simplified by "associations" between pieces of information. A kiosk provides information that is relevant to a particular location and can use valuable context information, such as the fact that a user is physically standing in front of the kiosk, to tailor the display. Its graphically rich screen is suitable for presenting information to the user and has a natural input modality requiring the user to simply touch the screen. However, a kiosk lacks mobility and cannot stay with the user as he or she moves about the environment. Also, information provided by the kiosk must be remembered by the user. Finally, it is difficult to add information to the kiosk, and so the kiosk remains an information display device.All this changes when a handset, such as a PDA or smart-phone, can interact with the kiosk. The handset acts like a personalized proxy of the kiosk. It accompanies the user serving as a memory device. It is also an excellent media creation device, capable of taking pictures and recording voice memos as well as short text messages. Associating newly created content with other currently selected content makes for a simpler user interface. Content and its associations can be uploaded to a kiosk allowing others to access to it.

Style translation is the process of transforming an input motion into a new style while preserving its original content. This problem is motivated by the needs of interactive applications, which require rapid processing of captured performances. Our solution learns to translate by analyzing differences between performances of the same content in input and output styles. It relies on a novel correspondence algorithm to align motions, and a linear time-invariant model to represent stylistic differences. Once the model is estimated with system identification, our system is capable of translating streaming input with simple linear operations at each frame.

This paper presents a complete system for scanning the geometry and surface color of a 3D object and for displaying realistic images of the object from arbitrary viewpoints. A stereo system with active light produces several views of dense range and color data. The data is registered and a surface that approximates the data is constructed. The surface estimate can be fairly coarse, as the appearance of fine detail is recreated by view-dependent texturing of the surface using color images.

We describe a model-based object recognition system that uses a probabilistic model for recognizing and locating objects. For each major view class of each 3D object, a probability model consisting of triplets of visible features, their parametrization, and their frequency of detection is constructed from a set of synthetic training images. These synthetic probability models are used to recognize and locate the 3D object from real 2D camera images. The features captured from the real images are then used to create a new, more accurate probability model.