Near-eye multi-focal (light field) displays are known to be more advantageous than conventional stereoscopic 3D displays, because they can alleviate the vergence-accommodation conflict. While stereoscopic 3D displays are restricted to projecting images on a single focal plane, near-eye light field displays can form multiple focal planes and volumetrically render 3D data. Most existing near-eye light field displays use simplistic, uniformly spaced focal plane configuration (in dioptric space). In this paper, we present a novel technique that optimizes the focal plane configuration based on characteristics of the content to be rendered. We show that this technique can significantly improve the perceived visual quality of content visualized on such displays.

We present a gonio-plenoptic imaging system that realizes a single-shot shape measurement for specular surfaces. The system is comprised of a collimated illumination source and a plenoptic camera. Unlike a conventional plenoptic camera, our system captures the BRDF variation of the object surface in a single image in addition to the light field information from the scene, which allows us to recover very fine 3D structures of the surface. The shape of the surface is reconstructed based on the reflectance property of the material rather than the parallax between different views. Since only a single-shot is required to reconstruct the whole surface, our system is able to capture dynamic surface deformation in a video mode. We also describe a novel calibration technique that maps the light field viewing directions from the object space to subpixels on the sensor plane. The proposed system is evaluated using a concave mirror with known curvature, and is compared to a parabolic mirror scanning system as well as a multi-illumination photometric stereo approach based on simulations and experiments.

Spectrally-coded plenoptic cameras have been developed for single-snapshot spectral imaging. Whereas reconstruction of spectral images from plenoptic sensor data has been studied in the past, rectification of those images according to a depth distribution in the scene has not been investigated. In this paper we describe a spectral parallax rectification algorithm for plenoptic cameras. The algorithm is designed to handle flexible filter partitions in the aperture and the small baselines inherent to plenoptic cameras. We demonstrate the algorithm's performance using checkerboard and captured scene data.

We present a study of depth-disparity calibration for augmented reality applications using binocular optical see-through displays. Two techniques were proposed and compared. The "paired-eyes" technique leverages the Panum's fusional area to help viewer find alignment between the virtual and physical objects. The "separate-eyes" technique eliminates the need of binocular fusion and involves using both eyes sequentially to check the virtual-physical object alignment on retinal images. We conducted a user study to measure the calibration results and assess the subjective experience of users with the proposed techniques.

We consider the calibration problem in plenoptic systems with the goal to estimate the system matrix given a set of known calibration patterns and their corresponding observations. The proposed calibration method is divided into a subspace identification step and a parameter estimation step. Exploiting newly discovered properties of the simulated system matrix, such as its low rank and the equality of its row and column spaces, we propose to identify the requisite subspace by computing the dominant left singular vectors of the observation matrix. Subsequently, we project each row of the observation matrix onto the subspace determined by these singular vectors. Our simulation results show that our method is able to accurately estimate the system matrix and significantly outperform a state-of-the-art low-rank recovery algorithm.

We present a method for 3D keypoint detection from light field data, typically obtained by planar camera arrays or plenoptic cameras. The proposed approach is based on construction of novel light field scale-depth spaces that are designed to leverage the specific properties of light fields. The constructed scale-depth spaces are based on a modified Gaussian kernel that is parametrized both in terms of scale of objects recorded by the light field and in terms of objects' depth. We prove theoretically that the new scale-depth space formulation and its spatial derivative satisfy the scale invariance property for all depths. By finding local extrema in such scale-depth spaces we locate 3D keypoints (such as 3D edges) and show that they outperform SURF keypoints on a 3D structure estimation task, both in accuracy and computational efficiency.

We propose a method for object reconstruction from images obtained by a plenoptic camera. Our approach exploits a plenoptic system model based on diffraction analysis in order to formulate an inverse problem for object reconstruction. To solve this inverse problem, we propose a dictionary learning algorithm for signal reconstruction from measurements obtained by a deterministic linear system. In contrast to prior work in Compressive Sensing, we do not impose constraints on the measurement matrix, but allow it to be defined by the properties of a specified camera system. Given the measurement matrix, the proposed algorithm learns a dictionary from a large database of examples and simultaneously minimizes the mutual coherence between the measurement matrix and the dictionary. We evaluate the performance of the algorithm on object reconstruction from plenoptic system measurements and show that it outperforms existing solutions.

We present a new method for estimating 3D scene layers from light field data obtained by plenoptic cameras or camera arrays. The proposed method is based on a novel sparse generative model for light fields, which uses a dictionary of ray-like functions and combines them in a non-linear way via a set of occlusion masks. By estimating the set of sparse coefficients and masks, our method divides the light field into layers corresponding to different depths in a 3D scene, while taking occlusions into account. The proposed method can thus be used for 3D scene segmentation and in a myriad of inverse problems in light field imaging, such as view interpolation, denoising, inpainting and super-resolution.

It has been shown that electro-optical imaging systems designed by integrating optics and digital processing provide system-level advantages such as extended depth-of-field and lower optical component costs. In such imaging systems, the strength of the optical aberration or blur can dramatically change with the f-number, and hence different digital filters in the subsequent digital processing are required to correct the captured images at different f-numbers. However, implementing such number of filters in hardware requires expensive computational resources, which in turn increase the overall system cost. In this paper, we propose a simple image filtering approach which uses a weighted sum of a set of component finite impulse response (FIR) filters to effectively apply a different composite FIR filter for each f-number. The simulation results demonstrate that our approach achieves desirable image quality for variable f-numbers, while substantially reducing the complexity in hardware implementation.

Small displays on mobile handheld devices, such as personal digital assistants (PDAs) and cellular phones, are the bottlenecks for usability of most content browsing applications. Generally, conventional content such as documents and Web pages need to be modified for effective presentation on mobile devices. This paper proposes a novel visualization for documents, called multimedia thumbnails, which consists of text and image content converted into playable multimedia clips. A multimedia thumbnail utilizes visual and audio channels of small portable devices as well as both spatial and time dimensions to communicate text and image information of a single document. The proposed algorithm for generating multimedia thumbnails includes 1) a semantic document analysis step, where salient content from a source document is extracted; 2) an optimization step, where a subset of this extracted content is selected based on time, display, and application constraints; and 3) a composition step, where the selected visual and audible document content is combined into a multimedia thumbnail. Scalability of MMNails that allows generation of multimedia clips of various lengths is also described. A user study is presented that evaluates the effectiveness of the proposed multimedia thumbnail visualization.

This paper introduces a new visual representation of a document or group of documents, a dynamic document icon, or Dydocon. Its representation is symbolic like an icon, but changes depending on document content. A Dydocon can be used for multiple documents and thus is useful for presentation of clustered search results. Because a large number of clusters may be returned by a search it is desirable to optimally select clusters given available display space. We cluster documents into a visual taxonomy described by a tree of Dydocons, define tree functionals for icon size and distortion of a cluster, and perform display-size adaptation through optimization using a BFOS-style algorithm.

It is difficult to view multipage, high resolution documents on devices with small displays. As a solution, we introduce a multimedia thumbnail representation, which can be seen as a multimedia clip that provides an automated guided tour through a document. Multimedia thumbnails are automatically generated by taking a document image as input and first performing visual and audible information analysis on the document to determine salient document elements. Next, the time and information attributes for each document element are computed by taking into account the display and application constraints. An optimization routine, given a time constraint, selects elements to be included in the multimedia thumbnail. Last, the selected elements are synthesized into animated images and audio to create the final multimedia representation

As small portable devices are becoming standard personal equipments, there is a great need for the adaptation of information content to small displays. Currently, no good solutions exist for viewing formatted documents, such as pdf documents, on these devices. Adapting content of web pages to small displays is usually achieved by complete redesign of a page or automatically reflowing text for small displays. Such techniques may not be applicable to documents whose format needs to be preserved. To address this problem, we propose a new document representation called Multimedia Thumbnail. Multimedia Thumbnail uses the visual and audio channels of small portable devices to communicate document information in form of a multimedia clip, which can be seen as a movie trailer for a document. Generation of such a clip includes a document analysis step, where salient document information is extracted, an optimization step, where the document information to be included in the thumbnail is determined based on display and time constraints, and a synthesis step, where visual and audible information are formed into a playable Multimedia Thumbnail. We also present user study results that evaluate an initial system design and point to further modification on analysis, optimization, and user interface components.

Browsing multi-page high-resolution documents on devices with small displays and limited navigational capabilities is very difficult. In this paper we demonstrate Multimedia Thumbnails, which are automatically computed guided-tours of documents. Multimedia Thumbnails utilize temporal dimension and the audio channel of the portable device in order to compensate for the limited resolution of the display area.

For practical image compression systems, handling images divided into independent tiles in the pixel domain is important. However, tiling introduces artifacts when transform coefficients are quantized The paper presents a "detiling" solution applicable to wavelet transform coefficients. The solution, called projection onto scaling functions, eliminates the blocking artifacts by computing approximations that match the smoothness of the compressed image inside a tile. This solution is applicable to any wavelet filter - extending previous work that applied only to a limited class of wavelet filters - and specifically targets decoding of JPEG2000 compressed images.

To bridge the mismatch between the sizes of images and display devices, we present an efficient and automatic algorithm to create an adaptive image representation called SmartNail. Given a digital image and rectangular display frame smaller than the image, we define the SmartNail as an appropriately cropped part of a suitably scaled-down image. We choose the SmartNail-defining parameters - down-scaling factor and cropping location - to maximize a bit-allocation-based cost function that quantifies the visual importance of the image content in the SmartNail. For JPEG2000-encoded images, the SmartNail parameters can be determined using just the header information available in the encoded file. Hence, only the wavelet coefficients required to reconstruct the SmartNail need to be decoded from the entire JPEG2000 code stream. Consequently, the SmartNail construction requires minimal computation and memory requirements. Simulations demonstrate the effectiveness of the SmartNail representations.

The problem of image enhancement arises in many applications such as scanners, copiers and digital cameras. Enhancement often includes a denoising and a deblurring or sharpening step. Similar to image compression, state-of-the-art denoising techniques use wavelet bases instead of Fourier bases since wavelet domain processing provides local adaptation in smooth and non-smooth parts due to the theoretical link between wavelets and smoothness spaces. In this paper the same smoothness spaces are used to propose a way of performing sharpening and smoothing of signals with wavelets (WSS) in Besov spaces. As an application the completely wavelet-based enhancement of a scanned document is discussed.

We propose an approximation of a continuous wavelet transform (CWT) that is based on a hierarchical scheme, similar to the fast discrete wavelet transform. This approximation keeps redundancies in time and scale. Furthermore, it preserves properties of the CWT regarding the characterization of singularities and leads to efficient applications in multiscale edge detection.