Image pipelines arise frequently in modern computational photography systems and consist of multiple processing stages where each stage produces an intermediate image that serves as input to a future stage. Inspired by recent work on loop perforation [Sidiroglou-Douskos et al. 2011], this article introduces image perforation , a new optimization technique that allows us to automatically explore the space of performance-accuracy tradeoffs within an image pipeline. Image perforation works by transforming loops over the image at each pipeline stage into coarser loops that effectively “skip” certain samples. These missing samples are reconstructed for later stages using a number of different interpolation strategies that are relatively inexpensive to perform compared to the original cost of computing the sample. We describe a genetic algorithm for automatically exploring the resulting combinatoric search space of which loops to perforate, in what manner, by how much, and using which reconstruction method. We also present a prototype language that implements image perforation along with several other domain-specific optimizations and show results for a number of different image pipelines and inputs. For these cases, image perforation achieves speedups of 2 × --10 × with acceptable loss in visual quality and significantly outperforms loop perforation.

Temporal coherence is one of the central challenges for rendering a stylized line. It is especially difficult for stylized contours of coarse meshes or non-uniformly sampled models, because those contours are polygonal feature edges on the models with no continuous correspondences between frames. We describe a novel and simple technique for constructing a 2D brush path along a 3D contour. We also introduce a 3D parameter propagation and re-parameterization procedure to construct stroke paths along the 2D brush path to draw coherently stylized feature lines with a wide range of styles. Our method runs in real-time for coarse or non-uniformly sampled models, making it suitable for interactive applications needing temporal coherence.

This paper presents a data structure that reduces approximate nearest neighbor query times for image patches in large datasets. Previous work in texture synthesis has demonstrated real-time synthesis from small exemplar textures. However, high performance has proved elusive for modern patch-based optimization techniques which frequently use many exemplar images in the tens of megapixels or above. Our new algorithm, PatchTable, offloads as much of the computation as possible to a pre-computation stage that takes modest time, so patch queries can be as efficient as possible. There are three key insights behind our algorithm: (1) a lookup table similar to locality sensitive hashing can be precomputed, and used to seed sufficiently good initial patch correspondences during querying, (2) missing entries in the table can be filled during pre-computation with our fast Voronoi transform, and (3) the initially seeded correspondences can be improved with a precomputed k-nearest neighbors mapping. We show experimentally that this accelerates the patch query operation by up to 9× over k-coherence, up to 12× over TreeCANN, and up to 200× over PatchMatch. Our fast algorithm allows us to explore efficient and practical imaging and computational photography applications. We show results for artistic video stylization, light field super-resolution, and multi-image editing.

This paper presents a method to render smooth stylized contours of 3D meshes with temporal coherence. Contour triangles that contain countour curves perform two functions: they ensure that smooth contour curves are coherent and reconstructed faithfully during the motion of the viewpoint and they help to establish the corresponding relationship of contours which can propagate parameters on contours between frames against sliding effects. Our method can also deal with zig-zags and short segments for coarse meshes and nonuniform sampling meshes in a wide range of styles very well.

We propose a real-time rendering system for automatically creating a caricature drawing, i.e., an exaggerated portrait, of a human face, based on simultaneous use of a range image (or 3D mesh) and a registered photograph of the same face. Combining these information sources provides complementary information. Significant geometric lines such as occluding contours and suggestive contours are extracted from the range data, while textured areas corresponding to shading features are extracted from the photograph. These are combined, and then distorted to produce the final caricature. The final output may be produced using a choice of non-photorealistic rendering styles. Our system method works well for low resolution range images; for these it is fast enough to allow the viewpoint to be chosen in real time. The final output combines significant lines, textured areas, and optional shading, giving a pleasing result which preserves not only the shape cues of the geometric description, but also other essential visual characteristics of the facial image that cannot be deduced from geometry alone.