We present an optimization framework for exploring gradient-domain solutions for image and video processing. The proposed framework unifies many of the key ideas in the gradient-domain literature under a single optimization formulation. Our hope is that this generalized framework will allow the reader to quickly gain a general understanding of the field and contribute new ideas of their own. We propose a novel metric for measuring local gradient saliency that identifies salient gradients that give rise to long, coherent edges, even when the individual gradients are faint. We present a general weighting scheme for gradient constraints that improves the visual appearance of results. We also provide a solution for applying gradient-domain filters to videos and video streams in a coherent manner. Finally, we demonstrate the utility of our formulation in creating effective yet simple to implement solutions for various image-processing tasks. To exercise our formulation we have created a new saliency-based sharpen filter and a pseudo image-relighting application. We also revisit and improve upon previously defined filters such as nonphotorealistic rendering, image deblocking, and sparse data interpolation over images (e.g., colorization using optimization).

We present a technique for computing a dense pixel correspondence between two images of a scene containing multiple large, rigid motions. We model each motion with either a homography (for planar objects) or a fundamental matrix. The various motions in the scene are first extracted by clustering an initial sparse set of correspondences between feature points; we then perform a multi-label graph cut optimization which assigns each pixel to an independent motion and computes its disparity with respect to that motion. We demonstrate our technique on several example scenes and compare our results with previous approaches.

We present an interactive system for efficiently extracting foreground objects from a video. We extend previous min-cut based image segmentation techniques to the domain of video with four new contributions. We provide a novel painting-based user interface that allows users to easily indicate the foreground object across space and time. We introduce a hierarchical mean-shift preprocess in order to minimize the number of nodes that min-cut must operate on. Within the min-cut we also define new local cost functions to augment the global costs defined in earlier work. Finally, we extend 2D alpha matting methods designed for images to work with 3D video volumes. We demonstrate that our matting approach preserves smoothness across both space and time. Our interactive video cutout system allows users to quickly extract foreground objects from video sequences for use in a variety of applications including compositing onto new backgrounds and NPR cartoon style rendering.