Common wisdom holds that once personal content such as photographs have been shared on the Internet, they will stay there forever. This paper explores how we could allow users to reclaim some degree of their privacy by "revoking" previously shared photographs, hindering (but not eliminating) any subsequent viewing or sharing by others. Our goal is not to build a system that can withstand determined efforts to subvert it, but rather to give well-intentioned users the ability to respect the privacy wishes of others. Achieving this goal at scale will eventually require the participation of large content aggregators, and they are unlikely (putting it mildly) to find our proposal compelling. We therefore propose an approach we call technology ecosystem transformation (TET) that begins with a transitional and more easily deployable (but not fully scalable) design that does not require the participation of large incumbents but is designed to change user and societal expectations enough so that these companies would find it in their interest to adopt the approach we propose here. The intellectual challenge in this TET approach is finding transitional designs that (i) have parties willing to deploy it and (ii) once deployed, would change the incentives for the incumbents so that they would be willing to adopt the proposal.

Many aspects of JPEG compression have been successfully used in the domain of photo forensics. Adding to this literature, we describe a JPEG artifact that can arise depending upon seemingly innocuous implementation details in a JPEG encoder. We describe the nature of these artifacts and show how a generic JPEG encoder can be configured to explain a wide range of these artifacts found in real-world cameras. We also describe an algorithm to simultaneously estimate the nature of these artifacts and localize inconsistencies that can arise from a wide range of image manipulations.

We describe the acquisition of a large, diverse set of rebroadcast images captured by a screen-grab, scanning a printed photo, or rephotographing a displayed or a printed photo. This dataset consists of 14,500 rebroadcast images captured from a diverse set of devices: 234 displays, 173 scanners, 282 printers, and 180 recapture cameras. The diversity of this dataset-across devices and types of rebroadcast-poses significant challenges to detecting rebroadcast attacks. We evaluate the efficacy of four different classifiers trained to simultaneously detect all types of rebroadcast images.

Modern computer graphics are capable of generating highly photorealistic images. Although this can be considered a success for the computer graphics community, it has given rise to complex forensic and legal issues. A compelling example comes from the need to distinguish between computer-generated and photographic images as it pertains to the legality and prosecution of child pornography in the United States. We performed psychophysical experiments to determine the accuracy with which observers are capable of distinguishing computer-generated from photographic images. We find that observers have considerable difficulty performing this task—more difficulty than we observed 5 years ago when computer-generated imagery was not as photorealistic. We also find that observers are more likely to report that an image is photographic rather than computer generated, and that resolution has surprisingly little effect on performance. Finally, we find that a small amount of training greatly improves accuracy.

We describe a new forensic technique for distinguishing between computer generated and human faces in video. This technique identifies tiny fluctuations in the appearance of a face that result from changes in blood flow. Because these changes result from the human pulse, they are unlikely to be found in computer generated imagery. We use the absence or presence of this physiological signal to distinguish computer generated from human faces.

We describe a method for detecting physical inconsistencies in lighting from the shading and shadows in an image. This method imposes a multitude of shading- and shadow-based constraints on the projected location of a distant point light source. The consistency of a collection of such constraints is posed as a linear programming problem. A feasible solution indicates that the combination of shading and shadows is physically consistent, while a failure to find a solution provides evidence of photo tampering.

We describe a geometric technique to detect physically inconsistent arrangements of shadows in an image. This technique combines multiple constraints from cast and attached shadows to constrain the projected location of a point light source. The consistency of the shadows is posed as a linear programming problem. A feasible solution indicates that the collection of shadows is physically plausible, while a failure to find a solution provides evidence of photo tampering.

We describe a geometric technique to detect physically implausible trajectories of objects in video sequences. This technique explicitly models the three-dimensional ballistic motion of objects in free-flight and the two-dimensional projection of the trajectory into the image plane of a static or moving camera. Deviations from this model provide evidence of manipulation. The technique assumes that the object's trajectory is substantially influenced only by gravity, that the image of the object's center of mass can be determined from the images, and requires that any camera motion can be estimated from background elements. The computational requirements of the algorithm are modest, and any detected inconsistencies can be illustrated in an intuitive, geometric fashion. We demonstrate the efficacy of this analysis on videos of our own creation and on videos obtained from video-sharing websites.

The advent of sophisticated photo editing software has made it increasingly easier to manipulate digital images. Often visual inspection cannot definitively distinguish the resulting forgeries from authentic photographs. In response, forensic techniques have emerged to detect geometric or statistical inconsistencies that result from specific forms of photo manipulation. In this article we describe a new forensic technique that focuses on geometric inconsistencies that arise when fake reflections are inserted into a photograph or when a photograph containing reflections is manipulated. This analysis employs basic rules of reflective geometry and linear perspective projection, makes minimal assumptions about the scene geometry, and only requires the user to identify corresponding points on an object and its reflection. The analysis is also insensitive to common image editing operations such as resampling, color manipulations, and lossy compression. We demonstrate this technique with both visually plausible forgeries of our own creation and commercially produced forgeries.

It is often desirable to determine if an image has been modified in any way from its original recording. The JPEG format affords engineers many implementation trade-offs which give rise to widely varying JPEG headers. We exploit these variations for image authentication. A camera signature is extracted from a JPEG image consisting of information about quantization tables, Huffman codes, thumbnails, and exchangeable image file format (EXIF). We show that this signature is highly distinct across 1.3 million images spanning 773 different cameras and cell phones. Specifically, 62% of images have a signature that is unique to a single camera, 80% of images have a signature that is shared by three or fewer cameras, and 99% of images have a signature that is unique to a single manufacturer. The signature of Adobe Photoshop is also shown to be unique relative to all 773 cameras. These signatures are simple to extract and offer an efficient method to establish the authenticity of a digital image.

An audio recording is subject to a number of possible distortions and artifacts. For example, the persistence of sound, due to multiple reflections from various surfaces in a room, causes temporal and spectral smearing of the recorded sound. This distortion is referred to as audio reverberation time. We describe a technique to model and estimate the amount of reverberation in an audio recording. Because reverberation depends on the shape and composition of a room, differences in the estimated reverberation can be used in a forensic and ballistic setting.

The manipulation of text on a sign or billboard is relatively easy to do in a way that is perceptually convincing. When text is on a planar surface and imaged under perspective projection, the text undergoes a specific distortion. When text is manipulated, it is unlikely to precisely satisfy this geometric mapping. We describe a technique for detecting if text in an image obeys the expected perspective projection, deviations from which are used as evidence of tampering.

When creating a digital forgery, it is often necessary to combine several images, for example, when compositing one person's head onto another person's body. If these images were originally of different JPEG compression quality, then the digital composite may contain a trace of the original compression qualities. To this end, we describe a technique to detect whether the part of an image was initially compressed at a lower quality than the rest of the image. This approach is applicable to images of high and low quality as well as resolution.

The availability of sophisticated digital imaging technology has given rise to digital forgeries that are increasing in sophistication and frequency. We describe a technique for exposing such fakes by detecting inconsistencies in lighting. We show how to approximate complex lighting environments with a low-dimensional model and, further, how to estimate the model's parameters from a single image. Inconsistencies in the lighting model are th.en used as evidence of tampering.

With the advent of high-quality digital video cameras and sophisticated video editing software, it is becoming increasingly easier to tamper with digital video. A growing number of video surveillance cameras are also giving rise to an enormous amount of video data. The ability to ensure the integrity and authenticity of these data poses considerable challenges. We describe two techniques for detecting traces of tampering in deinterlaced and interlaced video. For deinterlaced video, we quantify the correlations introduced by the camera or software deinterlacing algorithms and show how tampering can disturb these correlations. For interlaced video, we show that the motion between fields of a single frame and across fields of neighboring frames should be equal. We propose an efficient way to measure these motions and show how tampering can disturb this relationship.

In this correspondence, we propose a direct method for estimating the orientation of a plane from a single view under perspective projection. Assuming that the underlying planar texture has random phase, we show that the nonlinearities introduced by perspective projection lead to higher order correlations in the frequency domain. We also empirically show that these correlations are proportional to the orientation of the plane. Minimization of these correlations, using tools from polyspectral analysis, yields the orientation of the plane. We show the efficacy of this technique on synthetic and natural images.

Techniques for information hiding (steganography) are becoming increasingly more sophisticated and widespread. With high-resolution digital images as carriers, detecting hidden messages is also becoming considerably more difficult. We describe a universal approach to steganalysis for detecting the presence of hidden messages embedded within digital images. We show that, within multiscale, multiorientation image decompositions (e.g., wavelets), first- and higher-order magnitude and phase statistics are relatively consistent across a broad range of images, but are disturbed by the presence of embedded hidden messages. We show the efficacy of our approach on a large collection of images, and on eight different steganographic embedding algorithms.

Brain deformation models have proven to be a powerful tool in compensating for soft tissue deformation during image-guided neurosurgery. The accuracy of these models can be improved by incorporating intraoperative measurements of brain motion. We have designed and implemented a passive intraoperative stereo vision system capable of estimating the three-dimensional shape of the surgical scene in near real-time. This intraoperative shape is compared with the cortical surface in the co-registered preoperative magnetic resonance (MR) volume for the estimation of the cortical motion resulting from the open cranial surgery. The estimated cortical motion is then used to guide a full brain model, which updates a preoperative MR volume. We have found that the stereo vision system is accurate to within approximately 1 mm. Based on data from two representative clinical cases, we show that stereopsis guidance improves the accuracy of brain shift compensation both at and below the cortical surface.

We describe the design of finite-size linear-phase separable kernels for differentiation of discrete multidimensional signals. The problem is formulated as an optimization of the rotation-invariance of the gradient operator, which results in a simultaneous constraint on a set of one-dimensional low-pass prefilter and differentiator filters up to the desired order. We also develop extensions of this formulation to both higher dimensions and higher order directional derivatives. We develop a numerical procedure for optimizing the constraint, and demonstrate its use in constructing a set of example filters. The resulting filters are significantly more accurate than those commonly used in the image and multidimensional signal processing literature.

We have developed a general-purpose registration algorithm for medical images and volumes. This method models the transformation between images as locally affine but globally smooth. The model also explicitly accounts for local and global variations in image intensities. This approach is built upon a differential multiscale framework, allowing us to capture both large- and small-scale transformations. We show that this approach is highly effective across a broad range of synthetic and clinical medical images.

Techniques for information hiding have become increasingly more sophisticated and widespread. With high-resolution digital images as carriers, detecting hidden messages has become considerably more difficult. This paper describes a new approach to detecting hidden messages in images. The approach uses a wavelet-like decomposition to build higher-order statistical models of natural images. A Fisher (1936) linear discriminant analysis is then used to discriminate between untouched and adulterated images.

The luminance nonlinearity introduced by many imaging devices can often be described by a simple point-wise operation (gamma correction). This paper presents a technique for blindly estimating the amount of gamma correction in the absence of any calibration information or knowledge of the imaging device. The basic approach exploits the fact that gamma correction introduces specific higher-order correlations in the frequency domain. These correlations can be detected using tools from polyspectral analysis. The amount of gamma correction is then estimated by minimizing these correlations.

The image of an object can vary dramatically depending on lighting, specularities/reflections and shadows. It is often advantageous to separate these incidental variations from the intrinsic aspects of an image. This paper describes how the statistical tool of independent components analysis can be used to separate some of these incidental components. We describe the details of this method and show its efficacy with examples of separating reflections off glass, and separating the relative contributions of individual light sources.

Steerable filters have been used to analyze local orientation patterns in imagery. Such filters are typically based on directional derivatives, whose symmetry produces orientation responses that are periodic with period pi, independent of image structure. We present a more general set of steerable filters that alleviate this problem.

Steerable filters, as developed by Freeman and Adelson (1991), are a class of rotation-invariant linear operators that may be used to analyze local orientation patterns in imagery. The most common examples of such operators are directional derivatives of Gaussians and their 2D Hilbert transforms. The inherent symmetry of these filters produces an orientation response that is periodic with period /spl pi/, even when the underlying image structure does not have such symmetry. This problem may be alleviated by reconsidering the full class of steerable filters. We develop a family of even- and odd-symmetric steerable filters that have a spatially asymmetric "wedge-like" shape and are optimally localized in their orientation response. Unlike the original steerable filters, these filters are not based on directional derivatives and the Hilbert transform relationship is imposed on their angular components. We demonstrate the ability of these filters to properly represent oriented structures.>