Abstract Many biometry methods extract task-relevant information from images. In forensic applications, these images may stem from uncontrolled sources like surveillance cameras in the wild. Such devices oftentimes strongly compress the data, which can significantly complicate biometric tasks. This issue is exacerbated by the emergence of AI compression, which may provide visually appealing images that are of questionable value for biometric identification. The purpose of this work is to investigate potential pitfalls of AI compression. We evaluate six AI compression methods including the recently standardized JPEG AI on the four biometric modalities of irises, fingerprints, fabrics and tattoos. We qualitatively show multiple cases when AI compression achieves misleading results. Tattoos in particular includes misrepresentations of color or shapes at strong compression rates. The quantitative evaluation shows impact on recognition rates when there are few identifying features, such as in low-resolution iris images. Further results show that compressors with MSE loss are prone to omit important image details, and MSE+LPIPS loss may hallucinate features. The findings in this paper aim at raising awareness to these pitfalls, and aiding the development robust biometric algorithms for images in the wild.

Audio recordings may provide important evidence in criminal investigations. One such case is the forensic association of a recorded audio to its recording location. For example, a voice message may be the only investigative cue to narrow down the candidate sites for a crime. Up to now, several works provide supervised classification tools for closed-set recording environment identification under relatively clean recording conditions. However, in forensic investigations, the candidate locations are case-specific. Thus, supervised learning techniques are not applicable without retraining a classifier on a sufficient amount of training samples for each case and respective candidate set. In addition, a forensic tool has to deal with audio material from uncontrolled sources with variable properties and quality. In this work, we therefore attempt a major step towards practical forensic application scenarios. We propose a representation learning framework called EnvId, short for environment identification. EnvId avoids case-specific retraining by modeling the task as a few-shot classification problem. We demonstrate that EnvId can handle forensically challenging material. It provides good quality predictions even under unseen signal degradations, out-of-distribution reverberation characteristics or recording position mismatches. Code is available athttps://faui1-gitlab.cs.fau.de/mmsec/few-shot-recording-environment-identification.

Image forensics aims to provide computational tools for determining origin and authenticity of an image or for reconstructing the processing history of an image. In a broader definition, image forensics also includes the reconstruction of quantities of interest with the help of specific information within an image, for example the height of a person that is shown in the image.

High-quality artificially generated images are widely available now and increasingly realistic, posing challenges for image forensics in distinguishing them from real ones. Unfortunately, building a single detector that generalizes well to unseen generators is very difficult, creating the need for diverse cues. In this paper, we show that natural and synthetic images differ in their color statistics, possibly due to the widely used perceptual loss, which is more sensitive to brightness than to chroma differences. Consequently, color statistics offer valuable cues for forensic analysis and the development of robust detectors. Our experiments using simple hand-crafted color functions with a random forest achieve 91% accuracy averaged over all tested Diffusion Models, even with limited training samples.

The classical JPEG compression is a rich source of cues for forensic image analysis. However, this compression standard will in the near future be complemented by a new, highly efficient learning-based compression standard called JPEG-AI. JPEG-AI is fundamentally different from classical JPEG. Hence, its forensic traces can also be expected to be fundamentally different. We argue that there is a pressing need for image forensics research to investigate these traces. In this work, we characterize forensic compression traces of different AI compression algorithms. Our analysis investigates AI compression artifacts in frequency domain and in spatial domain. Both domains exhibit similar artifacts that likely stem from upsampling operations of the decoders. Additionally, we report for one AI codec another artifact in homogeneous regions. We also investigate the artifact detectability in several scenarios including unseen AI compression traces and postprocessing. Here, frequency and autocorrelation features are better on additive noise and classical JPEG post-compression, while RGB features perform better on blurred and downsampled images.

Hyperspectral remote sensing (HSRS) images have high dimensionality, and labeling HSRS data is expensive and therefore limited to small amounts of pixels. This makes it challenging to use deep neural networks for HSRS image classification. In extreme cases, deep neural networks are even outperformed by traditional models. In this work, we propose to use Bayesian convolutional neural networks (BCNNs) as a potential alternative to convolutional neural networks (CNNs). BCNNs benefit from Bayesian learning, which is more robust against overfitting and inherently provides a measure for uncertainty. We show in experiments on the Pavia Centre, Salinas, and Botswana datasets that a BCNN outperforms a similarly constructed non-Bayesian CNN, an off-the-shelf random forest (RF), and a state-of-the-art Bayesian neural network (BNN). We also show that BCNN is more robust against overfitting compared with the CNN. Furthermore, the BCNN exhibits a remarkably larger capacity for model compression, which makes BCNN a better candidate in hardware-constrained settings. Finally, we show that the BCNN’s uncertainty measure can effectively identify misclassified samples. This useful property can be used to detect mislabeled data or to reject predictions with low confidence.

Combining multiple classifiers is a promising approach to hardening forensic detectors against adversarial evasion attacks. The key idea is that an attacker must fool all individual classifiers to evade detection. The 1.5C classifier is one of these multiple-classifier detectors that is attack-agnostic, and thus even increases the difficulty for an omniscient attacker. Recent work evaluated the 1.5C classifier with SPAM features for image manipulation detection. Despite showing promising results, their security analysis leaves several aspects unresolved. Surprisingly, the results reveal that fooling only one component is often sufficient to evade detection. Additionally, the authors evaluate classifier robustness with only a black-box attack because, currently, there is no white-box attack against SPAM feature-based classifiers. This paper addresses these shortcomings and complements the previous security analysis. First, we develop a novel white-box attack against SPAM feature-based detectors. The proposed attack produces adversarial images with lower distortion than the previous attack. Second, by analyzing the 1.5C classifier’s acceptance region, we identify three pitfalls that explain why the current 1.5C classifier is less robust than a binary classifier in some settings. Third, we illustrate how to mitigate these pitfalls with a simple axis-aligned split classifier. Our experimental evaluation demonstrates the increased robustness of the proposed detector for SPAM feature-based image manipulation detection.

Learning-based algorithms for automated license plate recognition implicitly assume that the training and test data are well aligned. However, this may not be the case under extreme environmental conditions, or in forensic applications where the system cannot be trained for a specific acquisition device. Predictions on such out-of-distribution images have an increased chance of failing. But this failure case is oftentimes hard to recognize for a human operator or an automated system. Hence, in this work we propose to model the prediction uncertainty for license plate recognition explicitly. Such an uncertainty measure allows to detect false predictions, indicating an analyst when not to trust the result of the automated license plate recognition. In this paper, we compare three methods for uncertainty quantification on two architectures. The experiments on synthetic noisy or blurred low-resolution images show that the predictive uncertainty reliably finds wrong predictions. We also show that a multi-task combination of classification and super-resolution improves the recognition performance by 109% and the detection of wrong predictions by 29%.

Criminal investigations oftentimes need the identification of license plates of escape vehicles. The vehicles may be recorded by low-quality cameras in the wild. Their license plates may be unreadable for police officers. Recent efforts aim to use machine learning to forensically decipher license plates from such low-quality images. These methods operate near the information-theoretic limit of recognition and hence show quite high error rates. Unfortunately, it is unclear when such prediction errors occur, which makes it difficult to use these methods in practice. In this work, we propose a Bayesian Neural Network to inherently incorporate a reliability measure into the classifier. We additionally propose to integrate multiple estimations with an entropy weight to further improve the reliability. Our experiments show that this uncertainty metric dramatically reduces the number of false predictions while preserving most of the true predictions.

Forensic license plate recognition (FLPR) remains an open challenge in legal contexts such as criminal investigations, where unreadable license plates (LPs) need to be deciphered from highly compressed and/or low resolution footage, e.g., from surveillance cameras. In this work, we propose a side-informed Transformer architecture that embeds knowledge on the input compression level to improve recognition under strong compression. We show the effectiveness of Transformers for license plate recognition (LPR) on a low-quality real-world dataset. We also provide a synthetic dataset that includes strongly degraded, illegible LP images and analyze the impact of knowledge embedding on it. The network outperforms existing FLPR methods and standard state-of-the art image recognition models while requiring less parameters. For the severest degraded images, we can improve recognition by up to 8.9 percent points.1

Dynamic environments require adaptive applications. One particular machine learning problem in dynamic environments is open world recognition. It characterizes a continuously changing domain where only some classes are seen in one batch of the training data and such batches can only be learned incrementally. Open world recognition is a demanding task that is, to the best of our knowledge, addressed by only a few methods. This work introduces a modification of the widely known Extreme Value Machine (EVM) to enable open world recognition. Our proposed method extends the EVM with a partial model fitting function by neglecting unaffected space during an update. This reduces the training time by a factor of 28. In addition, we provide a modified model reduction using weighted maximum K-set cover to strictly bound the model complexity and reduce the computational effort by a factor of 3.5 from 2.1 s to 0.6 s. In our experiments, we rigorously evaluate openness with two novel evaluation protocols. The proposed method achieves superior accuracy of about 12 % and computational efficiency in the tasks of image classification and face recognition.

One common task in image forensics is to detect spliced images, where multiple source images are composed to one output image. Most of the currently best performing splicing detectors leverage high-frequency artifacts. However, after an image underwent strong compression, most of the high frequency artifacts are not available anymore. In this work, we explore an alternative approach to splicing detection, which is potentially better suited for images in-the-wild, subject to strong compression and downsampling. Our proposal is to model the color formation of an image. The color formation largely depends on variations at the scale of scene objects, and is hence much less dependent on high-frequency artifacts. We learn a deep metric space that is on one hand sensitive to illumination color and camera white-point estimation, but on the other hand insensitive to variations in object color. Large distances in the embedding space indicate that two image regions either stem from different scenes or different cameras. In our evaluation, we show that the proposed embedding space outperforms the state of the art on images that have been subject to strong compression and downsampling. We confirm in two further experiments the dual nature of the metric space, namely to both characterize the acquisition camera and the scene illuminant color. As such, this work resides at the intersection of physics-based and statistical forensics with benefits from both sides.

Recent advances in deep learning have led to significant improvements in single image super-resolution (SR) research. However, due to the amplification of noise during the upsampling steps, state-of-the-art methods often fail at reconstructing high-resolution images from noisy versions of their low-resolution counterparts. However, this is especially important for images from unknown cameras with unseen types of image degradation. In this work, we propose to jointly perform denoising and super-resolution. To this end, we investigate two architectural designs: "in-network" combines both tasks at feature level, while "pre-network" first performs denoising and then super-resolution. Our experiments show that both variants have specific advantages: The in-network design obtains the strongest results when the type of image corruption is aligned in the training and testing dataset, for any choice of denoiser. The pre-network design exhibits superior performance on unseen types of image corruption, which is a pathological failure case of existing super-resolution models. We hope that these findings help to enable super-resolution also in less constrained scenarios where source camera or imaging conditions are not well controlled. Source code and pretrained models are available at https://github.com/angelvillar96/super-resolution-noisy-images.

Supervised machine learning requires a large amount of labeled data to achieve proper test results. However, generating accurately labeled segmentation maps on remote sensing imagery, including images from synthetic aperture radar (SAR), is tedious and highly subjective. In this work, we propose to alleviate the issue of limited training data by generating synthetic SAR images with the pix2pix algorithm [1]. This algorithm uses conditional Generative Adversarial Networks (cGANs) to generate an artificial image while preserving the structure of the input. In our case, the input is a segmentation mask, from which a corresponding synthetic SAR image is generated. We present different models, perform a comparative study and demonstrate that this approach synthesizes convincing glaciers in SAR images with promising qualitative and quantitative results.

Abstract High resolution electroluminescence (EL) images captured in the infrared spectrum allow to visually and non-destructively inspect the quality of photovoltaic (PV) modules. Currently, however, such a visual inspection requires trained experts to discern different kinds of defects, which is time-consuming and expensive. Automated segmentation of cells is therefore a key step in automating the visual inspection workflow. In this work, we propose a robust automated segmentation method for extraction of individual solar cells from EL images of PV modules. This enables controlled studies on large amounts of data to understanding the effects of module degradation over time—a process not yet fully understood. The proposed method infers in several steps a high-level solar module representation from low-level ridge edge features. An important step in the algorithm is to formulate the segmentation problem in terms of lens calibration by exploiting the plumbline constraint. We evaluate our method on a dataset of various solar modules types containing a total of 408 solar cells with various defects. Our method robustly solves this task with a median weighted Jaccard index of $$94.47\%$$ 94.47% and an $$F_1$$ F1 score of $$97.62\%$$ 97.62% , both indicating a high sensitivity and a high similarity between automatically segmented and ground truth solar cell masks.

Identifying the model of a camera that has captured an image can be an important task in criminal investigations. Many methods assume that the image under analysis originates from a given set of known camera models. In practice, however, a photo can come from an unknown camera model, or its appearance could have been altered by unknown post-processing. In such a case, forensic detectors are prone to fail silently. One way to mitigate silent failures is to use a rejection mechanism for unknown examples. In this work, we propose Gaussian processes (GPs), which intrinsically provide such a rejection mechanism. This makes GPs a potentially powerful tool in multimedia forensics, where forensic analysts regularly work on images from unknown origins. We demonstrate that GPs scale well to the task of camera model identification. Probabilistic predictions from a GP classifier achieve high classification accuracy for known camera models while providing reliable uncertainty estimates. The built-in uncertainty estimates effectively tackle open-set camera model identification, outperforming two state-of-the-art methods.

With the ubiquity of social networks, images have become crucial in todays exchange of information. Most of these images are taken by smartphones. For forensic approaches relying on fixed image formation pipelines, the capabilities of smartphones using computational photography pose new challenges. But these new capabilities also offer opportunities for forensic analysis. A growing amount of commodity devices are able to capture 3-D information using various technologies such as stereo imaging or structured light. Modern smartphones commonly save such 3-D information as depth maps alongside regular images.In this work, we propose to use characteristic artifacts of depth reconstruction algorithms as trace for forensic analysis. The proposed method is able to infer the source algorithm of stereo reconstructions with an accuracy of up to 97%. We further demonstrate the applicability of the method to collected smartphone data. It is able to discriminate patches from different sources with an AUC of up to 0.88 and can be used for splicing localization in depth maps.

In multimedia forensics, learning-based methods provide state-of the-art performance in determining origin and authenticity of images and videos. However, most existing methods are challenged by out-of-distribution data, i.e., with characteristics that are not covered in the training set. This makes it difficult to know when to trust a model, particularly for practitioners with limited technical background.In this work, we make a first step toward redesigning forensic algorithms with a strong focus on reliability. To this end, we propose to use Bayesian neural networks (BNN), which combine the power of deep neural networks with the rigorous probabilistic formulation of a Bayesian framework. Instead of providing a point estimate like standard neural networks, BNNs provide distributions that express both the estimate and also an uncertainty range.We demonstrate the usefulness of this framework on a classical forensic task: resampling detection. The BNN yields state-of-the-art detection performance, plus excellent capabilities for detecting out-of-distribution samples. This is demonstrated for three pathologic issues in resampling detection, namely unseen resampling factors, unseen JPEG compression, and unseen resampling algorithms. We hope that this proposal spurs further research toward reliability in multimedia forensics.

Image reconstruction is particularly difficult when the type of image degradations are unknown. This may be the case if the acquisition device is unknown or the images stem from an uncontrolled environment like the internet. Yet, it may be important to reconstruct a specific piece of information from the image, such as digits from signs or vehicle license plates. Existing works incorporate such prior information with a sequential super-resolution and classification pipeline. However, this approach is prone to error propagation.In this work, we propose a new approach of connecting classification and super-resolution in parallel within a multi-task network. We show that this architecture is able to preserve structures and to remove noisy pixels although the network itself has never been trained on noisy data. We also show that this design allows to transparently trade classification and super-resolution quality. On upsampling by factor 4, we outperform sequential approaches in terms of SSIM by 10% and improve classification by 69%.

Capturing ground truth data to benchmark super-resolution (SR) is challenging. Therefore, current quantitative studies are mainly evaluated on simulated data artificially sampled from ground truth images. We argue that such evaluations overestimate the actual performance of SR methods compared to their behavior on real images. Toward bridging this simulated-to-real gap, we introduce the Super-Resolution Erlangen (SupER) database, the first comprehensive laboratory SR database of all-real acquisitions with pixel-wise ground truth. It consists of more than 80k images of 14 scenes combining different facets: CMOS sensor noise, real sampling at four resolution levels, nine scene motion types, two photometric conditions, and lossy video coding at five levels. As such, the database exceeds existing benchmarks by an order of magnitude in quality and quantity. This paper also benchmarks 19 popular single-image and multi-frame algorithms on our data. The benchmark comprises a quantitative study by exploiting ground truth data and qualitative evaluations in a large-scale observer study. We also rigorously investigate agreements between both evaluations from a statistical perspective. One interesting result is that top-performing methods on simulated data may be surpassed by others on real data. Our insights can spur further algorithm development, and the publicy available dataset can foster future evaluations.

Abstract Browser fingerprinting as a tracking technique to recognize users based on their browsers’ unique features or behavior has been known for more than a decade. We present the results of a 3-year online study on browser fingerprinting with more than 1,300 users. This is the first study with ground truth on user level, which allows the assessment of trackability based on fingerprints of multiple browsers and devices per user. Based on our longitudinal observations of 88,000 measurements with over 300 considered browser features, we optimized feature sets for mobile and desktop devices. Further, we conducted two user surveys to determine the representativeness of our user sample based on users’ demographics and technical background, and to learn how users perceive browser fingerprinting and how they protect themselves.

The goal of blind image forensics is to determine authenticity and origin of an image without using an explicitly embedded security scheme. Most existing forensic methods can roughly be grouped into statistical and physics-based approaches. Statistical methods can oftentimes be fully automated, and achieve impressive results on current state-of-the-art benchmarks. Physics-based methods explain image inconsistencies using an analytic model, and are more robust to common image processing operations such as resizing or recompression. In this work, we propose a physics-based forensic descriptor to characterize 2-D lighting environments of objects. The key idea is that the integral over a gradient field of an object indicates the direction of incident light in the image plane. In contrast to prior 2-D lighting methods, the proposed method is remarkably robust to changes in object color and variations in user input, as it operates on the whole object area instead of object contours. Furthermore, we show that the proposed method is unaffected by image resizing or compression, which makes it possible to analyze images that are impossible to analyze with current state-of-the-art statistical methods.

The detection of images that are spliced from multiple sources is one important goal of image forensics. Several methods have been proposed for this task, but particularly since the rise of social media, it is an ongoing challenge to devise forensic approaches that are highly robust to common processing operations such as strong JPEG recompression and downsampling.In this work, we make a first step towards a novel type of cue for image splicing, which is based on the color formation of an image. We make the assumption that the color formation is a joint result of the camera hardware, the software settings, and the depicted scene, and as such can be used to locate spliced patches that originally stem from different images. To this end, we train a two-stage classifier on the full set of colors from a Macbeth color chart, and compare two patches for their color consistency. Our preliminary results on a challenging dataset on downsampled data of identical scenes indicate that the color distribution can be a useful forensic tool that is highly resistant to JPEG compression.

The rapid progress in synthetic image generation and manipulation has now come to a point where it raises significant concerns for the implications towards society. At best, this leads to a loss of trust in digital content, but could potentially cause further harm by spreading false information or fake news. This paper examines the realism of state-of-the-art image manipulations, and how difficult it is to detect them, either automatically or by humans. To standardize the evaluation of detection methods, we propose an automated benchmark for facial manipulation detection. In particular, the benchmark is based on Deep-Fakes, Face2Face, FaceSwap and NeuralTextures as prominent representatives for facial manipulations at random compression level and size. The benchmark is publicly available and contains a hidden test set as well as a database of over 1.8 million manipulated images. This dataset is over an order of magnitude larger than comparable, publicly available, forgery datasets. Based on this data, we performed a thorough analysis of data-driven forgery detectors. We show that the use of additional domain-specific knowledge improves forgery detection to unprecedented accuracy, even in the presence of strong compression, and clearly outperforms human observers.

The JPEG compression format provides a rich source of forensic traces that include quantization artifacts, fingerprints of the container format, and numerical particularities of JPEG compressors. Such a diverse set of cues serves as the basis for a forensic examiner to determine origin and authenticity of an image. In this work, we present a novel artifact that can be used to fingerprint the JPEG compression library. The artifact arises from chroma subsampling in one of the most popular JPEG implementations. Due to integer rounding, every second column of the compressed chroma channel appears on average slightly brighter than its neighboring columns, which is why we call the artifact a "chroma wrinkle". We theoretically derive the chroma wrinkle footprint in DCT domain, and use this footprint for detecting chroma wrinkles. The artifact is detected with more than 90% accuracy on images of JPEG quality 75 and above. Our experiments indicate that the artifact can also be used for manipulation localization, and that it is robust to several global postprocessing operations.

Smart heating applications promise to increase energy efficiency and comfort by collecting and processing room climate data. While it has been suspected that the sensed data may leak crucial personal information about the occupants, this belief has up until now not been supported by evidence. In this work, we investigate privacy risks arising from the collection of room climate measurements. We assume that an attacker has access to the most basic measurements only: temperature and relative humidity. We train machine learning classifiers to predict the presence and number of room occupants and to discriminate between different types of activities. On data that was collected at three different locations, we show that occupancy can be detected from data measured by a single sensor with up to [Formula: see text] accuracy. One can even distinguish between the cases that no, one, or two persons are present with up to [Formula: see text] accuracy. Moreover, the four actions reading, working on a PC, standing, and walking, can be discriminated with up to [Formula: see text] accuracy, which is likewise clearly better than guessing ([Formula: see text]). Constraining the set of actions allows to achieve even higher prediction rates. For example, we discriminate standing and walking occupants with [Formula: see text] accuracy. In addition, we show that the accuracy can be increased in most cases if an attacker has access to measurements from two different sensors located in the same room. Our results provide evidence that even the leakage of such ‘inconspicuous’ data as temperature and relative humidity can seriously violate privacy.

Old master drawings were mostly created step by step in several layers using different materials. To art historians and restorers, examination of these layers brings various insights into the artistic work process and helps to answer questions about the object, its attribution and its authenticity. However, these layers typically overlap and are oftentimes difficult to differentiate with the unaided eye. For example, a common layer combination is red chalk under ink. In this work, we propose an image processing pipeline that operates on hyperspectral images to separate such layers. In particular, we propose to use two descriptors in hyperspectral historical document analysis, namely hyper-hue and extended multi-attribute profile (EMAP). We show that hyperspectral images enable better layer separation than RGB images, and that spectral focus stacking is an important preprocessing step towards that goal. Our comparative results with other features underline the efficacy of the three proposed improvements.

Labeling data for hyperspectral remote sensing image classification is a tedious and cost-intensive task. As a consequence, it is oftentimes necessary to perform classification when only very limited number of labeled training data is available. Several approaches have been proposed to address this problem. A recent proposal is to generate additional synthetic samples from a Gaussian Mixture Model for each class. One challenge with this approach lies in determining the number of components in the GMM. In this paper, we propose an approximation algorithm to select the number of components, namely Variational Bayesian (VB). The main advantage of VB is that it does not require multiple clustering computations in advance. Variational Bayesian not only greatly decreases the computational cost, but also generates comparable or better results in comparison to other methods.

The amount of training data that is required to train a classifier scales with the dimensionality of the feature data. In hyperspectral remote sensing (HSRS), feature data can potentially become very high dimensional. However, the amount of training data is oftentimes limited. Thus, one of the core challenges in HSRS is how to perform multiclass classification using only relatively few training data points. In this letter, we address this issue by enriching the feature matrix with synthetically generated sample points. These synthetic data are sampled from a Gaussian mixture model (GMM) fitted to each class of the limited training data. Although the true distribution of features may not be perfectly modeled by the fitted GMM, we demonstrate that a moderate augmentation by these synthetic samples can effectively replace a part of the missing training samples. Doing so, the median gain in classification performance is 5% on two datasets. This performance gain is stable for variations in the number of added samples, which makes it easy to apply this method to real-world applications.

Video content can be acquired with off-the-shelf hardware, and is thus increasingly used to record events. With the growing role of video data for communicating to a large audience, we need tools to ensure the authenticity of video content. However, until now, only few methods exist to forensically analyze videos. In this work, we propose a method for statistically comparing two video sequences. Per sequence, intra- and inter-frame residuals are computed. Optical flow is used to compensate for motion artifacts on inter-frame residuals. We use one sequence to build a statistical model, and compare it to the second sequence. From a forensic perspective, the proposed method enables two applications. First, manipulations can be accurately localized if both sequences are subsequences of the same video. Second, source cameras can be distinguished if both sequences stem from different videos. The proposed method is evaluated on collected smartphone data and green-screen splices. Further, it is quantitatively compared to both a recent PRNU-based approach and a technique based on autoencoders.

Many camera calibration techniques require the detection of a pattern with known geometry, e.g., a checkerboard. Typically, the pattern must be fully contained in the field of view. This brings several limitations, one of which is that lens distortion can not reliably be estimated in outer image regions. This paper presents the occluded checkerboard pattern detector (OCPAD) to find checkerboards, even in a) low-resolution images, b) images with high lens distortion and if c) the pattern is partly occluded or not completely within the field of view. We exploit that checkerboards can easily be represented by a graph. We use graph matching to find the largest partial checkerboard in the image. Our detector complements a state-of-the-art calibration algorithm. Quantitatively, detection rates are considerably improved over the state-of-the-art. Additionally, estimation of lens distortion is greatly improved at outer image regions. Here, the reprojection error is improved by up to 50%.

Most existing color constancy algorithms assume uniform illumination. However, in real-world scenes, this is not often the case. Thus, we propose a novel framework for estimating the colors of multiple illuminants and their spatial distribution in the scene. We formulate this problem as an energy minimization task within a conditional random field over a set of local illuminant estimates. In order to quantitatively evaluate the proposed method, we created a novel data set of two-dominant-illuminant images comprised of laboratory, indoor, and outdoor scenes. Unlike prior work, our database includes accurate pixel-wise ground truth illuminant information. The performance of our method is evaluated on multiple data sets. Experimental results show that our framework clearly outperforms single illuminant estimators as well as a recently proposed multi-illuminant estimation approach.

For decades, photographs have been used to document space-time events and they have often served as evidence in courts. Although photographers are able to create composites of analog pictures, this process is very time consuming and requires expert knowledge. Today, however, powerful digital image editing software makes image modifications straightforward. This undermines our trust in photographs and, in particular, questions pictures as evidence for real-world events. In this paper, we analyze one of the most common forms of photographic manipulation, known as image composition or splicing. We propose a forgery detection method that exploits subtle inconsistencies in the color of the illumination of images. Our approach is machine-learning-based and requires minimal user interaction. The technique is applicable to images containing two or more people and requires no expert interaction for the tampering decision. To achieve this, we incorporate information from physics- and statistical-based illuminant estimators on image regions of similar material. From these illuminant estimates, we extract texture- and edge-based features which are then provided to a machine-learning approach for automatic decision-making. The classification performance using an SVM meta-fusion classifier is promising. It yields detection rates of 86% on a new benchmark dataset consisting of 200 images, and 83% on 50 images that were collected from the Internet.

A copy-move forgery is created by copying and pasting content within the same image, and potentially postprocessing it. In recent years, the detection of copy-move forgeries has become one of the most actively researched topics in blind image forensics. A considerable number of different algorithms have been proposed focusing on different types of postprocessed copies. In this paper, we aim to answer which copy-move forgery detection algorithms and processing steps (e.g., matching, filtering, outlier detection, affine transformation estimation) perform best in various postprocessing scenarios. The focus of our analysis is to evaluate the performance of previously proposed feature sets. We achieve this by casting existing algorithms in a common pipeline. In this paper, we examined the 15 most prominent feature sets. We analyzed the detection performance on a per-image basis and on a per-pixel basis. We created a challenging real-world copy-move dataset, and a software framework for systematic image manipulation. Experiments show, that the keypoint-based features Sift and Surf, as well as the block-based DCT, DWT, KPCA, PCA, and Zernike features perform very well. These feature sets exhibit the best robustness against various noise sources and downsampling, while reliably identifying the copied regions.

Most algorithms for extracting illuminant chromaticity from arbitrary images, such as the images found on the web, are based on machine learning techniques. We will show how a physics-based methodology can be adapted to provide relative illumination information on real images. More specifically, we use the inverse-intensity chromaticity representation and show how the analysis of the histograms of illumination-chromaticity candidates provides information about the type of illumination(s) present in a scene. Experiments indicate that the estimate is quite robust towards noise, and that simple measurements on the histogram peak can be used to counter-check the reliability of the estimate.