Common image-text joint understanding techniques presume that images and the associated text can universally be characterized by a single implicit model. However, co-occurring images and text can be related in qualitatively different ways, and explicitly modeling it could improve the performance of current joint understanding models. In this paper, we train a Cross-Modal Coherence Model for text-to-image retrieval task. Our analysis shows that models trained with image–text coherence relations can retrieve images originally paired with target text more often than coherence-agnostic models. We also show via human evaluation that images retrieved by the proposed coherence-aware model are preferred over a coherence-agnostic baseline by a huge margin. Our findings provide insights into the ways that different modalities communicate and the role of coherence relations in capturing commonsense inferences in text and imagery.

We propose an end-to-end network for visual illustration of a sequence of sentences forming a story. At the core of our model is the ability to model the inter-related nature of the sentences within a story, as well as the ability to learn coherence to support reference resolution. The framework takes the form of an encoder-decoder architecture, where sentences are encoded using a hierarchical two-level sentence-story GRU, combined with an encoding of coherence, and sequentially decoded using a predicted feature representation into a consistent illustrative image sequence. We optimize all parameters of our network in an end-to-end fashion with respect to order embedding loss, encoding entailment between images and sentences. Experiments on the VIST storytelling dataset [9] highlight the importance of our algorithmic choices and efficacy of our overall model.

Detecting Contrast Enhancement (CE) in images and anti-forensic approaches against such detectors have gained much attention in multimedia forensics lately. Several contrast enhancement detectors analyze the first order statistics such as gray-level histogram of images to determine whether an image is CE or not. In order to counter these detectors various anti-forensic techniques have been proposed. This led to a technique that utilized second order statistics of images for CE detection. In this letter, we propose an effective anti-forensic approach that performs CE without significant distortion in both the first and second order statistics of the enhanced image. We formulate an optimization problem using a variant of the well known Total Variation (TV) norm image restoration formulation. Experiments show that the algorithm effectively overcomes the first and second order statistics based detectors without loss in quality of the enhanced image.

The availability of intelligent image editing techniques and antiforensic algorithms, make it convenient to manipulate an image and to hide the artifacts that it might have produced in the process. Real world forgeries are generally followed by the application of enhancement techniques such as filtering and/or conversion of the image format to suppress the forgery artifacts. Though several techniques evolved in the direction of detecting some of these manipulations, additional operations like recompression, nonlinear filtering, and other antiforensic methods during forgery are not deeply investigated. Toward this, we propose a robust method to detect whether a given image has undergone filtering (linear or nonlinear) based enhancement, possibly followed by format conversion after forgery. In the proposed method, JPEG quantization noise is obtained using natural image prior and quantization noise models. Transition probability features extracted from the quantization noise are used for machine learning based detection and classification. We test the effectiveness of the algorithm in classifying the class of the filter applied and the efficacy in detecting filtering in low resolution images. Experiments are performed to compare the performance of the proposed technique with state-of-the-art forensic filtering detection algorithms. It is found that the proposed technique is superior in most of the cases. Also, experiments against popular antiforensic algorithms show the counter antiforensic robustness of the proposed technique.

Smart image editing and processing techniques make it easier to manipulate an image convincingly and also hide any artifacts of tampering. Common real world forgeries can be accompanied by enhancement operations like filtering, compression and/or format conversion to suppress forgery artifacts. Out of these enhancement operations, filtering is very common and has received a lot of attention in forensics research lately. However, different filtering operations and image formats are not investigated deeply and simultaneously. We propose an algorithm to detect if a given image has undergone filtering based enhancement irrespective of the format of image or the type of filter applied. In the proposed algorithm, we exploit the correlation of spatial domain quantization noise of an image by extracting transition probability features and classify the image as filtered or unfiltered. Experiments are performed to evaluate the robustness and compare the performance of the proposed technique with popular forensic filtering detection algorithms and is found to be superior in most of the cases.

Intelligent video editing techniques can be used to tamper videos such as surveillance camera videos, defeating their potential to be used as evidence in a court of law. In this paper, we propose a technique to detect forgery in MPEG videos by analyzing the frame's compression noise characteristics. The compression noise is extracted from spatial domain by using a modified Huber Markov Random Field (HMRF) as a prior for image. The transition probability matrices of the extracted noise are used as features to classify a given video as single compressed or double compressed. The experiment is conducted on different YUV sequences with different scale factors. The efficiency of our classification is observed to be higher relative to the state of the art detection algorithms.