We present a framework for learning multimodal representations from unlabeled data using convolution-free Transformer architectures. Specifically, our Video-Audio-Text Transformer (VATT) takes raw signals as inputs and extracts multimodal representations that are rich enough to benefit a variety of downstream tasks. We train VATT end-to-end from scratch using multimodal contrastive losses and evaluate its performance by the downstream tasks of video action recognition, audio event classification, image classification, and text-to-video retrieval. Furthermore, we study a modality-agnostic single-backbone Transformer by sharing weights among the three modalities. We show that the convolution-free VATT outperforms state-of-the-art ConvNet-based architectures in the downstream tasks. Especially, VATT's vision Transformer achieves the top-1 accuracy of 82.1% on Kinetics-400, 83.6% on Kinetics-600, 72.7% on Kinetics-700, and 41.1% on Moments in Time, new records while avoiding supervised pre-training. Transferring to image classification leads to 78.7% top-1 accuracy on ImageNet compared to 64.7% by training the same Transformer from scratch, showing the generalizability of our model despite the domain gap between videos and images. VATT's audio Transformer also sets a new record on waveform-based audio event recognition by achieving the mAP of 39.4% on AudioSet without any supervised pre-training.

The electrical network frequency (ENF) signal is a time stamp that has been used by an emerging class of approaches for determining the creation time of digital audio and video recordings. However, in adversarial environments, anti-forensic operations may be conducted to manipulate ENF-based time stamps, and it is crucial to understand the resilience of ENF analysis against anti-forensics. This paper explores possible anti-forensic operations that can remove and alter the ENF signal while trying to preserve the host signal, and devises detection methods targeting these operations. Concealment techniques that can circumvent detection are also discussed and their corresponding trade-offs are examined. Based on the understanding of individual anti-forensic operations and countermeasures, this paper further characterizes the dynamic interplay between forensic analysts and adversaries by providing an evolutionary perspective and a game-theoretical perspective as well as studying representative scenarios and the optimal forensic/anti-forensic strategies.

A time stamp based on the power network signature called the Electrical Network Frequency (ENF) has been used by an emerging class of approaches for authenticating digital audio and video recordings in computer-to-computer communications. However, the presence of adversaries may render the time stamp insecure, and it is crucial to understand the robustness of ENF analysis against anti-forensic operations. This paper investigates possible anti-forensic operations that can remove and alter the ENF signal while trying to preserve the host signal, and develops detection methods targeting these operations. Improvements over anti-forensic operations that can circumvent the detection are also examined, for which various trade-offs are discussed. To develop an understanding of the dynamics between a forensic analyst and an adversary, an evolutionary perspective and a game-theoretical perspective are proposed, which allow for a comprehensive characterization of plausible anti-forensic strategies and countermeasures. Such an understanding has the potential to lead to more secure and reliable time stamp schemes based on ENF analysis.

Scale-Invariant Feature Transform (SIFT) is one of the most popular local image features that are widely used in computer vision, image processing and image retrieval. In this paper we study the relation between the SIFT descriptor and its matching accuracy. We propose a method to quantitatively assess the quality of a SIFT feature descriptor in terms of robustness and discriminability. This would enable us to gain a better understanding of the strength and limitations of SIFT in emerging applications of SIFT-based image hash, and also to improve matching accuracy and efficiency in applications such as object search. The experimental results demonstrate the effectiveness of the proposed method.

This paper presents a study of the video compression effect on source camera identification based on the Photo-Response Non-Uniformity (PRNU). Specifically, the reliability of different types of frames in a compressed video is first investigated, which shows quantitatively that I-frames are more reliable than P-frames for PRNU estimation. Motivated by this observation, a new mechanism for estimating the reference PRNU and two mechanisms for estimating the test-video PRNU are proposed to achieve higher accuracy with fewer frames used. Experiments are performed to validate the effectiveness of the proposed mechanisms.

Content fingerprinting provides a compact representation of multimedia objects for copy identification. This paper analyzes the impact of the ordinal-ranking based feature encoding on the performance of content fingerprinting. Expressions are derived for the identification performance of a fingerprinting system with and without ordinal ranking. The analysis indicates that when the number of features is moderately large, ordinal ranking can improve the robustness of the fingerprinting system to large distortions of the features and significantly increase the probability of detection. These results enhance understandings of ordinal ranking and provide design guidelines for choosing different system parameters to achieve a desired identification accuracy.

With the widespread popularity of digital images and the presence of easy-to-use image editing software, content integrity can no longer be taken for granted, and there is a strong need for techniques that not only detect the presence of tampering but also identify its type. This paper focusses on tampering-type identification and introduces a new approach based on the empirical frequency response (EFR) to address this problem. We show that several types of tampering operations, both linear shift invariant (LSI) and non-LSI, can be characterized consistently and distinctly by their EFRs. We then extend the approach to estimate the EFR for scenarios where only the final image is available. Theoretical reasoning supported by experimental results verify the effectiveness of this method for identifying the type of a tampering operation.

Palette re-indexing has been shown to be an effective way to improve the compression of color-indexed images. Based on two operations, i.e., sampling and swapping, we propose a novel re-indexing technique with the lowest complexity on the palette size and ability to further improve existing methods, which are two unique features that contrast others. Experiment results verify the effectiveness of this technique.

The process of limited-color image compression usually involves color quantization followed by palette re-indexing. Palette re-indexing could improve the compression of color-indexed images, but it is still complicated and consumes extra time. Making use of the topology-preserving property of self-organizing Kohonen feature map, we can generate a fairly good color index table to achieve both high image quality and high compression, without re-indexing. Promising experiment results will be presented.