Deepfakes can be used maliciously to sway public opinion, defame an individual, or commit fraud. Hence, it is vital for journalists and social media platforms, as well as the general public, to be able to detect deepfakes. Existing deepfake detection methods, while highly accurate on datasets they have been trained on, falter in open-world scenarios due to different deepfake generations algorithms, video formats, and compression levels. In this paper, we seek to address this by building on the XceptionNet-based deepfake detection technique that utilizes convolutional latent representations with recurrent structures. In particular, we explore how to leverage a combination of visual frames, edge maps, and dense optical flow maps together as inputs to this architecture. We evaluate these techniques using the FaceForensics++ and DFDC-mini datasets. We also perform extensive studies to evaluate the robustness of our network against adversarial post-processing as well as the generalization capabilities to out-of-domain datasets and manipulation strategies. Our methods, which we call XceptionNet*, achieve 100% accuracy on the popular Face-Forensics-s+ dataset and set new benchmark standards on the difficult DFDC-mini dataset. The XceptionNet* models are shown to exhibit superior performance on cross-domain testing and demonstrate surprising resilience to adversarial manipulations.

Point cloud analysis is an important field for 3D scene understanding. It has applications in self driving cars and robotics (via LIDAR sensors), 3D graphics, and computer-aided design. Neural networks have recently achieved strong results on point cloud analysis problems such as classification and segmentation. Each point cloud network has the challenge of defining a convolution that can learn useful features on unstructured points. Some recent point cloud convolutions create separate weight matrices for separate directions like a CNN, but apply every weight matrix to every neighbor with soft assignments. This increases computational complexity and makes relatively small neighborhood aggregations expensive to compute. We propose Hard Directional Graph Networks (HDGN), a point cloud model that both learns directional weight matrices and assigns a single matrix to each neighbor, achieving directional convolutions at lower computational cost. HDGN's directional modeling achieves state-of-the-art results on multiple point cloud vision benchmarks.

There is heightened concern over deliberately inaccurate news. Recently, so-called deepfake videos and images that are modified by or generated by artificial intelligence techniques have become more realistic and easier to create. These techniques could be used to create fake announcements from public figures or videos of events that did not happen, misleading mass audiences in dangerous ways. Although some recent research has examined accurate detection of deepfakes, those methodologies do not generalize well to real-world scenarios and are not available to the public in a usable form. In this project, we propose a system that will robustly and efficiently enable users to determine whether or not a video posted online is a deepfake. We approach the problem from the journalists' perspective and work towards developing a tool to fit seamlessly into their workflow. Results demonstrate accurate detection on both within and mismatched datasets.

Sign language is the primary way of communication between deaf people, but the majority of hearing people do not know how to sign. The reliance of deaf people on interpreters is both inconvenient and cost inefficient. Many research groups have experimented with using machine learning to develop automatic translators. Largely, these efforts have been constrained to restrictive dictionaries or insufficiently small signers or signed content. We introduce the world's largest sign language dataset to date- a collection of 50,000 video snippets taken from a pool of 10,000 unique utterances signed by 50 signers. We further propose several sequence-to-sequence deep learning approaches to automatically translate from Chinese sign language to both English and Mandarin written text. These methods utilize body joint position, facial expression, as well as finger articulation. While models can overfit on training sets, generalization to unforeseen utterances remains challenging with real-world data. The introduced dataset and methods demonstrate how modern machine learning methods are able to close the communication gap between deaf and hearing people.

Image-to-image translation is the task of translating images between domains while maintaining the identities of the images. Generative Adversarial Networks (GANs), and in particular conditional GANs have recently shown incredible success in image-to-image translation and semantic manipulation. Such methods require paired data, meaning that an image must have ground-truth translations across domains. Cycle-consistent GANs solve this problem by using unpaired data. Such methods work well for translations that involve color and texture changes but fail when shape changes are required. This paper firstly analyzes the trade-offs between the cycle-consistency importance and the necessary shape changes required for natural looking imagery. We then propose computationally simple architectural and loss changes to allow the model to perform color, texture, and shape changes as required. The results demonstrate improved translations between domains that require shape changes. We additionally show how the embeddings learned by our model learn interesting and useful attention/segmentation information about the translated images.

Humans have an incredible ability to process and understand information from multiple sources such as images, video, text, and speech. Recent success of deep neural networks has enabled us to develop algorithms which give machines the ability to understand and interpret this information. There is a need to both broaden their applicability and develop methods which correlate visual information along with semantic content. We propose a unified model which jointly trains on images and captions, and learns to generate new captions given either an image or a caption query. We evaluate our model on three different tasks namely cross-modal retrieval, image captioning, and sentence paraphrasing. Our model gains insight into cross-modal vector embeddings, generalizes well on multiple tasks and is competitive to state of the art methods on retrieval.

Handwritten text recognition is challenging because of the virtually infinite ways a human can write the same message. Our fully convolutional handwriting model takes in a handwriting sample of unknown length and outputs an arbitrary stream of symbols. Our dual stream architecture uses both local and global context and mitigates the need for heavy preprocessing steps such as symbol alignment correction as well as complex post processing steps such as connectionist temporal classification, dictionary matching or language models. Using over 100 unique symbols, our model is agnostic to Latin-based languages, and is shown to be quite competitive with state of the art dictionary based methods on the popular IAM and RIMES datasets. When a dictionary is known, we further allow a probabilistic character error rate to correct errant word blocks. Finally, we introduce an attention based mechanism which can automatically target variants of handwriting, such as slant, stroke width, or noise.

Image captioning has demonstrated models that are capable of generating plausible text given input images or videos. Further, recent work in image generation has shown significant improvements in image quality when text is used as a prior. Our work ties these concepts together by creating an architecture that can enable bidirectional generation of images and text. We call this network Multi-Modal Vector Representation (MMVR). Along with MMVR, we propose two improvements to the text conditioned image generation. Firstly, a n-gram metric based cost function is introduced that generalizes the caption with respect to the image. Secondly, multiple semantically similar sentences are shown to help in generating better images. Qualitative and quantitative evaluations demonstrate that MMVR improves upon existing text conditioned image generation results by over 20%, while integrating visual and text modalities.

Recent work has demonstrated that neural embedding from multiple modalities can be utilized to focus the results of generative adversarial networks. However, little work has been done towards developing a procedure to combine vectors from different modalities for the purpose of reconstructing input. Generally, embeddings from different modalities are concatenated to create a larger input vector. In this paper, we propose learning a Common Vector Space (CVS) where similar inputs from different modalities cluster together. We develop a framework to analyze the extent of reconstruction and robustness offered by CVS. We apply the CVS for the purpose of annotating, generating and captioning images on MS-COCO. We show that CVS is on par with techniques used for multiple modality embeddings while offering more flexibility as the number of modalities increases.

Depth sensors used in autonomous driving and gaming systems often report back 3D point clouds. The lack of structure from these sensors does not allow these systems to take advantage of recent advances in convolutional neural networks which are dependent upon traditional filtering and pooling operations. Analogous to image based convolutional architectures, recently introduced graph based architectures afford similar filtering and pooling operations on arbitrary graphs. We adopt these graph based methods to 3D point clouds to introduce a generic vector representation of 3D graphs, we call graph 3D (G3D). We believe we are the first to use large scale transfer learning on 3D point cloud data and demonstrate the discriminant power of our salient latent representation of 3D point clouds on unforeseen test sets. By using our G3D network (G3DNet) as a feature extractor, and then pairing G3D feature vectors with a standard classifier, we achieve the best accuracy on ModelNet10 (93.1%) and ModelNet 40 (91.7%) for a graph network, and comparable performance on the Sydney Urban Objects dataset to other methods. This general-purpose feature extractor can be used as an off-the-shelf component in other 3D scene understanding or object tracking works.

Voxels are an effective approach to 3D mesh and point cloud classification because they build upon mature Convolutional Neural Network concepts. We show however that their cubic increase in dimensionality is unsuitable for more challenging problems such as object detection in a complex point cloud scene. We observe that 3D meshes are analogous to graph data and can thus be treated with graph signal processing techniques. We propose a Graph Convolutional Neural Network (Graph-CNN), which enables mesh data to be represented exactly (not approximately as with voxels) with quadratic growth as the number of vertices increases. We apply Graph-CNN to the ModelNet10 classification dataset and demonstrate improved results over a previous graph convolution method.

Gradient control plays an important role in feed-forward networks applied to various computer vision tasks. Previous work has shown that Recurrent Highway Networks minimize the problem of vanishing or exploding gradients. They achieve this by setting the eigenvalues of the temporal Jacobian to 1 across the time steps. In this work, batch normalized recurrent highway networks are proposed to control the gradient flow in an improved way for network convergence. Specifically, the introduced model can be formed by batch normalizing the inputs at each recurrence loop. The proposed model is tested on an image captioning task using MSCOCO dataset. Experimental results indicate that the batch normalized recurrent highway networks converge faster and performs better compared with the traditional LSTM and RHN based models.

Long videos captured by consumers are typically tied to some of the most important moments of their lives, yet ironically are often the least frequently watched. The time required to initially retrieve and watch sections can be daunting. In this work we propose novel techniques for summarizing and annotating long videos. Existing video summarization techniques focus exclusively on identifying keyframes and subshots, however evaluating these summarized videos is a challenging task. Our work proposes methods to generate visual summaries of long videos, and in addition proposes techniques to annotate and generate textual summaries of the videos using recurrent networks. Interesting segments of long video are extracted based on image quality as well as cinematographic and consumer preference. Key frames from the most impactful segments are converted to textual annotations using sequential encoding and decoding deep learning models. Our summarization technique is benchmarked on the VideoSet dataset, and evaluated by humans for informative and linguistic content. We believe this to be the first fully automatic method capable of simultaneous visual and textual summarization of long consumer videos.

Many computer vision algorithms are biologically inspired and designed based on the human visual system. Convolutional neural networks (CNNs) are similarly inspired by the primary visual cortex in the human brain. However, the key difference between current visual models and the human visual system is how the visual information is gathered and processed. We make eye movements to collect information from the environment for navigation and task performance. We also make specific eye movements to important regions in the stimulus to perform the task-at-hand quickly and efficiently. Researchers have used expert scanpaths to train novices for improving the accuracy of visual search tasks. One of the limitations of such a system is that we need an expert to examine each visual stimuli beforehand to generate the scanpaths. In order to extend the idea of gaze guidance to a new unseen stimulus, there is a need for a computational model that can automatically generate expert-like scanpaths. We propose a model for automatic scanpath generation using a convolutional neural network (CNN) and long short-term memory (LSTM) modules. Our model uses LSTMs due to the temporal nature of eye movement data (scanpaths) where the system makes fixation predictions based on previous locations examined.

Current research in computer vision and machine learning has demonstrated some great abilities at detecting and recognizing objects in natural images. The promising results in these areas have inspired research towards solving more complex multi-modal learning problems in the image/video domains such as automatic annotation, segmentation, labelling, and generic understanding. Although solutions have been provided for one or more of these problems, their approaches have been application specific. This paper introduces an end-to-end trainable Fusion-based Recurrent Multi-Modal (FRMM) model to address multi-modal applications. FRMM allows each input modality to be independent in terms of architecture, parameters and length of input sequences. FRMM image description models seamlessly blend convolutional neural network feature descriptors with sequential language data in a recurrent framework. For training and testing we used the Flickr30K and MSCOCO datasets, demonstrating state-of-the-art description results.

Within the fields of visual effects and animation, humans have historically spent painstaking hours mastering the skill of drawing frame-by-frame animations. One such animation technique that has been widely used is called “rotoscoping” and has allowed uniquely stylized animations to capture the motion of real life action sequences. Automating this arduous process would free animators from performing frame by frame stylization to concentrate on artistic contributions. We introduce a new artificial system based on an existing neural style transfer method which creates artistically stylized animations that simultaneously reproduce both the motion of the original videos that they are derived from and the unique style of a given artistic work. This system utilizes a convolutional neural network framework to extract a hierarchy of image features used for generating images that appear visually similar to a given artistic style while at the same time faithfully preserving temporal content. The use of optical flow allows the combination of style and content to be integrated directly with the apparent motion over frames of a video to produce smooth and visually appealing transitions. This implementation demonstrates how biologically-inspired systems such as convolutional neural networks are rapidly approaching human-level behavior in tasks that were once thought impossible. Further, this research provides unique insights into the way that humans who produce artistically stylized animations perceive temporal information.

Surveillance cameras have become big business, with most metropolitan cities spending millions of dollars to watch residents, both from street corners, public transportation hubs, and body cameras on officials. Watching and processing the petabytes of streaming video is a daunting task, making automated and user assisted methods of searching and understanding videos critical to their success. Although numerous techniques have been developed, large scale video classification remains a difficult task due to excessive computational requirements. In this paper, we conduct an in-depth study to investigate effective architectures and semantic features for efficient and accurate solutions to activity recognition. We investigate different color spaces, optical flow, and introduce a novel deep learning fusion architecture for multi-modal inputs. The introduction of key frame extraction, instead of using every frame or a random representation of video data, make our methods computationally tractable. Results further indicate that transforming the image stream into a compressed color space reduces computational requirements with minimal affect on accuracy.

Hierarchical decomposition enables increased number of classes in a classification problem. Class similarities guide the creation of a family of course to fine classifiers which solve categorical problems more effectively than a single flat classifier. High accuracies require precise configurations for each of the family of classifiers. This paper proposes a method to adaptively select the configuration of the hierarchical family of classifiers. Linkage statistics from overall and sub-classification confusion matrices define categorical groupings for an efficient and accurate classification framework. Depending on the number of classes and the complexity of the problem, an adaptive configuration manager chooses between a multi-layer perceptron or a deep convolutional neural network, then selects the complexity of each. Experiments on CalTech101, CalTech256, CIFAR100 and the ImageNet datasets demonstrate performance of adaptive hierarchical models on an image classification task.

Domestic abuse affects people of every race, class, age, and nation. There is sig-nificant research on the prevalence and ef-fects of domestic abuse; however, such re-search typically involves population-based surveys that have high financial costs. This work provides a qualitative analysis of do-mestic abuse using data collected from the social and news-aggregation website red-dit.com. We develop classifiers to detect submissions discussing domestic abuse, achieving accuracies of up to 92%, a sub-stantial error reduction over its baseline. Analysis of the top features used in detect-ing abuse discourse provides insight into the dynamics of abusive relationships. 1

The introduction of low cost depth cameras along with advances in computer vision have spawned an exciting new era in Human Computer Interaction. Real time gesture recognition systems have become commonplace and attention has now turned towards making these systems invariant to within-user and user-to-user variation. Active difference signatures have been used to describe temporal motion as well as static difference from a canonical resting position. Geometric features, such as joint angles, and joint topological distances can be used along with active difference signatures as salient feature descriptors. To achieve robustness to natural gesture variation, this paper introduces active body part recognition along with these features into the Hidden Markov Model framework. The proposed method is bench-marked against other methods, achieving state of the art results on the MSR3D and ChaLearn datasets.

Music classification is a challenging problem with many applications in today's large-scale datasets with Gigabytes of music files and associated metadata and online streaming services. Recent success with deep neural network architectures on large-scale datasets has inspired numerous studies in the machine learning community for various pattern recognition and classification tasks such as automatic speech recognition, natural language processing, audio classification and computer vision. In this paper, we explore a two-layer neural network with manifold learning techniques for music genre classification. We compare the classification accuracy rate of deep neural networks with a set of well-known learning models including support vector machines (SVM and '1-SVM), logistic regression and '1-regression in combination with hand-crafted audio features for a genre classification task on a public dataset. Our experimental results show that neural networks are comparable with classic learning models when the data is represented in a rich feature space.

Nicolas Schrading, Cecilia Ovesdotter Alm, Raymond Ptucha, Christopher Homan. Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2015.

With the introduction of low cost depth sensors, gesture recognition systems for computer interface control are becoming a reality and require accurate recognition in real time. In this paper, we introduce a salient feature for gesture recognition called active difference signature, obtained by robust processing of depth maps and kinematic joint information. This feature is classified with variants of semi-supervised Linear extension of Graph Embedding and sparse representations to demonstrate advanced gesture control using depth cameras. Our active difference signature technique delivers highly accurate gesture recognition that is invariant to subject distance to camera, subject size, subject location within the camera field of view, and speed at which the subject performs the gestures.

The parsimonious nature of sparse representations has been successfully exploited for the development of highly accurate classifiers for various scientific applications. Despite the successes of Sparse Representation techniques, a large number of dictionary atoms as well as the high dimensionality of the data can make these classifiers computationally demanding. Furthermore, sparse classifiers are subject to the adverse effects of a phenomenon known as coefficient contamination, where, for example, variations in pose may affect identity and expression recognition. We analyze the interaction between dimensionality reduction and sparse representations, and propose a technique, called Linear extension of Graph Embedding K-means-based Singular Value Decomposition (LGE-KSVD) to address both issues of computational intensity and coefficient contamination. In particular, the LGE-KSVD utilizes variants of the LGE to optimize the K-SVD, an iterative technique for small yet over complete dictionary learning. The dimensionality reduction matrix, sparse representation dictionary, sparse coefficients, and sparsity-based classifier are jointly learned through the LGE-KSVD. The atom optimization process is redefined to allow variable support using graph embedding techniques and produce a more flexible and elegant dictionary learning algorithm. Results are presented on a wide variety of facial and activity recognition problems that demonstrate the robustness of the proposed method.

Active Shape Models (ASMs) have been widely used in facial feature representation and related applications. In this paper, we exploit the availability of registered depth and color information from a low resolution camera sensor to improve Active Shape Model fitting accuracy and efficiency. Two independent Active Shape Model profile models are constructed based on the gradient profiles of depth and color intensity information. These two channels are fused using full or partial depth information. The algorithm determines the optimal point localization using the color/depth profiles during the update step of the ASM fitting process. Results show improvements in accuracy over standard methods using color intensity or depth information alone.

The field of sparse representations has found applications in a variety of computer vision and scientific fields. Although sparse representations were initially considered for reconstruction, they have been successfully adapted for classification. In this paper, we demonstrate the application of sparse representations in matching salient keypoint descriptors for image alignment, registration, or stitching. Our method initially builds a dictionary from keypoints in a reference image. Then keypoints associated with one or more secondary images are sparsely represented using the reference dictionary. The sparse coefficient signatures are analyzed to determine if there is a matched pair and identify the reference keypoint of the match. Top keypoint matches are used to construct the homography transformation for image registration. The usefulness of our methodology is demonstrated across several types of imagery showing robust performance while delivering state-of-the-art image alignment and registration.

Facial expression research on unconstrained spontaneous expressions has benefited from recent advances in feature extraction, dimensionality reduction, and classification techniques. While the facial action coding relies on temporal predictors, state-of-the-art facial expression recognition techniques have been slow to adapt to temporal methods. Further, despite strong evidence of sparsity in the visual cortex, few approaches to facial understanding utilize sparse representations. This paper proposes using a temporal detector based upon facial dynamics, a static expression detector based on sparse representations, and an advanced temporal-sparse fused emotion estimator. Our approach leverages techniques from both computer vision and human brain research to produce a state-of-the-art emotion estimator on unconstrained faces in natural conditions.

Research on facial expression recognition has steadily been moving from analysis of deliberative frontal expressions to analysis of unconstrained spontaneous expressions. This shift has spawned complex 3D models and computationally expensive geometric methods that prevent usage on resource constrained platforms such as smart phones. This paper presents manifold learning techniques for accurate multi-view facial expression on low resolution 2D images. Our results indicate that mixed class local pose and expression manifold methods perform better than global expression techniques and work just as well as fusing together results from multiple manifolds.

This paper presents robust facial pose estimation techniques based on the underlying low dimensional manifolds embedded in facial images of varying pose. In our approach, facial feature points of training faces are converted to a low dimensional projection space to form a smooth manifold surface. Subsequent faces are automatically detected, facial feature points are extracted and mapped onto the low dimensional projection surface, where regression models robustly estimate pose. The benefit of using facial feature points for manifold learning over raw facial images is demonstrated by a variety of experiments. Linear, nonlinear, unsupervised and supervised methods are considered including Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), Locally Linear Embedding (LLE), Isomap, unsupervised Locality Preserving Projections (LPP), and supervised LPP (SLPP).

This paper presents a robust approach to performing facial pose estimation by examining the behavior of key facial features over a wide range of poses. Such methods are useful in intelligent vision systems for entertainment, human computer interaction, and security. In our approach, faces of varying pose are automatically detected, eyes and mouth are located and an active shape model is superimposed. A facial pose estimator is developed using predictor models based on the position, size, and symmetry of facial features. By modeling these predictors over pose positions with varying yaw and pitch, excellent results are obtained without the need for complex computationally intensive methods.