In this paper we propose a novel human-centered approach for detecting forgery in face images, using dynamic prototypes as a form of visual explanations. Currently, most state-of-the-art deepfake detections are based on black-box models that process videos frame-by-frame for inference, and few closely examine their temporal inconsistencies. However, the existence of such temporal artifacts within deepfake videos is key in detecting and explaining deepfakes to a supervising human. To this end, we propose Dynamic Prototype Network (DPNet) - an interpretable and effective solution that utilizes dynamic representations (i.e., prototypes) to explain deepfake temporal artifacts. Extensive experimental results show that DPNet achieves competitive predictive performance, even on unseen testing datasets such as Google's DeepFakeDetection, DeeperForensics, and Celeb-DF, while providing easy referential explanations of deepfake dynamics. On top of DPNet's prototypical framework, we further formulate temporal logic specifications based on these dynamics to check our model's compliance to desired temporal behaviors, hence providing trustworthiness for such critical detection systems.

Machine learning transparency calls for interpretable explanations of how inputs relate to predictions. Feature attribution is a way to analyze the impact of features on predictions. Feature interactions are the contextual dependence between features that jointly impact predictions. There are a number of methods that extract feature interactions in prediction models; however, the methods that assign attributions to interactions are either uninterpretable, model-specific, or non-axiomatic. We propose an interaction attribution and detection framework called Archipelago which addresses these problems and is also scalable in real-world settings. Our experiments on standard annotation labels indicate our approach provides significantly more interpretable explanations than comparable methods, which is important for analyzing the impact of interactions on predictions. We also provide accompanying visualizations of our approach that give new insights into deep neural networks.

Low-rank tensor factorization has been widely used for many real world tensor completion problems. While most existing factorization models assume a multilinearity relationship between tensor entries and their corresponding factors, real world tensors tend to have more complex interactions than multilinearity. In many recent works, it is observed that multilinear models perform worse than nonlinear models. We identify one potential reason for this inferior performance: the nonlinearity inside data obfuscates the underlying low-rank structure such that the tensor seems to be a high-rank tensor. Solving this problem requires a model to simultaneously capture the complex interactions and preserve the low-rank structure. In addition, the model should be scalable and robust to missing observations in order to learn from large yet sparse real world tensors. We propose a novel convolutional neural network (CNN) based model, named CoSTCo (Convolutional Sparse Tensor Completion). Our model leverages the expressive power of CNN to model the complex interactions inside tensors and its parameter sharing scheme to preserve the desired low-rank structure. CoSTCo is scalable as it does not involve computation- or memory- heavy tasks such as Kronecker product. We conduct extensive experiments on several real world large sparse tensors and the experimental results show that our model clearly outperforms both linear and nonlinear state-of-the-art tensor completion methods.

Neural networks are known to model statistical interactions, but they entangle the interactions at intermediate hidden layers for shared representation learning. We propose a framework, Neural Interaction Transparency (NIT), that disentangles the shared learning across different interactions to obtain their intrinsic lower-order and interpretable structure. This is done through a novel regularizer that directly penalizes interaction order. We show that disentangling interactions reduces a feedforward neural network to a generalized additive model with interactions, which can lead to transparent models that perform comparably to the state-of-the-art models. NIT is also flexible and efficient; it can learn generalized additive models with maximum $K$-order interactions by training only $O(1)$ models.

We explored gesture recognition applied to the problem of classifying natural physical bullying behaviors by children. To capture natural bullying behavior data, we developed a humanoid robot that used hand-coded gesture recognition to identify basic physical bullying gestures and responded by explaining why the gestures were inappropriate. Children interacted with the robot by trying various bullying behaviors, thereby allowing us to collect a natural bullying behavior dataset for training the classifiers. We trained three different sequence classifiers using the collected data and compared their effectiveness at classifying different types of common physical bullying behaviors. Overall, Hidden Conditional Random Fields achieved the highest average F1 score (0.645) over all tested gesture classes.

We introduce the concept of the Temporal Thumbnail, used to quickly convey information about the amount of time spent viewing specific areas of a virtual 3D model. Temporal Thumbnails allow for large amounts of time-based information collected from model viewing sessions to be rapidly visualized by collapsing the time dimension onto the space of the model, creating a characteristic impression of the overall interaction. We describe three techniques that implement the Temporal Thumbnail concept and present a study comparing these techniques to more traditional video and storyboard representations. The results suggest that Temporal Thumbnails have potential as an effective technique for quickly analyzing large amounts of viewing data. Practical and theoretical issues for visualization and representation are also discussed.

We review the Boom Chameleon, a novel input/output device consisting of a flat-panel display mounted on a tracked mechanical armature. The display acts as a physical window into 3D virtual environments, through which a one-to-one mapping between real and virtual space is preserved. The Boom Chameleon is further augmented with a touch-screen and a microphone/speaker combination. We created a 3D annotation application that exploits this unique configuration in order to simultaneously capture viewpoint, voice and gesture information. Results of an informal user study show that the Boom Chameleon annotation facilities have the potential to be an effective, and intuitive system for reviewing 3D designs.

We introduce the Boom Chameleon, a novel input/output device consisting of a flat-panel display mounted on a tracked mechanical boom. The display acts as a physical window into 3D virtual environments, through which a one-to-one mapping between real and virtual space is preserved. The Boom Chameleon is further augmented with a touch-screen and a microphone/speaker combination. We present a 3D annotation application that exploits this unique configuration in order to simultaneously capture viewpoint, voice and gesture information. Design issues are discussed and results of an informal user study on the device and annotation software are presented. The results show that the Boom Chameleon annotation facilities have the potential to be an effective, easy to learn and operate 3D design review system.