The ability to discover abstract physical concepts and understand how they work in the world through observing lies at the core of human intelligence. The acquisition of this ability is based on compositionally perceiving the environment in terms of objects and relations in an unsupervised manner. Recent approaches learn object-centric represen-tations and capture visually observable concepts of objects, e.g., shape, size, and location. In this paper, we take a step forward and try to discover and represent intrinsic physical concepts such as mass and charge. We introduce the PHYsi-cal Concepts Inference NEtwork (PHYCINE), a system that infers physical concepts in different abstract levels with-out supervision. The key insights underlining PHYCINE are two-fold, commonsense knowledge emerges with pre-diction, and physical concepts of different abstract levels should be reasoned in a bottom-up fashion. Empirical eval-uation demonstrates that variables inferred by our system work in accordance with the properties of the corresponding physical concepts. We also show that object representations containing the discovered physical concepts variables could help achieve better performance in causal reasoning tasks, i.e., ComPhy.

Bilinear models are very powerful in multimodal fusion tasks like Visual Question Answering. The predominant bilinear methods can all be seen as a kind of tensor-based decomposition operation that contains a key kernel called “core tensor.” Current approaches usually focus on reducing the computation complexity by applying low-rank constraint on the core tensor. In this article, we propose a novel bilinear architecture called Block Term Decomposition Pooling (BTDP), which not only maintains the advantages of previous bilinear methods but also conducts sparse bilinear interactions between modalities. Our method is based on Block Term Decompositions theory of tensor, which will result in a sparse and learnable block-diagonal core tensor for multimodal fusion. We prove that using such a block-diagonal core tensor is equivalent to conducting many “tiny” bilinear operations in different feature spaces. Thus, introducing sparsity into the bilinear operation can significantly increase the performance of feature fusion and improve VQA models. What is more, our BTDP is very flexible in design. We develop several variants of BTDP and discuss the effects of the diagonal blocks of core tensor. Extensive experiments on two challenging VQA-v1 and VQA-v2 datasets show that our BTDP method outperforms current bilinear models, achieving state-of-the-art performance.

Using dropout in Visual Question Answering (VQA) is a common practice to prevent overfitting. However, in multi-path networks, the current way to use dropout may cause two problems: the co-adaptations of neurons and the explosion of output variance. In this paper, we propose the coherent dropout and the siamese dropouy to solve the two problems, respectively. Specifically, in coherent dropout, all relevant dropout layers in multiple paths are forced to work coherently to maximize the ability of preventing neuron co-adaptations. We show that the coherent dropout is simple in implementation but very effective to overcome overfitting. As for the explosion of output variance, we develop a siamese dropout mechanism to explicitly minimize the difference between the two output vectors produced from the same input data during training phase. Such mechanism can reduce the gap between training and inference phases and make the VQA model more robust. Extensive experiments are conducted to verify the effectiveness of coherent dropout and siamese dropout. And the results also show that our methods can bring additional improvements on the state-of-the-art VQA models.

This paper extends previous work automatically detecting stereotypical motor movements (SMM) in individuals on the autism spectrum. Using three-axis accelerometer data obtained through wearable wireless sensors, we compare recognition results for two different classifiers -- Support Vector Machine and Decision Tree -- in combination with different feature sets based on time-frequency characteristics of accelerometer data. We use data collected from six individuals on the autism spectrum who participated in two different studies conducted three years apart in classroom settings, and observe an average accuracy across all participants over time ranging from 81.2% (TPR: 0.91; FPR: 0.21) to 99.1% (TPR: 0.99; FPR: 0.01) for all combinations of classifiers and feature sets. We also provide analyses of kinematic parameters associated with observed movements in an attempt to explain classifier-feature specific performance. Based on our results, we conclude that real-time, person-dependent, adaptive algorithms are needed in order to accurately and consistently measure SMM automatically in individuals on the autism spectrum over time in real-word settings.