Person discovery in the absence of prior identity knowledge requires accurate association of visual and auditory cues. In broadcast data, multimodal analysis faces additional challenges due to narrated voices over muted scenes or dubbing in different languages. To address these challenges, we define and analyze the problem of dubbing detection in broadcast data, which has not been explored before. We propose a method to represent the temporal relationship between the auditory and visual streams. This method consists of canonical correlation analysis to learn a joint multimodal space, and long short term memory (LSTM) networks to model cross-modality temporal dependencies. Our contributions also include the introduction of a newly acquired dataset of face-speech segments from TV data, which we have made publicly available. The proposed method achieves promising performance on this real world dataset as compared to several baselines.

This paper describes a novel method called Deep Dynamic Neural Networks (DDNN) for multimodal gesture recognition. A semi-supervised hierarchical dynamic framework based on a Hidden Markov Model (HMM) is proposed for simultaneous gesture segmentation and recognition where skeleton joint information, depth and RGB images, are the multimodal input observations. Unlike most traditional approaches that rely on the construction of complex handcrafted features, our approach learns high-level spatio-temporal representations using deep neural networks suited to the input modality: a Gaussian-Bernouilli Deep Belief Network (DBN) to handle skeletal dynamics, and a 3D Convolutional Neural Network (3DCNN) to manage and fuse batches of depth and RGB images. This is achieved through the modeling and learning of the emission probabilities of the HMM required to infer the gesture sequence. This purely data driven approach achieves a Jaccard index score of 0.81 in the ChaLearn LAP gesture spotting challenge. The performance is on par with a variety of state-of-the-art hand-tuned feature-based approaches and other learning-based methods, therefore opening the door to the use of deep learning techniques in order to further explore multimodal time series data.

To boost up power of unsupervised feature learning and deep learning, there has been a great effort in optimizing network structure to learn more efficient high level features. It is crucial for a network to have a sufficient amount of learnable parameters yet still be able to capture in variances in data. In this paper, the authors propose spatial boosting networks, which employ convolutional feature learning networks as learning components. Each component in a network is assigned to a certain spatial region. This allows the network learn more adaptive features for each region. In order to make spatial boosting networks to capture relationship between regions of the visual field, we also propose convolutional pooling procedure. By expanding pooling scope into overlapping regions, we expect the features pooled in higher level to be more robust to noises and more invariant to transformation. Experiments show that using spatial boosting networks boosts up accuracy up to 3% from conventional approaches in standard datasets CIFAR and STL. Moreover, these results are competitive in comparison with other methods by using only a basic feature learning algorithm.

Lateral connection is a fundamental element of human neural networks which enables sparse learning and topographical order in feature maps. Due to high complexity and computational cost, computer scientists tend to simplify it in practical implementations. To utilize the simplicity of traditional networks while preserving the effects of interconnections, the authors employ numerical filters in unsupervised learning networks. These filters suppress low activations and decorrelate high ones, which are similar to how inhibitory lateral connections behave. Inhibitory networks outperform conventional approach in both standard datasets CIFAR-10 and STL-10. Our method also yields competitive results in comparison with other single-layer unsupervised networks. Furthermore, it is promising to apply inhibitory networks into deep learning systems for complex recognition problem.