Film grammar is often used to invoke certain emotional experiences from audiences through changing visual, speech, and musical elements of videos. Such film grammar, referred to as domain knowledge, is of great importance for video affective content analysis but has not been thoroughly examined in research. In this paper, we propose an improved method for emotion recognition and regression from videos through exploring domain knowledge. We first investigate the domain knowledge of visual, speech, and musical elements, and infer probabilistic dependencies between elements and emotions from the summarized film grammar. Then, we transfer the summarized dependencies between elements and emotions as constraints, and formulate video affective content analysis, including both emotion recognition and emotion regression from video content, as a constrained optimization problem. Experiments on the LIRIS-ACCEDE database, the FilmStim database, and the DEAP database demonstrate that the proposed video affective content analysis method can successfully leverage well-established film grammar to improve emotion recognition and regression from video content.

Current multimodal deep learning approaches rarely explicitly exploit the dependencies inherent in multiple labels, which are crucial for multimodal multi-label classification. In this paper, we propose a multimodal deep learning approach for multi-label classification. Specifically, we introduce deep networks for feature representation learning and construct classifiers with the objective function which is constrained with dependencies among both labels and modals. We further propose effective training algorithm to learn deep networks and classifiers jointly. Thus, we explicitly leverage the relations among labels and modals to facilitate multimodal multi-label classification. Experiments of multi-label classification and cross-modal retrieval on the Pascal VOC dataset and the La-belMe dataset demonstrate the effectiveness of the proposed approach.

The well-established film grammar is often used to change visual and audio elements of videos to invoke audiences' emotional experience. Such film grammar, referred to as domain knowledge, is crucial for affective video content analyses, but has not been thoroughly explored yet. In this paper, we propose a novel method to analyze video affective content through exploring domain knowledge. Specifically, take visual elements as an example, we first infer probabilistic dependencies between visual elements and emotions from the summarized film grammar. Then, we transfer the domain knowledge as constraints, and formulate affective video content analyses as a constrained optimization problem. Experiments on the LIRIS-ACCEDE database and the DEAP database demonstrate that the proposed affective content analyses method can successfully leverage well-established film grammar for better emotion classification from video content.

The main stream of current research of emotion recognition from electroencephalogram (EEG) signals infers users' emotion directly from EEG signals, without considering users' profile, which carries crucial information for emotion recognition. To address this, we propose a novel approach of emotion recognition from EEG signals enhanced by users' profile, which is only required during training. Specifically, we propose to use discriminative Restricted Boltzmann Machine (DRBM) to capture the inherent relationships among users' profile, EEG signals and emotion labels. During training, the proposed RBM classifier learns the mapping function from EEG signals and users' profile to emotion labels through discriminative learning algorithm. During testing, users' emotion can be recognized from EEG signals through marginalizing over users' profile, since RBM is a generative model. Experimental results on three benchmark databases, i.e. the DEAP database, the MAHNOB-HCI database and the USTC-ERVS database, demonstrate the effectiveness of the proposed approach in leveraging users' profile for enhancing emotion recognition from EEG signals and the superior emotion recognition performance of the proposed approach over existing approaches.