Brain activity is a dynamic combination of different sensory responses and thus brain activity/state is continuously changing over time. However, the brain's dynamical functional states recognition at fast time-scales in task fMRI data have been rarely explored. In this paper, we propose a novel 5-layer deep sparse recurrent neural network (DSRNN) model to accurately recognize the brain states across the whole scan session. Specifically, the DSRNN model includes an input layer, one fully-connected layer, two recurrent layers, and a softmax output layer. The proposed framework has been tested on seven task fMRI data sets of Human Connectome Project. Extensive experiment results demonstrate that the proposed DSRNN model can accurately identify the brain's state in different task fMRI data sets and significantly outperforms other auto-correlation methods or non-temporal approaches in the dynamic brain state recognition accuracy. In general, the proposed DSRNN offers a new methodology for basic neuroscience and clinical research.

Task-based functional magnetic resonance imaging (tfMRI) has been widely used to study functional brain networks under task performance. Modeling tfMRI data is challenging due to at least two problems: the lack of the ground truth of underlying neural activity and the highly complex intrinsic structure of tfMRI data. To better understand brain networks based on fMRI data, data-driven approaches have been proposed, for instance, independent component analysis (ICA) and sparse dictionary learning (SDL). However, both ICA and SDL only build shallow models, and they are under the strong assumption that original fMRI signal could be linearly decomposed into time series components with their corresponding spatial maps. As growing evidence shows that human brain function is hierarchically organized, new approaches that can infer and model the hierarchical structure of brain networks are widely called for. Recently, deep convolutional neural network (CNN) has drawn much attention, in that deep CNN has proven to be a powerful method for learning high-level and mid-level abstractions from low-level raw data. Inspired by the power of deep CNN, in this paper, we developed a new neural network structure based on CNN, called deep convolutional auto-encoder (DCAE), in order to take the advantages of both data-driven approach and CNN's hierarchical feature abstraction ability for the purpose of learning mid-level and high-level features from complex, large-scale tfMRI time series in an unsupervised manner. The DCAE has been applied and tested on the publicly available human connectome project tfMRI data sets, and promising results are achieved.

What makes a good movie trailer? It's a big challenge to answer this question because of the complexity of multimedia in both low level sensory features and high level semantic features. However, human perception and reactivity could be straightforward evidence for evaluation. Modern Electro-encephalography (EEG) technology provides measurement of consequential brain neural activity to external stimuli. Meanwhile, visual perception and attention could be captured and interpreted by Eye Tracking technology. Intuitively, simultaneous EEG and Eye Tracker recording of human audience with multimedia stimuli could bridge the gap between human comprehension and multimedia analysis, and provide a new way for movie trailer evaluation. In this paper, we propose a novel platform to simultaneously record EEG and eye movement for participants with video stimuli by integrating 256-channel EEG, Eye Tracker and video display device as a system. Based on the proposed system a novel experiment has been designed, in which independent and joint features of EEG and Eye tracking data were mined to evaluate the movie trailer. Our analysis has shown interesting features that are corresponding with trailer quality and video shoot changes.