Abstract With the global aging problem becoming more and more serious, the initial screening for Alzheimer's disease (AD) will become increasingly important. We understand that facial expressions are related to the severity of dementia, but there is no face‐related data in the existing Alzheimer's dataset. This article attempts to establish a facial video‐based AD recognition dataset through a human‐computer interaction method. This interactive task was designed for AD in attention, execution, visual space ability, facial apraxia, and facial changes in task success and failure. Using this task as the collection method, the final dataset includes 102 faces video data, specific task scores, and emotional self‐evaluation. For baseline evaluation, the improved local binary pattern on three orthogonal planes and RF were employed respectively for feature extraction and classification with the 5‐fold cross‐validation method. The best performance was 76.00% for 3‐class classification. In addition, a frame attention network based on fine‐grained local region localization was proposed, which improved the accuracy of cognitive classification to 84.45%. Finally, the analysis was conducted for the association of expressions with cognition and emotion in the AD dataset. This study aims to solve the current lack of standards for AD in the field of facial recognition and contribute to future research and clinical applications.

Regression modeling is a significant aspect of multi-modal continuous dimensional emotion recognition. Despite the developments of this domain, one of the limitations that severely impede the application of emotion recognition is that most methods utilized for regression modeling merely capture the temporal information. Motivated by this, we propose a new branch feature fusion framework BF-Net, whose core idea is to deeply combine local features captured by convolutional neural networks and temporal dependencies captured by long-short-term memory recurrent neural networks. The framework consists of two main components: two-branch structure and fusion of branches. The former captures local features and temporal dependencies respectively in an effective way. Besides, the latter utilizes an attention mechanism to fuse the features on a deep level. In this way, the framework achieves full utilization of the local features and temporal dependencies. The experiments on ULM-TSST dataset show that the proposed method is competitive or superior to the state-of-the-art works.

Abstract Facial micro‐expression is a superposition of micro‐expression features and identity information of a subject. For the problem of identity information interference in micro‐expression recognition, this study proposes a new method for facial micro‐expression recognition by de‐identity information, called two‐stream difference network (TSDN). First, a two‐stream encoder‐decoder network is trained by a convolutional neural network, where the input of the micro‐expression stream is a micro‐expression image, and the identity stream is a facial identity image. The micro‐expression image is the apex image, and the identity image is the onset image in the micro‐expression sequence. The identity information and micro‐expression features are recorded in the intermediate layer of the micro‐expression stream, while the intermediate layer of the identity stream contains only the identity information of a subject. Then, the identity information is removed by the difference network, but micro‐expression features are stored in the intermediate layer of the micro‐expression stream. Given the sequence of the micro‐expressions, the TSDN model of de‐identity information learns the difference that stores in the expression stream. Two public spontaneous facial micro‐expression data sets (SMIC and CASME II) are employed in our experiments. The experiment results show that our model can achieve a superior performance in micro‐expression recognition.

Facial micro-expressions are short and imperceptible expressions that involuntarily reveal the true emotions that a person may be attempting to suppress, hide, disguise, or conceal. Such expressions can reflect a person's real emotions and have a wide range of application in public safety and clinical diagnosis. The analysis of facial micro-expressions in video sequences through computer vision is still relatively recent. In this research, a comprehensive review on the topic of spotting and recognition used in microexpression analysis databases and methods, is conducted, and advanced technologies in this area are summarized. In addition, we discuss challenges that remain unresolved alongside future work to be completed in the field of micro-expression analysis.

To reduce the impact of low intensity on the spot micro-expressions in long video sequences when facial microexpressions occur, this paper presents a method based on Local Bilinear Convolutional Neural Network (LBCNN) for spotting macro- and micro-expressions in long videos sequences. Considering the low intensity of facial micro-expressions, and the occurrence of micro-expressions is only related to the local area of the facial, we turn the micro-expression spot in long videos sequences into a fine-grained image recognition. Bilinear Convolutional Neural Network (BCNN) has proven to be effective in fine-grained image recognition. Therefore, we use the BCNN structure to extract the global and local features of the face area of each frame of the image in the long video sequence and obtains the final classification result by fusing the global and local features. The metric F1-scores of our proposed methods were evaluated on the CAS(ME)2and SAMM Long Videos dataset of the Third Facial Micro-Expression Grand Challenge (MEGC 2020). For the CAS(ME)2, the overall F1-scores are 0.0595 for macro- and micro-expressions; for SAMM Long Videos, the overall F1-scores are 0.0813 for macro- and microexpressions. The experiments show that our method achieved superiorly higher results than the baseline method (MDMD) provided. https://github.com/panhang1023/MEGC2020.