Affective Computing (AC) aims to recognize and interpret human behavioral states, but modern deep models often lack transparent decision processes. Existing explainable AI approaches in AC are typically post-hoc or map-based, highlighting important regions or features without providing structured, concept-level explanations aligned with domain knowledge. Feature-based models offer more interpretable representations but are decoupled from end-to-end learning and often underperform. In this work, we propose the Attention-Guided Concept Model (AGCM), a concept-based framework that links predictions to predefined affective concepts and, for visual inputs, their spatial evidence. AGCM jointly learns concept representations and task predictions, enabling the model to identify what concepts contribute to a prediction and where supporting evidence is located. We evaluate AGCM on standard facial expression recognition benchmarks and a multimodal engagement estimation task. Results show that AGCM achieves competitive performance with strong black-box models while providing structured concept-level and spatially grounded interpretations. Video demonstration and pretrained models are available at https://github.com/xinyuli-uofg/agcm.

Stress detection is essential for advancing mental health diagnostics, optimizing workplace well-being, and enhancing cognitive performance monitoring. Existing approaches rely mostly on unimodal signals, such as physiological data or facial expressions, which is insufficient to capture the complex and multifaceted nature of stress responses. Despite the observational evidence of the importance of body gestures in affective computing applications, body expressions has not been studied in the domain of automatic stress detection. In this paper, we introduce a novel multimodal framework for stress detection that combines physiological signals (BVP, EDA) with behavioral features extracted from facial expressions and body gestures analysis. We demonstrate that our multimodal system outperforms state-of-the-art stress detection models, when evaluated on the UBFC-Phys dataset. Detailed feature importance analysis showed the significance and complementary nature of face and body modalities in stress detection. This work establishes a new benchmark in stress classification by demonstrating the effectiveness of multimodal fusion — especially the contribution of gesture and posture cues — in improving both accuracy and robustness.

Black-box Machine Learning (ML) models are widely used in Affective Computing applications, including in high-stake tasks, such as autonomous driving or mental health applications. Despite their state-of-the-art performance, black-box models are usually criticized as non-interpretable as they only produce a final prediction without providing insight into the decision-making process. Previous research on explainable affective computing applications mainly focuses on post-hoc techniques or reverting to traditional ML approaches, which may compromise either the explainability or the performance. Recently, concept-based models have demonstrated great success in general object classification tasks by training the ML model to learn both the classification label as well as the underlying concepts that contribute to the prediction. However, the experimental scenario is different in affective computing applications, for example, facial expression recognition or emotion detection, where it is challenging to select meaningful concepts because of the inherent uncertainty of the tasks. In this paper, we propose a novel concept-based modeling framework for Affective Computing applications, using Facial Expression Recognition (FER) as a benchmark example. Our work is the first to present a Concept-based FER approach and explore concept selection for this application area leveraging concept-based models, including Concept Bottleneck Models (CBMs) and Concept Embedding Models (CEMs). Experimental evaluation on two benchmark datasets: Aff-Wild2 and AffectNet, shows that our proposed framework provides robust explanations for each prediction while maintaining competitive performance compared to state-of-the-art black-box models. We believe the proposed framework paves the way for future interpretable investigations into downstream applications in human behavior understanding.

Abstract Advances in animal motion tracking and pose recognition have been a game changer in the study of animal behavior. Recently, an increasing number of works go ‘deeper’ than tracking, and address automated recognition of animals’ internal states such as emotions and pain with the aim of improving animal welfare, making this a timely moment for a systematization of the field. This paper provides a comprehensive survey of computer vision-based research on recognition of pain and emotional states in animals, addressing both facial and bodily behavior analysis. We summarize the efforts that have been presented so far within this topic—classifying them across different dimensions, highlight challenges and research gaps, and provide best practice recommendations for advancing the field, and some future directions for research.

Psychological distress is a significant and growing issue in society. In particular, depression and anxiety are leading causes of disability that often go undetected or late-diagnosed. Automatic detection, assessment, and analysis of behavioural markers of psychological distress can help improve identification and support prevention and early intervention efforts. Compared to modalities such as face, head, and vocal, research investigating the use of the body modality for these tasks is relatively sparse, which is partly due to the limited available datasets and difficulty in automatically extracting useful body features. To enable our research, we have collected and analyzed a new dataset containing full body videos for interviews and self-reported distress labels. We propose a novel approach to automatically detect self-adaptors and fidgeting, a subset of self-adaptors that has been shown to correlate with psychological distress. We perform analysis on statistical body gestures and fidgeting features to explore how distress levels affect behaviors. We then propose a multi-modal approach that combines different feature representations using Multi-modal Deep Denoising Auto-Encoders and Improved Fisher Vector Encoding. We demonstrate that our proposed model, combining audio-visual features with detected fidgeting behavioral cues, can successfully predict depression and anxiety in the dataset.

MOTIVATION: Imaging single molecules has emerged as a powerful characterization tool in the biological sciences. The detection of these under various noise conditions requires the use of algorithms that are dependent on the end-user inputting several parameters, the choice of which can be challenging and subjective. RESULTS: In this work, we propose DeepSinse, an easily trainable and useable deep neural network that can detect single molecules with little human input and across a wide range of signal-to-noise ratios. We validate the neural network on the detection of single bursts in simulated and experimental data and compare its performance with the best-in-class, domain-specific algorithms. AVAILABILITYAND IMPLEMENTATION: Ground truth ROI simulating code, neural network training, validation code, classification code, ROI picker, GUI for simulating, training and validating DeepSinse as well as pre-trained networks are all released under the MIT License on www.github.com/jdanial/DeepSinse. The dSTORM dataset processing code is released under the MIT License on www.github.com/jdanial/StormProcessor. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Psychological distress is a significant and growing issue in society. Automatic detection, assessment, and analysis of such distress is an active area of research. Compared to modalities such as face, head, and vocal, research investigating the use of the body modality for these tasks is relatively sparse. This is, in part, due to the lack of available datasets and difficulty in automatically extracting useful body features. Recent advances in pose estimation and deep learning have enabled new approaches to this modality and domain. We propose a novel method to automatically detect self-adaptors and fidgeting, a subset of self-adaptors that has been shown to be correlated with psychological distress. We also propose a multi-modal approach that combines different feature representations using Multi-modal Deep Denoising Auto-Encoders and Improved Fisher Vector encoding. We also demonstrate that our proposed model, combining audio-visual features with automatically detected fidgeting behavioral cues, can successfully predict distress levels in a dataset labeled with self-reported anxiety and depression levels. To enable this research we introduce a new dataset containing full body videos for short interviews and self-reported distress labels.

Pain in farm animals harms the economics of farming and affects animal welfare. However, prey animals tend to not openly express signs of weakness, making the pain assessment process difficult. We propose a novel hierarchical model for disease progression evaluation, adapted for a wide range of head poses, according to which relevant information is extracted. A fine-tuned CNN is applied for face detection, followed by a CNN-based pose estimation and pose-informed landmark location method. Then multi-modal features are extracted, combining the appearance of regions-of-interest, described using a Histogram of Oriented Gradients, with geometric features and the pose values, leading to a binary Support Vector Machine classifier. To evaluate the efficiency of the complete pipeline, videos of the same sheep recorded at initial and advanced stages of treatment were tested, showing a decrease in the average pain score detected. The pain evaluation method significantly outperformed the existing state-of-the-art approach, being the first to apply a pose-based feature extraction in sheep pain detection.

Automatic detection of pain and protective behaviour can help chronic pain patients to get proper assistance and helpful treatment with the help of medical professionals. Using the EmoPain datasetwe study how autoencoder-based and attention-based deep learning models can be used to automatically detect pain and protective behavior that is usually associated with it. We propose a deep learning architecture called Autoencoder-LSTM-Attention-Net (ALANet), which can improve the automatic detection of pain and protective behaviors. Through comparative experiments with other machine learning models trained on the EmoPain dataset, we found that by using a combination of autoencoder and attention mechanisms, we can not only improve the recognition performance, but also greatly increase the speed of training the model. In addition, we analyse the effect of extracting temporal information from each body part separately compared to all body parts combined.

Current methods for mental disorder recognition mostly depend on clinical interviews and self-reported scores that can be highly subjective. Building an automatic recognition system can help in early detection of symptoms and providing insights into the biological markers for diagnosis. It is, however, a challenging task as it requires taking into account indicators from different modalities, such as facial expressions, gestures, acoustic features and verbal content. To address this issue, we propose a general-purpose multimodal deep learning framework, in which multiple modalities - including acoustic, visual and textual features - are processed individually with the cross-modality correlation considered. Specifically, a Multimodal Deep Denoising Autoencoder (multi- DDAE) is designed to obtain multimodal representations of audio-visual features followed by the Fisher Vector encoding which produces session-level descriptors. For textual modality, a Paragraph Vector (PV) is proposed to embed the transcripts of interview sessions into document representations capturing cues related to mental disorders. Following an early fusion strategy, both audio-visual and textual features are then fused prior to feeding them to a Multitask Deep Neural Network (DNN) as the final classifier. Our framework is evaluated on the automatic detection of two mental disorders: bipolar disorder (BD) and depression, using two datasets: Bipolar Disorder Corpus (BDC) and the Extended Distress Analysis Interview Corpus (E-DAIC), respectively. Our experimental evaluation results showed comparable performance to the state-of-the-art in BD and depression detection, thus demonstrating the effective multimodal representation learning and the capability to generalise across different mental disorders.

Landmark localisation is a vital step in automatic analysis of facial expressions of animals. Head motion is one of the most challenging problems for face alignment for humans and animals. For animals this is exacerbated by the increased amounts of self-occlusion resulting from variations in head pose. In this paper, we present a novel model for detection of an extensive set of facial landmarks for sheep. A dataset of 850 sheep facial images, annotated with a 25 facial landmark scheme and occlusion information, is introduced: the Sheep Facial Landmarks in the Wild (SFLW) dataset, including a wide range of variations in head-pose and occlusion. Data augmentation techniques are introduced using thin-plate-spline warping and negatively correlated augmentation to boost representation of extreme head poses. We then present a novel pose-informed landmark localisation method based on a fine-tuned CNN model for human head pose estimation. This method is shown to significantly outperform the existing state-of-the-art approach on the introduced SFLW dataset and the viability of the technique for real-world use is demonstrated through the implementation of a near real-time video pipeline.

“Serious games” are becoming extremely relevant to individuals who have specific needs, such as children with an autism spectrum condition (ASC). Often, individuals with an ASC have difficulties in interpreting verbal and nonverbal communication cues during social interactions. The ASC-Inclusion EU-FP7 funded project aims to provide children who have an ASC with a platform to learn emotion expression and recognition, through play in the virtual world. In particular, the ASC-Inclusion platform focuses on the expression of emotion via facial, vocal, and bodily gestures. The platform combines multiple analysis tools, using onboard microphone and webcam capabilities. The platform utilizes these capabilities via training games, text-based communication, animations, video, and audio clips. This paper introduces current findings and evaluations of the ASC-Inclusion platform and provides detailed description for the different modalities.

Driver fatigue is one of the main causes of road accidents. It is essential to develop a reliable driver drowsiness detection system which can alert drivers without disturbing them and is robust to environmental changes. This paper explores yawning behaviour as a sign of drowsiness in spontaneous expressions of drowsy drivers in simulated driving scenarios. We analyse a labelled dataset of videos of sleep-deprived versus alert drivers and demonstrate the correlation between hand-over-face touches, face occlusions and yawning. We propose that face touches can be used as a novel cue in automated drowsiness detection alongside yawning and eye behaviour. Moreover, we present an automatic approach to detect yawning based on extracting geometric and appearance features of both mouth and eye regions. Our approach successfully detects both hand-covered and uncovered yawns with an accuracy of 95%. Ultimately, our goal is to use these results in designing a hybrid drowsiness-detection system.

This paper presents a multimodal approach to in-vehicle classification of driver glances. Driver glance is a strong predictor of cognitive load and is a useful input to many applications in the automotive domain. Six descriptive glance regions are defined and a classifier is trained on video recordings of drivers from a single low-cost camera. Visual features such as head orientation, eye gaze and confidence ratings are extracted, then statistical methods are used to perform failure analysis and calibration on the visual features. Non-visual features such as steering wheel angle and indicator position are extracted from a RaceLogic VBOX system. The approach is evaluated on a dataset containing multiple 60 second samples from 14 participants recorded while driving in a natural environment. We compare our multimodal approach to separate unimodal approaches using both Support Vector Machine (SVM) and Random Forests (RF) classifiers. RF Mean Decrease in Gini Index is used to rank selected features which gives insight into the selected features and improves the classifier performance. We demonstrate that our multimodal approach yields significantly higher results than unimodal approaches. The final model achieves an average F1score of 70.5% across the six classes.

Assessing pain levels in animals is a crucial, but time-consuming process in maintaining their welfare. Facial expressions in sheep are an efficient and reliable indicator of pain levels. In this paper, we have extended techniques for recognising human facial expressions to encompass facial action units in sheep, which can then facilitate automatic estimation of pain levels. Our multi-level approach starts with detection of sheep faces, localisation of facial landmarks, normalisation and then extraction of facial features. These are described using Histogram of Oriented Gradients, and then classified using Support Vector Machines. Our experiments show an overall accuracy of 67% on sheep Action Units classification. We argue that with more data, our approach on automated pain level assessment can be generalised to other animals.

Current navigational systems rarely consider generic road landmarks in their navigation instructions, which can lead to mistakes, frustration, and distraction. However, automatic detection of road landmarks is difficult, as current approaches to object detection focus either on out-of-context objects which have special characteristics or on very specific domains. This work presents a future direction for a user-friendly navigational system based on state-of-the-art computer vision techniques that use deep learning for object detection. We propose an automatic hierarchical approach for detecting and classifying a set of static and dynamic road landmarks that would be useful in automatic navigational systems. We further demonstrate a set of optimisations that improve performance and accuracy of the basic system. We evaluate our approach on a natural, 'in-the-wild' dataset to determine how well it handles natural automotive input. Finally, we demonstrate a use-case for our system that extracts information about a vehicle's location and intention.

One of the main factors that limit the accuracy of facial analysis systems is hand occlusion. As the face becomes occluded, facial features are lost, corrupted, or erroneously detected. Hand-over-face occlusions are considered not only very common but also very challenging to handle. However, there is empirical evidence that some of these hand-over-face gestures serve as cues for recognition of cognitive mental states. In this article, we present an analysis of automatic detection and classification of hand-over-face gestures. We detect hand-over-face occlusions and classify hand-over-face gesture descriptors in videos of natural expressions using multi-modal fusion of different state-of-the-art spatial and spatio-temporal features. We show experimentally that we can successfully detect face occlusions with an accuracy of 83%. We also demonstrate that we can classify gesture descriptors ( hand shape , hand action , and facial region occluded ) significantly better than a naïve baseline. Our detailed quantitative analysis sheds some light on the challenges of automatic classification of hand-over-face gestures in natural expressions.

Perception of emotion through facial expressions and head motion is of interest to both psychology and affective computing researchers. However, very little is known about the importance of each modality individually, as they are often treated together rather than separately. We present a study which isolates the effect of head motion from facial expression in the perception of complex emotions in videos. We demonstrate that head motions carry emotional information that is complementary rather than redundant to the emotion content in facial expressions. Finally, we show that emotional expressivity in head motion is not limited to nods and shakes and that additional gestures (such as head tilts, raises and general amount of motion) could be beneficial to automated recognition systems.

One of the main factors that limit the accuracy of facial analysis systems is hand occlusion. As the face becomes occluded, facial features are either lost, corrupted or erroneously detected. Hand-over-face occlusions are considered not only very common but also very challenging to handle. Moreover, there is empirical evidence that some of these hand-over-face gestures serve as cues for recognition of cognitive mental states. In this paper, we detect hand-over-face occlusions and classify hand-over-face gesture descriptors in videos of natural expressions using multi-modal fusion of different state-of-the-art spatial and spatio-temporal features. We show experimentally that we can successfully detect face occlusions with an accuracy of 83%. We also demonstrate that we can classify gesture descriptors (hand shape, hand action and facial region occluded) significantly higher than a naive baseline. To our knowledge, this work is the first attempt to automatically detect and classify hand-over-face gestures in natural expressions.

Prolonged durations of rhythmic body gestures were proved to be correlated with different types of psychological disorders. To-date, there is no automatic descriptor that can robustly detect those behaviours. In this paper, we propose a cyclic gestures descriptor that can detect and localise rhythmic body movements by taking advantage of both colour and depth modalities. We show experimentally how our rhythmic descriptor can successfully localise the rhythmic gestures as: hands fidgeting, legs fidgeting or rocking, significantly higher than the majority vote classification baseline. Our experiments also demonstrate the importance of fusing both modalities, with a significant increase in performance when compared to individual modalities.

We present a real-time system for detecting facial action units and inferring emotional states from head and shoulder gestures and facial expressions. The dynamic system uses three levels of inference on progressively longer time scales. Firstly, facial action units and head orientation are identified from 22 feature points and Gabor filters. Secondly, Hidden Markov Models are used to classify sequences of actions into head and shoulder gestures. Finally, a multi level Dynamic Bayesian Network is used to model the unfolding emotional state based on probabilities of different gestures. The most probable state over a given video clip is chosen as the label for that clip. The average F1 score for 12 action units (AUs 1, 2, 4, 6, 7, 10, 12, 15, 17, 18, 25, 26), labelled on a frame by frame basis, was 0.461. The average classification rate for five emotional states (anger, fear, joy, relief, sadness) was 0.440. Sadness had the greatest rate, 0.64, anger the smallest, 0.11.