Museums are interested in designing emotional visitor experiences to complement traditional interpretations. HCI is interested in the relationship between Affective Computing and Affective Interaction. We describe Sensitive Pictures, an emotional visitor experience co-created with the Munch art museum. Visitors choose emotions, locate associated paintings in the museum, experience an emotional story while viewing them, and self-report their response. A subsequent interview with a portrayal of the artist employs computer vision to estimate emotional responses from facial expressions. Visitors are given a souvenir postcard visualizing their emotional data. A study of 132 members of the public (39 interviewed) illuminates key themes: designing emotional provocations; capturing emotional responses; engaging visitors with their data; a tendency for them to align their views with the system's interpretation; and integrating these elements into emotional trajectories. We consider how Affective Computing can hold up a mirror to our emotions during Affective Interaction

Virtual human technologies are now being widely explored as therapy tools for mental health disorders including depression and anxiety. These technologies leverage the ability of the virtual agents to engage in naturalistic social interactions with a user to elicit behavioural expressions which are indicative of depression and anxiety. Research efforts have focused on optimising the human-like expressive capabilities of the virtual human, but less attention has been given to investigating the effect of virtual human mediation on the expressivity of the user. In addition, it is still not clear what an optimal task is or what task characteristics are likely to sustain long term user engagement. To this end, this paper describes the design and evaluation of virtual human-mediated tasks in a user study of 56 participants. Half the participants complete tasks guided by a virtual human, while the other half are guided by text on screen. Self-reported PHQ9 scores, biosignals and participants' ratings of tasks are collected. Findings show that virtual-human mediation influences behavioural expressiveness and this observation differs for different depression severity levels. It further shows that virtual human mediation improves users' disposition towards tasks.

Access to healthcare advice is crucial to promote healthy societies. Many factors shape how access might be constrained, such as economic status, education or, as the COVID-19 pandemic has shown, remote consultations with health practitioners. Our work focuses on providing pre/post-natal advice to maternal women. A salient factor of our work concerns the design and deployment of embodied conversation agents (ECAs) which can sense the (health) literacy of users and adapt to scaffold user engagement in this setting. We present an account of a Wizard of Oz user study of 'ALTCAI', an ECA with three modes of interaction (i.e., adaptive speech and text, adaptive ECA, and non-adaptive ECA). We compare reported engagement with these modes from 44 maternal women who have differing levels of literacy. The study shows that a combination of embodiment and adaptivity scaffolds reported engagement, but matters of health-literacy and language introduce nuanced considerations for the design of ECAs.

The EmoPain 2020 Challenge is the first international competition aimed at creating a uniform platform for the comparison of multi-modal machine learning and multimedia processing methods of chronic pain assessment from human expressive behaviour, and also the identification of pain-related behaviours. The objective of the challenge is to promote research in the development of assistive technologies that help improve the quality of life for people with chronic pain via real-time monitoring and feedback to help manage their condition and remain physically active. The challenge also aims to encourage the use of the relatively underutilised, albeit vital bodily expression signals for automatic pain and pain-related emotion recognition. This paper presents a description of the challenge, competition guidelines, bench-marking dataset, and the baseline systems' architecture and performance on the Challenge's three sub-tasks: pain estimation from facial expressions, pain recognition from multimodal movement, and protective movement behaviour detection.

Pain assessment is a vital part of newborn treatment in Intensive Care Units. However, clinical pain assessment is highly subjective and does not support continual pain monitoring. Automated tools have been introduced to address this problem, but their performance is limited by inadequate training data and unsuitable pain annotations. In addition, current automated tools focus on pain detection rather than severity estimation, which is the unmet need in medical treatment. In this paper, we present: a) the Acute Pain in Neonates (APN-db) database, a public dataset to support research in this field and allow for benchmarking of new automated tools; b) a novel L1-point Neonatal Face and Limb Acute Pain Scale (NFLAPS), a visual behaviour-centric pain measurement tool, which is an adaptation of the Neonatal Infant Pain Scale (NIPS) and the Neonatal Facial Coding System (NFCS); and c) a system for neonatal pain assessment which encodes pain indicative-features using handcrafted algorithms and deep-learned features. Experiments show that our system performs well with an RMSE of 1.9 compared to human error of 1.65 on the same dataset, demonstrating its potential application to newborn health care.

Learning regression-based machine learning models for computer vision problems is a challenging task due to noisy features, variation in pose and illumination, occlusion, etc. Typically the problem is compounded by the non-uniform distribution of labels in the training data, resulting in parts of the label space that suffer from data sparsity and a problem of label imbalance in general. Deep Convolutional Neural Networks (CNN) have shown remarkable success on a number of computer vision tasks such as object classification and face recognition. However, they too suffer from sparse and imbalanced training datasets for regression problems, even when those datasets are very large. Cumulative Attributes have previously been proposed to address the issue of label imbalance, but to date this concept has not been integrated with Deep Learning. In this work, we propose a CNN-based framework for learning regression models by using Cumulative Attributes as intermediate features. We evaluate our method on a number of tasks which includes pain intensity estimation, Facial Action Unit intensity estimation and age estimation. Our results show that the proposed method is robust to imbalance and sparsity present in the training datasets, and performs significantly better than the current methods where CNNs are learnt directly for regression.

Automatic continuous time, continuous value assessment of a patient's pain from face video is highly sought after by the medical profession. Despite the recent advances in deep learning that attain impressive results in many domains, pain estimation risks not being able to benefit from this due to the difficulty in obtaining data sets of considerable size. In this work we propose a combination of hand-crafted and deep-learned features that makes the most of deep learning techniques in small sample settings. Encoding shape, appearance, and dynamics, our method significantly outperforms the current state of the art, attaining a RMSE error of less than 1 point on a 16-level pain scale, whilst simultaneously scoring a 67.3% Pearson correlation coefficient between our predicted pain level time series and the ground truth.

Pain estimation from face video is a hard problem in automatic behaviour understanding. One major obstacle is the difficulty of collecting sufficient amounts of data, with balanced amounts of data for all pain intensity levels. To overcome this, we propose to adopt Cumulative Attributes, which assume that attributes for high pain levels with few examples are a superset of all attributes of lower pain levels. Experimental results show a consistent relative performance increase in the order of 20% regardless of features used. Our final system significantly outperforms the state of the art on the UNBC McMaster Shoulder Pain database by using cumulative attributes with Relevance Vector Regression on a combination of features, including appearance, geometric, and deep learned features.