Nonverbal communication is an important part of human communication, including head nodding, eye gaze, proximity and body orientation. Recent research has identified specific patterns of head nodding linked to conversation, namely mimicry of head movements at 600 ms delay and fast nodding when listening. In this paper, we implemented these head nodding behaviour rules in virtual humans, and we tested the impact of these behaviours, and whether they lead to increases in trust and liking towards the virtual humans. We use Virtual Reality technology to simulate a face-to-face conversation, as VR provides a high level of immersiveness and social presence, very similar to face-to-face interaction. We then conducted a study with human-subject participants, where the participants took part in conversations with two virtual humans and then rated the virtual character social characteristics, and completed an evaluation of their implicit trust in the virtual human. Results showed more liking for and more trust in the virtual human whose nodding behaviour was driven by realistic behaviour rules. This supports the psychological models of nodding and advances our ability to build realistic virtual humans.

Nonverbal cues have multiple roles in social encounters, with gaze behaviour facilitating interactions and conversational flow. In this work, we explore the conversation dynamics in dyadic settings in a free-flow discussion. Using automatic analysis (rather than manual labelling), we investigate how the gaze behaviour of one person is related to how much the other person changes their gaze (frequency in gaze change) and what their gaze target is (direct or avert gaze). Our results show that when one person is looked at they change their gaze direction with a higher frequency compared to when they are not looked at. They also tend to maintain a direct gaze to the other person when they are not looked at.

Modelling of the breath signal is of high interest to both \nhealthcare professionals and computer scientists, as a source \nof diagnosis-related information, or a means for curating higher \nquality datasets in speech analysis research. The formation of \na breath signal gold standard is, however, not a straightforward \ntask, as it requires specialised equipment, human annotation \nbudget, and even then, it corresponds to lab recording settings, \nthat are not reproducible in-the-wild. Herein, we explore deep \nlearning based methodologies, as an automatic way to predict a \ncontinuous-time breath signal by solely analysing spontaneous \nspeech. We address two task formulations, those of continuousvalued signal prediction, as well as inhalation event prediction, \nthat are of great use in various healthcare and Automatic Speech \nRecognition applications, and showcase results that outperform \ncurrent baselines. Most importantly, we also perform an initial \nexploration into explaining which parts of the input audio signal \nare important with respect to the prediction.

The INTERSPEECH 2020 Computational Paralinguistics Challenge addresses three different problems for the first time in a research competition under well-defined conditions: In the Elderly Emotion Sub-Challenge, arousal and valence in the speech of elderly individuals have to be modelled as a 3-class problem; in the Breathing Sub-Challenge, breathing has to be assessed as a regression problem; and in the Mask Sub-Challenge, speech without and with a surgical mask has to be told apart.We describe the Sub-Challenges, baseline feature extraction, and classifiers based on the 'usual' COMPARE and BoAW features as well as deep unsupervised representation learning using the AUDEEP toolkit, and deep feature extraction from pre-trained CNNs using the DEEP SPECTRUM toolkit; in addition, we partially add deep end-to-end sequential modelling, and, for the first time in the challenge, linguistic analysis.

This paper introduces the idea of using wearable, multi-modal body and brain sensing, in a theatrical setting, for neuroscientific research. Wearable motion capture suits are used to track the body movements of two actors while they enact a sequence of scenes together. One actor additionally wears a functional near-infrared spectroscopy (fNIRS)-based headgear to record the activation patterns on his prefrontal cortex. Repetitions in the movement data are then used to automatically segment the fNIRS data for further analysis. This exploration reveals that the semi-structured and repeatable nature of theatre can provide a useful laboratory for neuroscience, and that wearable sensing is a promising method to achieve this. This is important because it points to a new way of researching the brain in a more natural, and social, environment than traditional lab-based methods.

We introduce a method of using wrist-worn accelerometers to measure non-verbal social coordination within a group that includes autistic children. Our goal was to record and chart the children's social engagement - measured using interpersonal movement synchrony - as they took part in a theatrical workshop that was specifically designed to enhance their social skills. Interpersonal synchrony, an important factor of social engagement that is known to be impaired in autism, is calculated using a cross-wavelet similarity comparison between participants' movement data. We evaluate the feasibility of the approach over 3 live performances, each lasting 2 hours, using 6 actors and a total of 10 autistic children. We show that by visualising each child's engagement over the course of a performance, it is possible to highlight subtle moments of social coordination that might otherwise be lost when reviewing video footage alone. This is important because it points the way to a new method for people who work with autistic children to be able to monitor the development of those in their care, and to adapt their therapeutic activities accordingly.

In recent years, the use of Virtual Characters in experimental studies has opened new research avenues in social neuroscience. In this paper, we present the design, implementation, and preliminary results of two case studies exploring different types of social cognition using interactive Virtual Characters animated with motion captured data.