Depression is a complex disorder for which there is growing interest in identifying objective behavioral markers that measure precise symptoms, such as anhedonia and blunted emotional reactivity. This study explores the feasibility of using smile and smirk expression dynamics, captured through our novel stimulus-based mobile affective probe, as candidate digital biomarkers of depression severity within a large-scale mobile health intervention trial, BeWell. Data from 684 BeWell participants (2,702 observations) are analyzed longitudinally for 16 weeks, comparing their PHQ-8 survey scores with their facial responses to short videos intended to elicit smiles. Mixed-effects models reveal that higher maximum Duchenne smile intensity in reaction to liked stimuli is associated with lower depression scores over time at both within- and between-person levels. We additionally share insights from our tool, including ease of use, perceptions of the stimulus, and technical challenges, which offer considerations for the future development of stimulus-based affect probes in real-world settings.

Approximately 400,000 youth in the US are living in foster care due to experiences with abuse or neglect at home[17]. For multiple reasons, these youth often don't receive adequate social support from those around them. Despite technology's potential, very little work has explored how these tools can provide more support to foster-involved youth. To begin to fill this gap, we worked with current and former foster-involved youth to develop the first digital tool that aims to increase social support for this population, creating a novel system in which users complete reflective check-ins in an online community setting. We then conducted a pilot study with 15 current and former foster-involved youth, comparing the effect of using the app for two weeks to two weeks of no intervention. We collected qualitative and quantitative data, which demonstrated that this type of interface can provide youth with types of social support that are often not provided by foster care services and other digital interventions. The paper details the motivation behind the app, the trauma-informed design process, and insights gained from this initial evaluation study. Finally, the paper concludes with recommendations for designing digital tools that effectively provide social support to foster-involved youth.

This paper explores the design, development and evaluation of a digital platform that aims to assist young people who have experienced trauma in understanding and expressing their emotions and fostering social connections. Integrating principles from expressive arts and narrative-based therapies, we collaborate with lived experts to iteratively design a novel, user-centered digital tool for young people to create and share comics that represent their experiences. Specifically, we conduct a series of nine workshops with N=54 trauma-impacted youth and young adults to test and refine our tool, beginning with three workshops using low-fidelity prototypes, followed by six workshops with Cube, a web version of the tool. A qualitative analysis of workshop feedback and empathic relations analysis of artifacts provides valuable insights into the usability and potential impact of the tool, as well as the specific needs of young people who have experienced trauma. Our findings suggest that the integration of expressive and narrative therapy principles into Cube can offer a unique avenue for trauma-impacted young people to process their experiences, more easily communicate their emotions, and connect with supportive communities. We end by presenting implications for the design of social technologies that aim to support the emotional well-being and social integration of youth and young adults who have faced trauma.

How does empathy influence creative problem solving? We introduce a computational empathy intervention based on context-specific affective mimicry and perspective taking by a virtual agent appearing in the form of a well-dressed polar bear. In an online experiment with 1,006 participants randomly assigned to an emotion elicitation intervention (with a control elicitation condition and anger elicitation condition) and a computational empathy intervention (with control virtual agent and an empathic virtual agent), we examine how anger and empathy influence participants' performance in solving a word game based on Wordle. We find participants who are assigned to the anger elicitation condition perform significantly worse on multiple performance metrics than participants assigned to the control condition. However, we find the empathic virtual agent counteracts the drop in performance induced by the anger condition such that participants assigned to both the empathic virtual agent and the anger condition perform no differently than participants in the control elicitation condition and significantly better than participants assigned to the control virtual agent and the anger elicitation condition. While empathy reduces the negative effects of anger, we do not find evidence that the empathic virtual agent influences performance of participants who are assigned to the control elicitation condition. By introducing a framework for computational empathy interventions and conducting a two-by-two factorial design randomized experiment, we provide rigorous, empirical evidence that computational empathy can counteract the negative effects of anger on creative problem solving.

This work introduces The Guardians: Unite the Realms, a novel free-to-play and publicly released mobile game that encourages the adoption of healthy real-world behaviours in exchange for rewards that enrich the gaming experience. We describe the game, its grounding in a mental health therapy known as behavioural activation, and how we designed it to keep players engaged over time. Instead of using traditional digital health gamification techniques such as badges or leaderboards, The Guardians creates a motivational pull by embedding the therapy into a complete mobile game. In-game items earned via the therapy have an immediate purpose in the game and, thus, they are considered intrinsically valuable by players. Analysis of game interaction data from 7,782 real-world users suggests 15-day and 30-day retention rates of 10.0% and 6.6%, respectively, which is more than double the average retention levels of most digital mental health interventions. Furthermore, players reported completion of a healthy real-world task on 69.0% of days played (37,574 completed tasks in 54,461 total days). We also report interaction metrics with game features and the effectiveness of the players' chosen real-world activities.

Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, Rosalind Picard. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.

Nonverbal vocalizations contain important affective and communicative information, especially for those who do not use traditional speech, including individuals who have autism and are non- or minimally verbal (nv/mv). Although these vocalizations are often understood by those who know them well, they can be challenging to understand for the community-at-large. This work presents (1) a methodology for collecting spontaneous vocalizations from nv/mv individuals in natural environments, with no researcher present, and personalized in-the-moment labels from a family member; (2) speaker-dependent classification of these real-world sounds for three nv/mv individuals; and (3) an interactive application to translate the nonverbal vocalizations in real time. Using support-vector machine and random forest models, we achieved speaker-dependent unweighted average recalls (UARs) of 0.75, 0.53, and 0.79 for the three individuals, respectively, with each model discriminating between 5 nonverbal vocalization classes. We also present first results for real-time binary classification of positive- and negative-affect nonverbal vocalizations, trained using a commercial wearable microphone and tested in real time using a smartphone. This work informs personalized machine learning methods for non-traditional communicators and advances real-world interactive augmentative technology for an underserved population.

Building an open-domain conversational agent is a challenging problem. Current evaluation methods, mostly post-hoc judgments of static conversation, do not capture conversation quality in a realistic interactive context. In this paper, we investigate interactive human evaluation and provide evidence for its necessity; we then introduce a novel, model-agnostic, and dataset-agnostic method to approximate it. In particular, we propose a self-play scenario where the dialog system talks to itself and we calculate a combination of proxies such as sentiment and semantic coherence on the conversation trajectory. We show that this metric is capable of capturing the human-rated quality of a dialog model better than any automated metric known to-date, achieving a significant Pearson correlation (r>.7, p<.05). To investigate the strengths of this novel metric and interactive evaluation in comparison to state-of-the-art metrics and human evaluation of static conversations, we perform extended experiments with a set of models, including several that make novel improvements to recent hierarchical dialog generation architectures through sentiment and semantic knowledge distillation on the utterance level. Finally, we open-source the interactive evaluation platform we built and the dataset we collected to allow researchers to efficiently deploy and evaluate dialog models.

A growing number of studies show links between changes in tongue appearance and human health conditions. This paper studies tongue color changes in the context of stress to explore the feasibility of providing a novel and non-invasive stress measurement method. In a laboratory study, 24 participants were asked to perform a calm and a stressful math task and to take a photo of their tongue right after each of the tasks. We observed subtle but consistent color differences between calm and stress tasks for up to 75% of the participants, which was consistent with both self-report and physiological metrics of stress. Moreover, we observed significant correlations of up to 0.72 between certain tongue colors and long-term stress assessed with the 10-item Perceived Stress Scale questionnaire. We discuss the potential implications of this work and highlight some lines of future research.