Serious Illness Conversations (SICs)—discussions about values and care preferences for patients with life-threatening illness—rarely occur in Emergency Departments (EDs), despite evidence that early conversations improve care alignment and reduce unnecessary interventions. We interviewed 11 ED providers to identify challenges in SICs and opportunities for technology support, with a focus on AI. Our analysis revealed a four-stage SIC workflow (identification, preparation, conduction, documentation) and barriers at each stage, including fragmented patient information, limited time and space, lack of conversational guidance, and burdensome documentation. Providers expressed interest in AI systems for synthesizing information, supporting real-time conversations, and automating documentation, but emphasized concerns about preserving human connection and clinical autonomy. This tension highlights the need for technologies that enhance efficiency without undermining the interpersonal nature of SICs. We propose design guidelines for ambient and peripheral AI systems to support providers while preserving the essential humanity of these conversations.

Learning therapeutic counseling involves significant role-play experience with mock patients, with current manual training methods providing only intermittent granular feedback. We seek to accelerate and optimize counselor training by providing frequent, detailed feedback to trainees as they interact with a simulated patient. Our first application domain involves training motivational interviewing skills for counselors. Motivational interviewing is a collaborative counseling style in which patients are guided to talk about changing their behavior, with empathetic counseling an essential ingredient. We developed and evaluated an LLM-powered training system that features a simulated patient and visualizations of turn-by-turn performance feedback tailored to the needs of counselors learning motivational interviewing. We conducted an evaluation study with professional and student counselors, demonstrating high usability and satisfaction with the system. We present design implications for the development of automated systems that train users in counseling skills and their generalizability to other types of social skills training.

Personalized and tailored health education materials have demonstrated greater effectiveness than generic content.Traditional tailoring methods in health communication typically rely on collecting user data through static forms and applying rule-based techniques to assemble health documents for print or digital delivery.In this project, we developed a novel system that combined a virtual agent (VA) with a large language model (LLM) to generate individualized breast cancer screening pamphlets.The VA conducted interactive interviews to gather user-specific information, including demographics, family history, and lifestyle factors.This information was then used by the LLM to generate a pamphlet that was both tailored and personalized.We conducted a pilot study to evaluate this approach, using breast cancer screening as a representative educational topic.Participants interacted with the virtual agent and provided relevant personal information before being randomized to receive either an LLM-generated pamphlet or a standard, generic pamphlet.We assessed group differences in knowledge, screening intentions, and satisfaction.Results showed that both groups experienced significant knowledge gains, with the Tailored group demonstrating greater improvement.Qualitative feedback revealed that participants who received the personalized and tailored pamphlets perceived the information as more personally relevant, often noting that the content reflected details they had shared during their interaction with the agent.These findings highlight the potential of integrating the VA and LLMs to enhance the personalization and effectiveness of health education materials.

The proliferation of Large Language Models and Intelligent Virtual Agents acting as psychotherapists offer expanded mental healthcare access.However, their deployment has led to serious harms due to a lack of methods to evaluate nuanced therapeutic risks.Current evaluation techniques lack the sensitivity to detect subtle changes in patient cognition and behavior during therapy sessions that may lead to subsequent decompensation.We introduce a risk ontology for the evaluation of conversational AI psychotherapists.Developed via literature review, expert interviews, and alignment with clinical criteria (DSM-5) and assessment tools (NEQ, UE-ATR), the ontology provides a structured approach to assessing patient harm.

Many laypeople are motivated to improve the health behavior of their family or friends but do not know where to start, especially if the health behavior is potentially stigmatizing or controversial. We present an approach that uses virtual agents to coach community-based volunteers in health counseling techniques, such as motivational interviewing, and allows them to practice these skills in role-playing scenarios. We use this approach in a virtual agent-based system to increase COVID-19 vaccination by empowering users to influence their social network. In a between-subjects comparative design study, we test the effects of agent system interactivity and role-playing functionality on counseling outcomes, with participants evaluated by standardized patients and objective judges. We find that all versions are effective at producing peer counselors who score adequately on a standardized measure of counseling competence, and that participants were significantly more satisfied with interactive virtual agents compared to passive viewing of the training material. We discuss design implications for interpersonal skills training systems based on our findings.

Public speaking experts intentionally take their audience on an emotional roller coaster, staying attuned to their audience's collective emotional feedback. In this research, we explore how bidirectional sharing of heart rates between a speaker and their audience facilitates this emotional exchange, through empathy, emotional awareness, and engagement. Firstly, in two design studies$(N=25)$, we evaluated this concept and identified design elements for a heart rate sharing interface. Subsequently, we developed Feeling-the-Beat, a system for sharing heart rate between speakers and audiences in real time. Finally, in a randomized, counter-balanced, within-subjects study$(N=36)$, we compared our system to one sharing fabricated heart rates and a baseline system without heart rate sharing. Feeling-the-Beat significantly increased audience empathy towards the speaker, audience engagement during moments of heightened speaker heart rate, and social presence.

We introduce the concept of “empathic grounding” in conversational agents as an extension of Clark’s conceptualization of grounding in conversation in which the grounding criterion includes listener empathy for the speaker’s affective state. Empathic grounding is generally required whenever the speaker’s emotions are foregrounded and can make the grounding process more efficient and reliable by communicating both propositional and affective understanding. Both speaker expressions of affect and listener empathic grounding can be multimodal, including facial expressions and other nonverbal displays. Thus, models of empathic grounding for embodied agents should be multimodal to facilitate natural and efficient communication. We describe a multimodal model that takes as input user speech and facial expression to generate multimodal grounding moves for a listening agent using a large language model. We also describe a testbed to evaluate approaches to empathic grounding, in which a humanoid robot interviews a user about a past episode of pain and then has the user rate their perception of the robot’s empathy. We compare our proposed model to one that only generates non-affective grounding cues in a between-subjects experiment. Findings demonstrate that empathic grounding increases user perceptions of empathy, understanding, emotional intelligence, and trust. Our work highlights the role of emotion awareness and multimodality in generating appropriate grounding moves for conversational agents.

Smartphone-based conversational agents offer a convenient way to deliver health education, particularly for managing complex health conditions such as chronic diseases. The present study explores using a smartphone-based virtual agent system to aid patients with a chronic heart condition—atrial fibrillation—to manage their health by encouraging the use of a heart rhythm sensor integrated with their smartphones. We report the results of a randomized clinical trial with 240 patients experiencing atrial fibrillation who were provided with smartphones and heart rhythm sensors for 4 months. Participants in the intervention group interacted with a virtual agent, while the control group received general health education via the WebMD app. Intervention participants completed a median 91 interactions with the agent over the 4 months of the study period, and agent features designed to increase engagement–such as storytelling–were effective at increasing use. The agent’s promotion of heart rhythm sensor use was effective, with intervention participants taking significantly more heart rhythm readings compared to those in the control group. Participants were followed for 8 months thereafter to assess the sustainability of the effects, specifically focusing on medication adherence.

Virtual health counselors offer the potential to provide users with information and counseling in complex areas such as disease management and health education. However, ensuring user engagement is challenging, particularly when the volume of information and length of counseling sessions increase. Agenda setting—–a clinical counseling technique where a patient and clinician collaboratively decide on session topics—–is an effective approach to tailoring discussions for individual patient needs and sustaining engagement. We explore the effectiveness of agenda setting in a virtual counselor system designed to counsel women for breast cancer genetic testing. In a between-subjects study, we assessed three versions of the system with varying levels of user control in the system’s agenda setting approach. We found that participants’ knowledge improved across all conditions. Although our results showed that any type of agenda setting was perceived as useful, regardless of user control, interviews revealed a preference for more collaboration and user involvement in the agenda setting process. Our study highlights the importance of using patient-centered approaches, such as tailored discussions, when using virtual counselors in healthcare.

We introduce a novel application of large language models (LLMs) in developing a virtual counselor capable of conducting motivational interviewing (MI) for alcohol use counseling. Access to effective counseling remains limited, particularly for substance abuse, and virtual agents offer a promising solution by leveraging LLM capabilities to simulate nuanced communication techniques inherent in MI. Our approach combines prompt engineering and integration into a user-friendly virtual platform to facilitate realistic, empathetic interactions. We evaluate the effectiveness of our virtual agent through a series of studies focusing on replicating MI techniques and human counselor dialog. Initial findings suggest that our LLM-powered virtual agent matches human counselors’ empathetic and adaptive conversational skills, presenting a significant step forward in virtual health counseling and providing insights into the design and implementation of LLM-based therapeutic interactions.

Standardized, validated questionnaires are vital tools in research and healthcare, offering dependable self-report data. Prior work has revealed that virtual agent-administered questionnaires are almost equivalent to self-administered ones in an electronic form. Despite being an engaging method, repeated use of virtual agent-administered questionnaires in longitudinal or pre-post studies can induce respondent fatigue, impacting data quality via response biases and decreased response rates. We propose using large language models (LLMs) to generate diverse questionnaire versions while retaining good psychometric properties. In a longitudinal study, participants interacted with our agent system and responded daily for two weeks to one of the following questionnaires: a standardized depression questionnaire, question variants generated by LLMs, or question variants accompanied by LLM-generated small talk. The responses were compared to a validated depression questionnaire. Psychometric testing revealed consistent covariation between the external criterion and focal measure administered across the three conditions, demonstrating the reliability and validity of the LLM-generated variants. Participants found that the variants were significantly less repetitive than repeated administrations of the same standardized questionnaire. Our findings highlight the potential of LLM-generated variants to invigorate agent-administered questionnaires and foster engagement and interest, without compromising their validity.

Faith institutions provide social support and community care for many in the United States (U.S.). Notably, churches with predominantly Black populations have served as a site for social change and care provision, historically and in contemporary society. However, the pandemic has emphasised how localising these care networks in physical spaces can limit access to social support. Information and communication technologies offer opportunities for expanding access to care in these communities. However, integrating care networks into online contexts remains a challenge for many churches, and the potential for technology to expand these networks is not well understood. Through interviews and focus groups with nine church members, we explore how hybrid faith communities that bridge offline and online contexts can enable social support and care provision. Our findings highlight care network structures in Black churches, barriers to embedding these networks online and strategies for building more seamless hybrid support systems.

Over 55 million adults worldwide have dementia, a syndrome characterized by deterioration in cognitive functioning. Screening for mild cognitive impairment is important to identify dementia early to facilitate diagnosis and initiate treatment that may modify the disease trajectory. However, standard cognitive screening tools are time-consuming, require expert administration, and make people feel as if they are being tested and are thus potentially stigmatizing. Consequently, standard cognitive screening tools are underutilized, and dementia is often identified much later in the disease process leading to greater health-related and societal harms. We explored cognitive ability assessments using virtual agents, in which assessments are made during conversational dialogues. We evaluated four strategies for the assessment of cognitive ability in a sample of 41 individuals with varying cognitive ability, comparing each strategy to scores on the Mini Mental Status Exam (MMSE), the gold standard clinical screening tool for cognitive impairment. One of the four conversational strategies demonstrated significant agreement with the MMSE, with a sensitivity of 92.3% and a specificity of 48.3% for identification of impairment compared to the MMSE. Participants reported moderate levels of satisfaction with the agent-based conversational assessment. This approach could be used to identify individuals warranting more evaluation for possible cognitive impairment.

Client-centered counseling, in which individuals are prompted to talk about their behavior, is the standard treatment for Alcohol misuse. However, open-ended conversations with virtual agent counselor raise potential safety concerns if the agent misunderstands and provides erroneous advice. Thus, while generative machine learning models have been successful for language understanding and generation tasks, these approaches may not be effective or safe for counseling. We present a hybrid dialog system that uses a machine-learning model to generate responses to individual client speech combined with a rule-based approach to transition through structured counseling sessions. The dialog system is used to drive a virtual agent alcohol misuse counselor. We evaluated this hybrid system by comparing it to a functionally equivalent system in which the dialog is driven by fully-constraining user utterances via multiple-choice menus among individuals with problematic drinking. Participants who interacted with the agent using the hybrid dialog system reported a higher degree of readiness to change their drinking habits compared to the system using constrained input. Additionally, the outputs of the models used by the system were judged to be safe and appropriate for the task by expert counselors.

Parental permission is required for medical care for children, and decisions may be made without incorporating children's views, even for adolescents. We explore the impact of including adolescents in virtual agent-based multiparty health counseling to promote Human Papillomavirus (HPV) vaccination. The agent is designed to encourage HPV vaccination for children aged 9-12 by engaging co-present parent/adolescent dyads in an online interaction. Several techniques are incorporated, including HPV education, motivational interviewing, persuasion, modeling, and enablement, to address parents' intent to vaccinate their children. We conduct a between-subjects randomized study comparing a version of the agent exclusively for the parent, to one that includes the adolescent in the conversation and incorporates the child's views in counseling strategies. We measure pre- and post-intervention changes in intent to vaccinate, vaccination hesitancy, and knowledge in both the parent and the adolescent, hypothesizing greater improvements in these measures when the adolescent is included in the conversation. We also examine the satisfaction, engagement, and comfort of the parent/child interactions with the virtual agent. We found significant pre-post increases in parent intent to vaccinate their adolescent for both versions of the agent. Our work provides insights into the effectiveness of the virtual agent in promoting HPV vaccination, the impact of child participation on healthcare decision-making, and the user experience of multi-party interaction with the virtual agent.

Recent trends and developments in robotics have enabled socially assistive mobile robotic platforms to be deployed in everyday human lives. These robots have the ability to navigate to a user’s location and engage in multimodal interactions to serve a variety of purposes such as promoting health behavior change. We conducted a randomized two-factor experiment to study the utility of robot mobility and multimodal cuing in a collaborative meal assembly task. We found that robot mobility, proxemics, and deictic and verbal cuing have significant positive effects on compliance with the robot’s food recommendations and resulting nutritional quality of assembled meals. These robot behaviors also led to stronger therapeutic alliance between the robot and the user and higher user engagement.

HCI researchers have increasingly examined how social context shapes health behaviors. Much of this work operates at the interpersonal level. Communities such as churches play important roles in supporting wellbeing and addressing health inequities. While some work has investigated creating digital health tools for religious populations, few have explicitly focused on the incorporation of community support in the form of prayer support. Embedding health interventions in any community has the potential to support or challenge the community’s dynamics. We report on findings from interviews with 17 church members who used a church-based mHealth application over a 4-week period and provide guidelines for developers based on these results. Through their use of the system, participants characterized several community dynamics including a desire for social intimacy, communicating care, creating opportunities for fellowship, maintaining privacy and discretion, and building community connections, and how these dynamics influence their aspirations for a church-based health app.

We describe an automated, multimodal embodied conversational agent that plays the role of a genetic counselor, designed to communicate breast cancer risk and the recommended medical guidelines to women. The counselor's dialogue is driven by intelligent tutoring systems techniques, risk communication principles, and information processing theories. The virtual counselor dynamically tailors its counseling based on user traits, preferred information processing methods, and dynamic comprehension assessments. We conducted a between-subject evaluation study with 30 women, comparing the adaptive counselor to a non-adaptive version of the counselor and a control condition. Women in the adaptive condition demonstrated a significantly greater increase in breast cancer genetics knowledge compared to women in the other conditions. Our results demonstrate the effectiveness of the multidimensional adaptation mechanisms for improving cancer genetic risk communication.

Public speaking is important in the sciences, but poor quality presentations are common, as are high rates of public speaking anxiety. In this work we explore the use of a mobile humanoid robot as a co-presenter that can share the stage with a scientist giving an oral presentation. We conducted a within-subjects experiment comparing presentations given with and without the robot and impacts on public speaking anxiety and speaker confidence. We found that participants reported significantly greater confidence and lower public speaking anxiety when co-presenting with the robot, compared to when they presented on their own, without the robot. Audiences accepted scientific presentations given with the robot, rating these presen-tations significantly greater than neutral on presentation quality.

We developed a virtual agent that motivates church-going users to change their health behavior by telling existing cultural narratives that have high relevance with the counseling topic in an engaging way. We evaluated this agent in a between-subjects experiment where participants interacted with an agent that counseled them on nutrition either without a story, with a story but told in a neutral speech style, or with a story using dramatic delivery inspired by church sermons. We found that interaction with either one of the storytelling agents leads to a significantly greater change in confidence to engage in the target behavior of healthy eating than interacting with a non-storytelling agent, demonstrating the efficacy of stories in health counseling by virtual agents.

Conversational agents (CAs) are systems that interact with humans through natural language user interfaces. They include systems with a range of conversational capabilities and modalities. For example, there are text- or voice-only question-answering interactions such as Apple Siri, Google Assistant, and Amazon Alexa, and there are also multimodal conversational AI agents that can engage users in long-term dialogues. Advances in speech recognition, natural language processing, and computer vision have resulted in a greater acceptance and use of CAs. CAs have already started to play important roles in various healthcare settings, including assisting clinicians during consultations, assisting consumers in changing health behaviours, and helping patients such as the elderly in their living environments. \n \nThere have been several systematic reviews on the use of CAs in health and wellbeing recently. Although the field still appears to be nascent, the emerging evidence has shown user acceptance of CAs in the healthcare domain as well as the early promises in boosting healthcare outcomes in both physical and mental health. Despite the increasing adoption and the benefits of using CAs to support health and wellbeing, the review studies also revealed (i) patient safety was rarely examined, (ii) health outcomes were inadequately measured, and (iii) no standardised evaluation methods were employed. There were also limitations in reporting the technical implementation details of CAs used, making the replicability of prior studies problematic. \n \nIn addition to addressing some of the current challenges and limitations, this special issue features cutting-edge research on designing, developing, and evaluating CAs for health and wellbeing that aim to improve health outcomes and services, and satisfy unique application needs (e.g., safety, trust, and user experience). The seven articles included in this special issue cover many application areas ranging from mental health and social support to information seeking to coaching. Amongst the accepted articles, mental health and social support themes represented the primary research foci. The articles also covered different population groups including older adults, young adults, and homeless people. Based on their foci, we have grouped the articles in this special issue by three areas: mental health, older adult wellbeing, and social support and coaching.

Churches have historically played an important role in Black American communities, catalyzing the pursuit of aims such as social justice, community organization, and health promotion. However, researchers have rarely examined how technology can support an assets-based approach to these efforts, nor the implications of race, traditions, and history when creating such systems. Addressing this gap, we conducted research with two predominantly Black churches to explore health promotion design opportunities. We used photovoice, a research method where participants led their own data collection and analysis. Participants provided nuanced descriptions of the racial and ethnic identities of their communities, and how church history and aspirations for the future impacted these identities. Our findings characterize tensions between tradition and ‘modernization,’ implications for technology design, and the need for a temporal approach to understanding communities. We conclude with broader implications for studying the intersection of race and religion in community technology design.

Job advertisements in white male-dominated organizations are often biased in ways that discourage female and minority candidates from applying. We explored the use of a female African American virtual agent who provides a first-person reaction to a biased job advertisement, providing an impassioned, vivid description of her feelings about the advertisement, the position, and the organization offering the job, and how the position---as described---would impact her life were she to take the job. We evaluate the impact interactions with this agent have on the effort study participants invest in editing the job advertisement following their interaction with the agent, compared to reading a page of standard educational text on diversity in hiring. Participants who interacted with the agent spent significantly more effort correcting the job ad, as measured both by the number of edit operations and the number of biased phrases removed, compared to participants in the control condition. Implications and a future research agenda for increasing diversity using virtual agents are presented.

We present Friendly Face - a virtual agent designed to reduce public speaking anxiety by standing within an audience. The agent senses the speaker's behavior during an oral presentation and provides emotional and instrumental support. The system unobtrusively tracks the motion, speech, and prosody of the presenter and provides an intuitive interface to give supportive feedback whenever the presenter looks at the agent, attentive listening behavior through agent gaze and backchannel listening behavior, and time and topic cueing based on real-time analysis of speech content compared to presentation slide contents. An evaluation of Friendly Face agent with a functionally equivalent control system demonstrated that the agent system led to significant reductions in public speaking anxiety compared to a control condition, assessed both objectively with physiological measures and validated self-report instruments.

Churches play a major role in providing social support to address health inequities within Black communities, in part by connecting members to key organizations and services. While public health has a history of disseminating interventions in faith communities, little work has explored the use of crowdsourcing to tailor interventions to the unique culture of each church community. Following Community Based Participatory Research principles, we partnered with two predominantly Black churches, and report on a series of three participatory design sessions with nine participants. We developed a novel storyboarding method to explore how crowdsourcing could promote health in these faith-based communities. Our findings characterize existing supports within the church community, and how church social structures impact member access to these supports. We further identify motivations to engage with a church-situated health application, and how these motivations translate to crowdsourcing tasks. Finally, we discuss considerations for public health crowdsourcing tasks.

Audience perceptions of public speakers' performance change over time. Some speakers start strong but quickly transition to mundane delivery, while others may have a few impactful and engaging portions of their talk preceded and followed by more pedestrian delivery. In this work, we model the time-varying qualities of a presentation as perceived by the audience and use these models both to provide diagnostic information to presenters and to improve the quality of automated performance assessments. In particular, we use HMMs to model various dimensions of perceived quality and how they change over time and use the sequence of quality states to improve feedback and predictions. We evaluate this approach on a corpus of 74 presentations given in a controlled environment. Multimodal features-spanning acoustic qualities, speech disfluencies, and nonverbal behavior were derived both automatically and manually using crowdsourcing. Ground truth on audience perceptions was obtained using judge ratings on both overall presentations (aggregate) and portions of presentations segmented by topic. We distilled the overall presentation quality into states representing the presenter's gaze, audio, gesture, audience interaction, and proxemic behaviors. We demonstrate that an HMM of state-based representation of presentations improves the performance assessments.

Alcohol, tobacco and illicit drug use cause significant loss quality of life, and represent a large economic burden on society. Standardized questionnaires have been developed to enable healthcare providers to screen individuals for substance misuse problems that may require intervention. Although routine use of these screeners is promoted in many healthcare organizations, their administration can be irregular and lack follow through. In addition, patients may be reluctant to report substance use. We report on the development and validation of a Virtual Agent designed to automate the administration of a standard substance use screening instrument. We report on two validation studies in which a total of 57 patients at a US Veterans Administration medical center were screened by the virtual agent, comparing first to a research assistant administering the standard screening questionnaire, and second to a standard computerized text-based implementation of the questionnaire. Assessment correlations between the agent and human administration were all significant, ranging from rho=0.83 to 0.94, with more disclosure to the agent. Correlations were lower for Study 2 (0.56 - 0.88), with participants significantly more satisfied with the Agent compared to the standard tool.

Culturally informed design for virtual agents has been shown to positively impact health outcomes when tailored to target audiences. We present a participatory design methodology for culturally tailoring virtual agents. Investigators worked with key informants from our target population, members of predominantly Black church communities, to design culturally-relevant and sensitive virtual agent health promotion interventions. In the first participatory session, key informants designed agents to assist them with different aspects of their lives, providing input on agent appearance and agent functionality. In a second design session, participants re-wrote the content of a health conversation with an agent, to include personally-relevant content related to their community (e.g., religious and scriptural references). We report design principles for religious tailoring derived from these studies. We conducted a validation study to assess the effects of applying these principles to agents that promoted two health behaviors, finding that participants responded very positively to the tailored agents.

Research on the psychology of humor indicates that humor can increase the impact of persuasive messages in certain circumstances, suggesting that virtual agents could use humor to improve the effectiveness of motivational counseling with users. To explore this idea, we developed two virtual agents that attempt to motivate users to perform healthy behaviors - either to increase exercise or fruit and vegetable consumption - and systematically vary whether they use humor or not in their counseling conversations. Human-authored jokes were selected from a large corpus based on the nature of the humor (empathetic affiliative humor that serves to build relational closeness), health topic relevance, and subjective ratings of funniness, as well as relevance to user stage of behavior change. We evaluated our agents in a two-treatment counterbalanced within-subjects experiment, where participants interacted with a humorous and non-humorous agent motivating either exercise or healthy diet. We found the interaction with the humorous agent led to a significantly greater change in motivation to engage in the target behavior than interacting with the non-humorous agent.

Public speaking is an essential task for success in many careers, yet fear of public speaking makes this an undesirable activity for most people and negatively affects the quality of many presentations. Cognitive behavioral therapy is an effective tool for helping people overcome social anxieties disorders, including public speaking anxiety. However, most people do not engage in this therapy for public speaking anxiety, due to time constraints and other barriers. We present a virtual coach that uses cognitive behavioral therapy techniques to help presenters restructure irrational thoughts associated with public speaking anxiety. The design of the virtual coach was informed by an analysis of a corpus of cognitive restructuring examples generated by a clinical psychologist. In a between-subjects experiment comparing the virtual coach to a control condition, the virtual coach was shown to be significantly better at reducing thoughts associated with speech anxiety and improving a presentation experience for speakers.

Intimate relationships are integral parts of human societies, yet many relationships are in distress. Couples counseling has been shown to be effective in preventing and alleviating relationship distress, yet many couples do not seek professional help, due to cost, logistic, and discomfort in disclosing private problems. In this paper, we describe our efforts towards the development a fully automated couples counselor robot, and focus specifically on the problem of identifying and processing “collaborative responses”, in which a human couple co-construct a response to a query from the robot. We present an analysis of collaborative responses obtained from a pilot study, then develop a data-driven model to detect end of collaborative responses for regulating turn taking during a counseling session. Our model uses a combination of multimodal features, and achieves an offline weighted F-score of 0.81. Finally, we present findings from a quasi-experimental study with a robot facilitating a counseling session to promote intimacy with romantic couples. Our findings suggest that the session improves couples intimacy and positive affect. An online evaluation of the end-of-collaborative-response model demonstrates an F-score of 0.72.

Group meetings are ubiquitous, with millions of meetings held across the world every day. However, meeting quality, group performance, and outcomes are challenged by a variety of dysfunctional behaviors, unproductive social dynamics, and lack of experience in conducting efficient and productive meetings. Previous studies have shown that meeting facilitators can be advantageous in helping groups reach their goals more effectively, but many groups do not have access to human facilitators due to a lack of resources or other barriers. In this paper, we describe the development of a multimodal robotic meeting facilitator that can improve the quality of small group decision-making meetings. This automated group facilitation system uses multimodal sensor inputs (user gaze, speech, prosody, and proxemics), as well as inputs from a tablet application, to intelligently enforce meeting structure, promote time management, balance group participation, and facilitate group decision-making processes. Results of a between-subject study of 20 user groups (N=40) showed that the robot facilitator is accepted by group members, is effective in enforcing meeting structure, and users found it helpful in balancing group participation. We also report design implications derived from the findings of our study.

Oral presentations are important and challenging tasks that most people struggle with due to public speaking anxiety. Virtual agents have been shown to be effective communication skills training coaches and effective as virtual audiences for inducing anxiety in public speaking exposure therapy. To date, there are fewer research interventions, including virtual agents, that are designed to assist presenters manage their anxiety in real-time during presentations. Our research attempts to fill this gap by exploring ways in which technologies, including virtual agents, can help reduce anxiety during presentations. To help presenters manage their anxiety, we will design and evaluate an automated real-time framework for detecting public speaking anxiety and explore behavioral just-in-time techniques that presenters can use while presenting. Our automated framework will be guided by a physiological detection model, use a virtual agent as the user interface, and will assist presenters during their presentation.

Our research explores the development of new interaction formats for oral presentations that leverage a life-sized virtual agent that co-delivers a scientific talk with a human presenter. We developed a taxonomy of 36 novel interaction formats as well as 37 roles the agent can take on in co-presentations. We evaluated the impact of these formats and roles by selecting 10 from the taxonomy and recording brief presentations on the same topic using the different formats. Judges ranked dynamic agent roles higher on engagement and rated non-standard interaction formats no lower on appropriateness, compared to standard turn-taking co-presentations.

Conversational humanoid robots are being increasingly used for health education and counseling. Prior research provides mixed indications regarding the best modalities to use for these systems, including user inputs spanning completely constrained multiple choice options vs. unconstrained speech, and embodiments of humanoid robots vs. virtual agents, especially for potentially sensitive health topics such as breastfeeding. We report results from an experiment comparing five different interface modalities, finding that all result in significant increases in user knowledge and intent to adhere to recommendations, with few differences among them. Users are equally satisfied with constrained (multiple choice) touch screen input and unconstrained speech input, but are relatively unsatisfied with constrained speech input. Women find conversational robots are an effective, safe, and non-judgmental medium for obtaining information about breastfeeding.

Current slideware, such as PowerPoint, reinforces the delivery of linear oral presentations. In settings such as question answering sessions or review lectures, more extemporaneous and dynamic presentations are required. An intelligent system that can automatically identify and display the slides most related to the presenter's speech, allows for more speaker flexibility in sequencing their presentation. We present Quester, a system that enables fast access to relevant presentation content during a question answering session and supports nonlinear presentations led by the speaker. Given the slides contents and notes, the system ranks presentation slides based on semantic closeness to spoken utterances, displays the most related slides, and highlights the corresponding content keywords in slide notes. The design of our system was informed by findings from interviews with expert presenters and analysis of recordings of lectures and conference presentations. In a within-subjects study comparing our dynamic support system with a static slide navigation system during a question answering session, presenters expressed a strong preference for our system and answered the questions more efficiently using our system.

When deployed on mobile devices, virtual agents have the potential to deliver advice regarding medical conditions, as well as provide a ubiquitous channel for health education and behavior change for a variety of chronic health conditions. We describe design guidelines for mobile agent dialogues to support chronic disease management, a general-purpose smartphone-based architecture for a conversational virtual agent that simulates face-to-face health counseling conversations with patients, and an initial agent implementation that provides counseling to patients with the chronic heart condition atrial fibrillation in conjunction with a mobile heart rhythm monitor that is attached to the back of the phone. Preliminary results from a randomized trial with 120 patients with atrial fibrillation indicate that the agent results in significant improvements in self-reported quality of life relative to a standard of care control group.

Prior studies of public speaking behavior have demonstrated that public speaking anxiety monotonically decreases with the number of co-presenters giving an oral presentation and increases with the size of the audience. However, speaker "hand off" behavior---the verbal and nonverbal cues used to transition from one speaker to another---and its effect on speaker anxiety and presentation quality has not been systematically studied. In this work we report on two empirical studies of speaker hand-off behavior used during human co-presentations. We find that the cues used for hand-offs during prepared and rehearsed presentations differ significantly from the cues observed in face-to-face conversational turn-taking. We describe two systems that leverage automatic recognition of these verbal and nonverbal cues to drive hand-offs during co-presentations with a life-sized virtual agent.

The visual design of virtual agents and scenes presents developers with a plethora of choices. We conducted two studies in which we manipulated the design of a virtual health counselor agent's attire and its virtual environment. Our studies demonstrate that virtual agents dressed to fit their health care role are perceived to be more professional, trustworthy, reassuring, and more persuasive regarding medical decisions, compared to agents whose appearance is not role-appropriate. Furthermore, the effect of an agent's attire is greater than that of its virtual environment. The results of this research can help create visual guidelines for the development of virtual agents in tasks that rely on user trust.

Longitudinal agent-based interventions only work if people continue using them on a regular basis, thus identifying users who are at risk of disengaging from these applications is important for retention and efficacy. We develop machine learning models that predict long-term user engagement in three longitudinal virtual agent-based health interventions. We achieve accuracies of 74% to 90% in predicting user dropout in a given prediction period of the intervention based on the user's past interactions with the agent. Our models contain features related to session frequency and duration, health behavior, and user-agent dialogue content. We find that the features most predictive of dropout include number of user utterances, percent of user utterances that are questions, and the percent of user health behavior goals met during the observation period. Ramifications for the design of virtual agents for longitudinal applications are discussed.

Shared decision making (SDM) is a process in which patients and clinicians work together to make medical decisions based on clinical evidence that balances risks and expected outcomes with patient preferences and values. SDM has been advocated as the ideal model for patient decision-making in healthcare. However, SDM is rarely used in practice due, in part, to limited availability of healthcare professionals trained in these techniques. In this work, we describe the design of a virtual decision coach to help women participate in SDM about prenatal testing for Down syndrome. In a quasi-experimental evaluation study, participants demonstrated significant increases in knowledge, high levels of satisfaction with their final decision and low levels of decisional conflict and regret, indicating that virtual agents can effectively perform in the role of decision coach and facilitate the implementation of SDM.

Substance misuse is a major public health problem worldwide, and substance use counseling represents an important application domain for virtual agents. We report the design of a virtual counseling agent for alcohol use disorder, and a study of user gaze for individuals with different levels of alcohol use.

This article reports on the development of capabilities for (on-screen) virtual agents and robots to support isolated older adults in their homes. A real-time architecture was developed to use a virtual agent or a robot interchangeably to interact via dialog and gesture with a human user. Users could interact with either agent on 12 different activities, some of which included on-screen games, and forms to complete. The article reports on a pre-study that guided the choice of interaction activities. A month-long study with 44 adults between the ages of 55 and 91 assessed differences in the use of the robot and virtual agent.

The fear of forgetting what to say next is a common problem in oral presentation delivery. To maintain good content coverage and reduce anxiety, many presenters use written notes in conjunction with presentation slides. However, excessive use of notes during delivery can lead to disengagement with the audience and low quality presentations. We designed IntelliPrompter, a speech-based note display system that automatically tracks a presenter’s coverage of each slide’s content and dynamically adjusts the note display interface to highlight the most likely next topic to present. We developed two versions of our intelligent teleprompters using Google Glass and computer screen displays. The design of our system was informed by findings from 36 interviews with presenters and analysis of a presentation note corpus. In a within-subjects study comparing our dynamic screen-based and Google Glass note display interfaces with a static note system, presenters and independent judges expressed a strong preference for the dynamic screen-based system.

Intimate relationships are crucially important in all human societies, yet many relationships are in some degree of distress. Couple psychotherapy has been demonstrated to be effective at reducing relationship distress, yet most couples do not seek help from professionals. Automated couples counselors could provide help to many couples who avoid professional help due to cost, logistics, or discomfort disclosing personal problems. We explore reactions to and acceptance of a humanoid robot that takes the role of a couples counselor in promoting positive communication skills among asymptomatic intimate couples. Couples were comfortable with the robot in this role; displaying intimate behavior during the counseling session. They followed the directions of the robot in practicing interpersonal communication skills, were largely satisfied with the experience, and described several advantages to working with a robot compared to human counselors or self-help materials.

Most persons with spinal cord injury (SCI) require training and support for self-care management to help prevent the development of serious secondary conditions after hospital discharge. We have designed a virtual coach system, in which an animated character engages users in simulated face-to-face conversation to provide health education and motivate healthy behavior. We conducted an exploratory study with nine participants who have SCI to examine the acceptance and attitudes towards our system. Results of the study show that participants are highly receptive of the virtual coach technology and recognize it as an effective medium to promote self-care.

Virtual agents, aka embodied conversational agents or virtual humans, are anthropomorphic animated characters that engage people in simulated face-to-face conversation. Applications of virtual agents have moved beyond entertainment and demos to real-world, serious interventions in health, training, education, and other domains. In this talk, I will focus on several conversational agents that have been developed as virtual health counselors and evaluated in large-scale randomized clinical trials. These interventions span cancer self-care, antipsychotic medication adherence, breastfeeding promotion, depression counseling, meditation and yoga guidance for treating chronic pain, preconception care promotion, exercise promotion for geriatrics patients, and more. I will discuss the technical challenges in building conversational agents for the real world, capable of interacting with thousands of patients for months or years, and the evidence base of their efficacy. I will also present a vision for how such agents can be effectively deployed in VR, and a range of promising serious applications that could be deployed using this technology.

In greeting encounters, first impressions of personality and attitude are quickly formed and might determine important relational decisions, such as the likelihood and frequency of subsequent encounters. An anthropomorphic user interface is not immune to these judgments, specifically when exhibiting social interaction skills in public spaces. A favorable impression may help engaging users in interaction and attaining acceptance for long-term interactions. We present three studies implementing a model of first impressions for initiating user interactions with an anthropomorphic museum guide agent with socio-relational skills. We focus on nonverbal behavior exhibiting personality and interpersonal attitude. In two laboratory studies, we demonstrate that impressions of an agent's personality are quickly formed based on proximity, whereas interpersonal attitude is conveyed through smile and gaze. We also found that interpersonal attitude has greater impact than personality on the user's decision to spend time with the agent. These findings are then applied to a museum guide agent exhibited at the Boston Museum of Science. In this field study, we show that employing our model increases the number of visitors engaging in interaction.

The quality of most professional oral presentations is often poor, owing to a number of factors, including public speaking anxiety. We present DynamicDuo, a system that uses an automated, life-sized, animated agent to help inexperienced speakers deliver their presentations in front of an audience. The design of the system was informed by an analysis of TED talks given by two human presenters to identify the most common dual-presentation formats and transition behaviors used. In a within-subjects study (N=12) comparing co-presenting with DynamicDuo against solo-presenting with conventional presentation software, we demonstrated that our system led to significant improvements in public speaking anxiety and speaking confidence for non-native English speakers. Judges who viewed videotapes of these presentations rated those with DynamicDuo significantly higher on speech quality and overall presentation quality for all presenters.

Loneliness and social isolation are significant problems in older adult populations. We describe a conversational agent-based system designed to provide longitudinal social support to isolated older adults. Results from an exploratory pilot study indicate that when the agent proactively draws elders into interactions, it is more effective at addressing loneliness than when the agent passively relies upon elders to initiate interactions. We discuss future research opportunities for affective computing to address this important societal problem.

In this paper we describe a prototype of a serious game designed to improve hospital patients' confidence in managing their hospital stay. We present findings of an evaluation of our visual novel computer game by comparing it with traditional approaches to increasing patient empowerment and patient self-efficacy. The results indicated a significant effect of the type of intervention on improvement in empowerment, but only for participants with gaming experience or who achieved a high level of engagement in the game. This study provides evidence that games may outperform other types of health intervention under certain conditions.

We present an empirical investigation of conversational behavior in dyadic interaction spanning multiple conversations, in the context of a developing interpersonal relationship between a health counselor and her clients. Using a longitudinal video corpus of behavior change counseling conversations, we show systematic changes in verbal and nonverbal behavior during greetings (within the first minute of conversations). Both the number of prior conversations and self-reported assessments of the strength of the interpersonal relationship are predictive of changes in verbal and nonverbal behavior.

Research on social responses to computers often assesses only first-impression reactions during a single experimental session, providing limited knowledge about the lasting effect of the results. In this work, we assess the lasting strength of social desirability bias effects on an interface designed to track exercise, manipulated to have high or low personalization (text vs. anthropomorphic conversational character). After 40 days of daily interactions by 25 participants, we found that self-reported exercise was more accurate when reported to the character vs. text. We also find that, for both conditions, participants' decision to initiate a session is greater when they have done more exercise. Moreover, we show that this effect significantly increases over time for participants in the character condition, and decreases for participants in the text condition. This study demonstrates that Media Equation effects can grow stronger or weaker over time, depending upon the presentation of the interface.

Depression affects approximately 15% of the US population, and is recognized as an important risk factor for poor outcomes among patients with various illnesses. Automated health education and behavior change programs have the potential to help address many of the shortcomings in health care. However, the role of these systems in the care of patients with depression has been insufficiently examined. In the current study, we sought to evaluate how hospitalized medical patients would respond to a computer animated conversational agent that has been developed to provide information in an empathic fashion about a patient's hospital discharge plan. In particular, we sought to examine how patients who have a high level of depressive symptoms respond to this system. Therapeutic alliance-the trust and belief that a patient and provider have in working together to achieve a desired therapeutic outcome- was used as the primary outcome measure, since it has been shown to be important in predicting outcomes across a wide range of health problems, including depression. In an evaluation of 139 hospital patients who interacted with the agent at the time of discharge, all patients, regardless of depressive symptoms, rated the agent very high on measures of satisfaction and ease of use, and most preferred receiving their discharge information from the agent compared to their doctors or nurses in the hospital. In addition, we found that patients with symptoms indicative of major depression rated the agent significantly higher on therapeutic alliance compared to patients who did not have major depressive symptoms. We conclude that empathic agents represent a promising technology for patient assessment, education and counseling for those most in need of comfort and caring in the inpatient setting.

We describe an animated, conversational computer agent designed to promote antipsychotic medication adherence among patients with schizophrenia. In addition to medication adherence, the agent also promotes physical activity and system usage, and includes verbal and nonverbal behavior designed to foster a therapeutic alliance with patients. We discuss special considerations in designing interventions for this patient population, and challenges in developing and evaluating conversational agents in the mental health domain. Results from a pilot evaluation study of the agent indicate that it is accepted and effective.

We describe a series of experiments with an agent designed to model human conversational touch-capable of physically touching users in synchrony with speech and other nonverbal communicative behavior-and its use in expressing empathy to users in distress. The agent is composed of an animated human face that is displayed on a monitor affixed to the top of a human mannequin, with touch conveyed by an air bladder that squeezes a user's hand. We demonstrate that when touch is used alone, hand squeeze pressure and number of squeezes are associated with user perceptions of affect arousal conveyed by an agent, while number of squeezes and squeeze duration are associated with affect valence. We also show that, when affect-relevant cues are present in facial display, speech prosody, and touch used simultaneously by the agent, facial display dominates user perceptions of affect valence, and facial display and prosody are associated with affect arousal, while touch had little effect. Finally, we show that when touch is used in the context of an empathic, comforting interaction (but without the manipulation of affect cues in other modalities), it can lead to better perceptions of relationship with the agent, but only for users who are comfortable being touched by other people.

Ninety million Americans have inadequate health literacy, resulting in a reduced ability to read and follow directions in the healthcare environment. We describe an animated, empathic virtual nurse interface for educating and counseling hospital patients with inadequate health literacy in their hospital beds at the time of discharge. The development methodology, design rationale, and two iterations of user testing are described. Results indicate that hospital patients with low health literacy found the system easy to use, reported high levels of satisfaction, and most said they preferred receiving the discharge information from the agent over their doctor or nurse. Patients also expressed appreciation for the time and attention provided by the virtual nurse, and felt that it provided an additional authoritative source for their medical information.

We present an empirical study of the effect of a computer agent designed to engage a user in a persuasive counseling dialogue on attitudes towards regular exercise. We used two manipulations: (1) how closely the agent simulated human conversation, using either an embodied conversational agent (ECA) or a text-only agent, and (2) whether the agent attempted to build a user-agent relationship through social dialogue. Participants demonstrated a significant increase in positive attitudes (persuasion) following the persuasive dialogue; however, this change was significantly smaller when the agent used social dialogue. Participants' perceptions of the dialogue were most positive for an ECA with social dialogue, or a text-only agent without.

Relational agents—computational artifacts designed to build and maintain long-term social–emotional relationships with users—may provide an effective interface modality for older adults. This is especially true when the agents use simulated face-to-face conversation as the primary communication medium, and for applications in which repeated interactions over long time periods are required, such as in health behavior change. In this article, we discuss the design of a relational agent for older adults that plays the role of an exercise advisor, and report on the results of a longitudinal study involving 21 adults aged 62–84, half of whom interacted with the agent daily for 2 months in their homes and half who served as a standard-of-care control. Results indicate the agent was accepted and liked, and was significantly more efficacious at increasing physical activity (daily steps walked) than the control.

This research investigates the meaning of “human-computer relationship” and presents techniques for constructing, maintaining, and evaluating such relationships, based on research in social psychology, sociolinguistics, communication and other social sciences. Contexts in which relationships are particularly important are described, together with specific benefits (like trust) and task outcomes (like improved learning) known to be associated with relationship quality. We especially consider the problem of designing for long-term interaction, and define relational agents as computational artifacts designed to establish and maintain long-term social-emotional relationships with their users. We construct the first such agent, and evaluate it in a controlled experiment with 101 users who were asked to interact daily with an exercise adoption system for a month. Compared to an equivalent task-oriented agent without any deliberate social-emotional or relationship-building skills, the relational agent was respected more, liked more, and trusted more, even after four weeks of interaction. Additionally, users expressed a significantly greater desire to continue working with the relational agent after the termination of the study. We conclude by discussing future directions for this research together with ethical and other ramifications of this work for HCI designers.

This paper addresses the issue of designing embodied conversational agents that exhibit appropriate posture shifts during dialogues with human users. Previous research has noted the importance of hand gestures, eye gaze and head nods in conversations between embodied agents and humans. We present an analysis of human monologues and dialogues that suggests that postural shifts can be predicted as a function of discourse state in monologues, and discourse and conversation state in dialogues. On the basis of these findings, we have implemented an embodied conversational agent that uses Collagen in such a way as to generate postural shifts.

Building trust with users is crucial in a wide range of applications, such as financial transactions, and some minimal degree of trust is required in all applications to even initiate and maintain an interaction with a user. Humans use a variety of relational conversational strategies, including small talk, to establish trusting relationships with each other. We argue that such strategies can also be used by interface agents, and that embodied conversational agents are ideally suited for this task given the myriad cues available to them for signaling trustworthiness. We describe a model of social dialogue, an implementation in an embodied conversation agent, and an experiment in which social dialogue was demonstrated to have an effect on trust, for users with a disposition to be extroverts.

The Behavior Expression Animation Toolkit (BEAT) allows animators to input typed text that they wish to be spoken by an animated human figure, and to obtain as output appropriate and synchronized nonverbal behaviors and synthesized speech in a form that can be sent to a number of different animation systems. The nonverbal behaviors are assigned on the basis of actual linguistic and contextual analysis of the typed text, relying on rules derived from extensive research into human conversational behavior. The toolkit is extensible, so that new rules can be quickly added. It is designed to plug into larger systems that may also assign personality profiles, motion characteristics, scene constraints, or the animation styles of particular animators.

Prior research into embodied interface agents has found that users like them and find them engaging. In this paper, we argue that embodiment can serve an even stronger function if system designers use actual human conversational protocols in the design of the interface. Communicative behaviors such as salutations and farewells, conversational turn-taking with interruptions, and referring to objects using pointing gestures are examples of protocols that all native speakers of a language already know how to perform and that can thus be leveraged in an intelligent interface. We discuss how these protocols are integrated into Rea, an embodied, multi-modal conversational interface agent who acts as a real-estate salesperson, and we show why embodiment is required for their successful implementation.

In this paper, we argue for embodied corrversational characters as the logical extension of the metaphor of human - computer interaction as a conversation. We argue that the only way to fully model the richness of human I&+ to-face communication is to rely on conversational analysis that describes sets of conversational behaviors as fi~lfilling conversational functions, both interactional and propositional. We demonstrate how to implement this approach in Rea, an embodied conversational agent that is capable of both multimodal input understanding and output generation in a limited application domain. Rea supports both social and task-oriented dialogue. We discuss issues that need to be addressed in creating embodied conversational agents, and describe the architecture of the Rea interface.

The Digestor system automatically converts web-based documents designed for desktop viewing into formats appropriate for handheld devices with small display screens, such as Palm-PCs, PDAs, and cellular phones. Digestor employs a heuristic planning algorithm and a set of structural page transformations to produce the ‘best’ looking document for a given display size. Digestor can also be instructed, via a scripting language, to render portions of documents, thereby avoiding navigation through many screens of information. Two versions of Digestor have been deployed, one that re-authors HTML into HTML for conventional browsers and one that converts HTML into HDML for Phone.com's micro-browsers. Digestor provides a crucial technology for rapidly accessing, scanning and processing information from arbitrary web-based documents from any location reachable by wired or unwired communication.

MultiLex is a lexer generator designed to facilitate creation of lexical analyzers, particularly lexical analyzers for LALR(1) parsers of legacy languages. Innovative features of MultiLex include its pipeline architecture, lexical pattern-matching over a larger space of objects than just characters, reconfigurability for languages that include sub-languages, and lexically-scoped dictionary mechanism. We discuss the place of lexers in re-engineering of legacy languages, the features of MultiLex, and compare it to prior work on lexers. © 1997 by John Wiley & Sons, Ltd.

A basic agent has been constructed which integrates limited natural language understanding and generation, temporal planning and reasoning, plan execution, simulated symbolic perception, episodic memory, and some general world knowledge. The agent is cast as a robot submarine operating in a two‐dimensional simulated “Seaworld” about which it has only partial knowledge. It can communicate with people in a vocabulary of about 800 common English words using a medium coverage grammar. The agent maintains an episodic memory of events in its life and has a limited ability to reflect on those events. A person can make statements to the agent, ask it questions, and give it commands. In response to commands, a temporal task planner is invoked to synthesize a plan, which is then executed at an appropriate future time. A large variety of temporal references in natural language are interpreted with respect to agent time. The agent can form and retain compound future plans, and replan in response to new information or new commands. Natural language verbs are represented in a state transition semantics for compatibility with the planner. The agent is able to give terse answers to questions about its past experiences, present activities and perceptions, future intentions, and general knowledge. No other artificial intelligence artifact with this range of capabilities has previously been constructed.