Public attitudes toward robots are often shaped by indirect exposure (e.g., media, staged demos), leaving open how direct, hands-on experience influences acceptance. In this study, we investigate how interacting with Boston Dynamics’ Spot, an agile, state-of-the-art quadruped robot, in a public pop-up booth affects perceptions of comfort and suitability across everyday and high-stakes environments. In a walk-up, 10-week pop-up booth, participants (N=753) completed pre–post surveys before and after driving Spot within curated Drive Scenes (Factory, Home, Hospital, Outdoor/Disaster). Measures captured comfort encountering robots and perceived suitability across Rated Contexts (RCs), affective reactions, and open-ended reflections. Hands-on control significantly increased comfort across all RCs, with the largest gains in Outdoor/Disaster, and increased perceived suitability—most in Home/Office/Hospital where baselines were lower. Improvements generalized beyond the experienced Drive Scene to other contexts. Age, gender, and prior familiarity moderated baseline levels and some changes, but hands-on exposure raised scores for all groups and attenuated several gaps. Thematic analysis showed memorable moments tied to locomotion, terrain adaptation, and expressive tilt; imagined roles consistently emphasized domestic assistance (e.g., cleaning, mobility), with entertainment/play and companionship emerging post-interaction. Together, these results demonstrate that brief, agency-granting encounters with a high-capability quadruped can broaden where people see robots as appropriate and diversify envisioned roles, offering a scalable model for public-facing HRI that fosters comfort, enthusiasm, and acceptance.

Conversational breakdowns in close relationships are deeply shaped by personal histories and emotional context, yet most NLP research treats conflict detection as a general task, overlooking the relational dynamics that influence how messages are perceived.In this work, we leverage nonviolent communication (NVC) theory to evaluate LLMs in detecting conversational breakdowns and assessing how relationship backstory influences both human and model perception of conflicts.Given the sensitivity and scarcity of real-world datasets featuring conflict between familiar social partners with rich personal backstories, we contribute the PERSONACONFLICTS CORPUS 1 , a dataset of N = 5, 772 naturalistic simulated dialogues spanning diverse conflict scenarios between friends, family members, and romantic partners.Through a controlled human study, we annotate a subset of dialogues and obtain finegrained labels of communication breakdown types on individual turns, and assess the impact of backstory on human and model perception of conflict in conversation.We find that the polarity of relationship backstories significantly shifted human perception of communication breakdowns and impressions of the social partners, yet models struggle to meaningfully leverage those backstories in the detection task.Additionally, we find that models consistently overestimate how positively a message will make a listener feel.Our findings underscore the critical role of personalization to relationship contexts in enabling LLMs to serve as effective mediators in human communication for authentic connection.

This research designs and applies inclusive child-robot interactions for collaborative creativity, where elementary school children and a social robot collaboratively create and publish picture stories. The robot offers creativity scaffolding during parts of the creative process of storytelling through social interactions such as feedback, question asking, divergent thinking, and positive reinforcement. The collaborative tasks and robot interactions are personalized for neurodivergent children's unique needs. Through a five-session user study with 32 children (ages 5–9) over 8 months, we investigate the impact of the social robot on children's exhibited creativity in storytelling over time, their creative interactions with the robot, and their perceptions of the robot as a creative collaborator. Our research revealed that inclusive design practices eliminated creative barriers for children with neurodevelopmental disorders. The robot's creativity scaffolding interactions positively influenced children's verbal creativity in storytelling, and had an influence on their storytelling creative process. After multiple sessions interacting with the robot, we observed the emergence of diverse creator styles among neurodivergent learners. We propose Inclusive Co-creative Child-robot Interaction (ICCRI) guidelines for fostering creativity in children, and accommodating diverse creator styles in complex, open-ended creative tasks.

Despite living in an increasingly connected world, social isolation is a prevalent issue today. While social robots have been explored as tools to enhance social connection through companionship, their potential as asynchronous social platforms for fostering connection towards humanity has received less attention. In this work, we introduce the design of a social support companion that facilitates the exchange of emotionally relevant stories and scaffolds reflection to enhance feelings of connection via five design dimensions. We investigate how social robots can serve as “social proxies” facilitating human stories, passing stories from other human narrators to the user. To this end, we conduct a real-world deployment of 40 robot stations in users' homes over the course of two weeks. Through thematic analysis of user interviews, we find that social proxy robots can foster connection towards other people's experiences via mechanisms such as identifying connections across stories or offering diverse perspectives. We present design guidelines from our study insights on the use of social robot systems that serve as social platforms to enhance human empathy and connection.

Vocal health plays a crucial role in peoples' lives, significantly impacting their communicative abilities and interactions. However, despite the global prevalence of voice disorders, many lack access to convenient diagnosis and treatment. This paper introduces VocalAgent, an audio large language model (LLM) to address these challenges through vocal health diagnosis. We leverage Qwen-Audio-Chat fine-tuned on three datasets collected in-situ from hospital patients, and present a multifaceted evaluation framework encompassing a safety assessment to mitigate diagnostic biases, cross-lingual performance analysis, and modality ablation studies. VocalAgent demonstrates superior accuracy on voice disorder classification compared to state-of-the-art baselines. Its LLM-based method offers a scalable solution for broader adoption of health diagnostics, while underscoring the importance of ethical and technical validation.

In this paper, we introduce a novel conceptual model for a robot's behavioral adaptation in its long-term interaction with humans, integrating dynamic robot role adaptation with principles of flow experience from psychology. This conceptualization introduces a hierarchical interaction objective grounded in the flow experience, serving as the overarching adaptation goal for the robot. This objective intertwines both cognitive and affective sub-objectives and incorporates individual and group-level human factors. The dynamic role adaptation approach is a cornerstone of our model, highlighting the robot's ability to fluidly adapt its support roles-from leader to follower-with the aim of maintaining equilibrium between activity challenge and user skill, thereby fostering the user's optimal flow experiences. Moreover, this work delves into a comprehensive exploration of the limitations and potential applications of our proposed conceptualization. Our model places a particular emphasis on the multi-person HRI paradigm, a dimension of HRI that is both under-explored and challenging. In doing so, we aspire to extend the applicability and relevance of our conceptualization within the HRI field, contributing to the future development of adaptive social robots capable of sustaining long-term interactions with humans.

Foundation models are becoming valuable tools in medicine. Yet despite their promise, the best way to leverage Large Language Models (LLMs) in complex medical tasks remains an open question. We introduce a novel multi-agent framework, named **M**edical **D**ecision-making **Agents** (**MDAgents**) that helps to address this gap by automatically assigning a collaboration structure to a team of LLMs. The assigned solo or group collaboration structure is tailored to the medical task at hand, a simple emulation inspired by the way real-world medical decision-making processes are adapted to tasks of different complexities. We evaluate our framework and baseline methods using state-of-the-art LLMs across a suite of real-world medical knowledge and clinical diagnosis benchmarks, including a comparison of LLMs’ medical complexity classification against human physicians. MDAgents achieved the **best performance in seven out of ten** benchmarks on tasks requiring an understanding of medical knowledge and multi-modal reasoning, showing a significant **improvement of up to 4.2\%** ($p$ < 0.05) compared to previous methods' best performances. Ablation studies reveal that MDAgents effectively determines medical complexity to optimize for efficiency and accuracy across diverse medical tasks. Notably, the combination of moderator review and external medical knowledge in group collaboration resulted in an average accuracy **improvement of 11.8\%**. Our code can be found at https://github.com/mitmedialab/MDAgents.

Given the widespread prevalence of depression and its consequential impact on individuals and society, it is crucial to obtain objective measures for early diagnosis and intervention. As a multidisciplinary topic, these objective measures should be interpretable and accessible to health care professionals, ensuring effective collaboration and treatment planning in the realm of mental health care. Even though current automated depression diagnosis approaches improved over the last decade, a critical gap exists as they often lack affect-specificity and interpretability, limiting their practical application and potential impact on mental health care. In particular, interpretability from temporal activities from videos when deep models are used is not fully explored. In this study, we present a novel framework for analyzing Deep Neural Networks’ decisions when trained on facial videos, specifically focusing on automatic depression severity diagnosis. By fine-tuning Deep Convolutional Neural Networks (DCNN) pre-trained on Action Recognition datasets on depression severity facial videos from AVEC depression dataset, our framework is able to interpret the model’s saliency maps by examining face regions and temporal expression semantics. Our approach generates both visual and quantitative explanations for the model’s decisions, providing greater insight into its reasoning. In addition to this interpretability, our video-based modeling has improved upon previous single-face benchmarks for visual depression diagnosis, resulting in enhanced predictive performance. Overall, our work demonstrates the successful development of a framework capable of generating hypotheses from a facial model’s decisions while simultaneously improving depression’s predictive capabilities.

Social support plays a crucial role in managing and enhancing one's mental health and well-being. In order to explore the role of a robot's companion-like behavior on its therapeutic interventions, we conducted an eight-week-long deployment study with seventy participants to compare the impact of (1) acontrol robot with only assistant-like skills, (2) acoach-like robot with additional instructive positive psychology interventions, and (3) acompanion-like robot that delivered the same interventions in a peer-like and supportive manner. The companion-like robot was shown to be the most effective in building a positive therapeutic alliance with people, enhancing participants' well-being and readiness for change. Our work offers valuable insights into how companion AI agents could further enhance the efficacy of the mental health interventions by strengthening their therapeutic alliance with people for long-term mental health support.

Pedagogical agent research has yielded fruitful results in both academic skill learning and meta-cognitive skill acquisition, often studied in instructional or peer-to-peer paradigms. In the past decades, child-centric pedagogical research, which emphasizes the learner's active participation in learning with self-motivation, curiosity, and exploration, has attracted scholarly attention. Studies show that combining child-driven pedagogy with appropriate adult guidance leads to efficient learning and a strengthened feeling of self-efficacy. However, research on using social robots for guidance in child-driven learning still remains open and under-explored. In our study, we focus on children's exploration as the vehicle in literacy learning and develop a social robot companion that provides guidance to encourage and motivate children to explore during a storybook reading interaction. To investigate the effect of the robot's explorative guidance, we compare it against a control condition in which children have full autonomy to explore and read the storybooks. We conduct a between-subjects study with 31 children aged 4 to 6, and the result shows that children who receive explorative guidance from the social robot exhibit a growing trend of self-exploration. Further, children's self-exploration in the explorative guidance condition is found correlated to their learning outcome. We conclude the study with recommendations for designing social agents to guide children's exploration and future research directions in child-centric AI-assisted pedagogy.

As we move closer to real-world social AI systems, AI agents must be able to deal with multiparty (group) conversations. Recognizing and interpreting multiparty behaviors is challenging, as the system must recognize individual behavioral cues, deal with the complexity of multiple streams of data from multiple people, and recognize the subtle contingent social exchanges that take place amongst group members. To tackle this challenge, we propose the Multiparty-Transformer (Multipar- T), a transformer model for multiparty behavior modeling. The core component of our proposed approach is Crossperson Attention, which is specifically designed to detect contingent behavior between pairs of people. We verify the effectiveness of Multipar-T on a publicly available video-based group engagement detection benchmark, where it outperforms state-of-the-art approaches in average F-1 scores by 5.2% and individual class F-1 scores by up to 10.0%. Through qualitative analysis, we show that our Crossperson Attention module is able to discover contingent behaviors.

High-quality parent-child conversational interactions are crucial for children's social, emotional, and cognitive development. However, many children have limited exposure to these interactions at home. As increasingly accessible and scalable interventions in child development, interactive technologies, such as social robots, have great potential for facilitating parent-child interactions. However, such technology-based interventions are still underexplored, as the technologies' limited ability to understand the social-emotional dynamics of human dyadic interactions impedes their effective delivery of timely, adaptive interventions. To advance research on resolving this roadblock, we present a “dyadic affect in multimodal interaction-parent to child” (DAMI-P2C) dataset collected during a study of 34 parent-child pairs, where parents and children (3-7 years old) engaged in reading storybooks together. In contrast to existing public datasets for social-emotional behaviors in dyadic interactions, each instance for both participants in our dataset was annotated for affect by three labelers. Additionally, the dataset contains audiovisual recordings as well as each dyad's sociodemographic profiles, co-reading behaviors, affect labels, and body joints. We describe the dataset's main characteristics and provide a preliminary analysis of the interrelations between sociodemographic profiles, co-reading behaviors, and affect labels. The dataset provides us with useful insights into the computing and social science fields.

Despite the increase in awareness and support for mental health, college students' mental health is reported to decline every year in many countries. Several interactive technologies for mental health have been proposed and are aiming to make therapeutic service more accessible, but most of them only provide one-way passive contents for their users, such as psycho-education, health monitoring, and clinical assessment. We present a robotic coach that not only delivers interactive positive psychology interventions but also provides other useful skills to build rapport with college students. Results from our on-campus housing deployment feasibility study showed that the robotic intervention showed significant association with increases in students' psychological well-being, mood, and motivation to change. We further found that students' personality traits were associated with the intervention outcomes as well as their working alliance with the robot and their satisfaction with the interventions. Also, students' working alliance with the robot was shown to be associated with their pre-to-post change in motivation for better well-being. Analyses on students' behavioral cues showed that several verbal and nonverbal behaviors were associated with the change in self-reported intervention outcomes. The qualitative analyses on the post-study interview suggest that the robotic coach's companionship made a positive impression on students, but also revealed areas for improvement in the design of the robotic coach. Results from our feasibility study give insight into how learning users' traits and recognizing behavioral cues can help an AI agent provide personalized intervention experiences for better mental health outcomes.

In the past two decades, human-robot interaction (HRI) researchers have increasingly deployed autonomous and reliable robots long-term in various social contexts including the home. Our work provides a mixed-method approach for analyzing older adults' long-term robot usage data patterns combining quantitative data of robot usage logs with qualitative descriptions from participants' own experience. Overall, this provides a fuller picture to how older adults use and experience social robots in their homes. Our work involves a robot hosting period for at least a month (up to 12 months) in older adults' homes with an experience debrief session held a month into the robot hosting time period. We propose reflections on the novelty effect with respect to older adults' usage data and highlight feelings of guilt, the robot's proactivity and movement, meeting (or not meeting) user expectations, and the robot's persona as key aspects of the hosting experience that promoted usage or non-usage. Finally, we provide design guidelines for structuring future mixed-method long-term robot usage studies being mindful of ethical considerations in this space.

Social agents have been mostly designed to engage with children one-on-one as tutors or learning peers. Besides this child-robot dyadic interaction paradigm, they have the potential to empower parents to more actively interact with their children. Robot-assisted parent-child interaction could be a sustainable future approach for promoting children’s in-home learning. Motivated by this new design direction, this work takes an iterative design approach to explore how we design triadic interactions through "lived technology experiences" and interviews. For 3-6 weeks, we deployed and remotely teleoperated a social robot in the homes of 12 families with 3-7-year-old children to engage in a triadic story reading activity with both parent and child for six 25-min sessions. Before and after the deployment, we conducted a semi-structured interview with participants on their triadic interaction experience and desired robot design features. The results of our qualitative analysis show that social robots can improve various aspects of parent-child interaction. We propose design guidelines for robot-assisted parent-child interactions at home, the considerations of participants’ values around technology design, and promotion of their long-term lived technology experiences as critical sources for design knowledge.

As social robots rapidly become mainstream technologies, it is critical for HRI researchers and practitioners to consider their societal and ethical impacts as well as their ability to perpetuate or mitigate intersectional social inequities and hierarchies relating to race, class, gender, disability, and other social axes. Through an equity, ethics, and justice-centered audit of human-robot interaction (HRI) scholarship, we reveal how the HRI community has engaged with these topics over the past two decades. We use the five senses ethical framework that has been proposed specifically for use in HRI contexts to perform the review paired with an analysis of equity and justice. We then expand the Design Justice framework (a framework for analyzing how design impacts society and distributes benefits and burdens to society through the lenses of equity, values, scope, ownership, and accountability) to HRI contexts through the inclusion of HRI-specific topics such as autonomy, transparency, deception, and policies. We invite researchers and practitioners to explore the HRI Equitable Design framework to work towards designing equitable and inclusive HRI research studies and technologies.

Recent state-of-the-art approaches in opendomain dialogue include training end-to-end deep-learning models to learn various conversational features like emotional content of response, symbolic transitions of dialogue contexts in a knowledge graph and persona of the agent and the user, among others.While neural models have shown reasonable results, modelling the cognitive processes that humans use when conversing with each other may improve the agent's quality of responses.A key element of natural conversation is to tailor one's response such that it accounts for concepts that the speaker and listener may or may not know and the contextual relevance of all prior concepts used in conversation.We show that a rich representation and explicit modeling of these psychological processes can improve predictions made by existing neural network models.In this work, we propose a novel probabilistic approach using Markov Random Fields (MRF) to augment existing deep-learning methods for improved next utterance prediction.Using human and automatic evaluations, we show that our augmentation approach significantly improves the performance of existing state-ofthe-art retrieval models for open-domain conversational agents.

Parent-child nonverbal communication plays a crucial role in understanding their relationships and assessing their interaction styles. However, prior works have seldom studied the exchange of these nonverbal cues between the dyad and focused on isolated cues from one person at a time. In contrast, this work analyzes both parents' and children's individual and dyadic nonverbal behaviors in relation to their four relationship characteristics, i.e., child temperament, parenting style, parenting stress, and home literacy environment. We utilize a state-of-the-art feature selection framework on a dataset of 31 parent-child interactions to automatically extract and select a set of temporal nonverbal behaviors as key indicators of the dyad's relationship characteristics. The results show that relationship characteristics were associated with both individuals' and dyads' nonverbal behaviors. This finding highlights the importance of accounting for both individual- and dyad-scale nonverbal behaviors when predicting dyadic relationship characteristics as well as the potential limitations of utilizing single persons' nonverbal data in isolation. It therefore motivates future work on this topic to take a holistic and relational approach. The dataset and extracted nonverbal data are made public to aid the development of automated detection tools for parent-child relationship characteristics that trains on visual recordings of their dyadic interactions.

Self-disclosure is an important part of mental health treatment process. As interactive technologies are becoming more widely available, many AI agents for mental health prompt their users to self-disclose as part of the intervention activities. However, most existing works focus on linguistic features to classify self-disclosure behavior, and do not utilize other multi-modal behavioral cues. We present analyses of people's non-verbal cues (vocal acoustic features, head orientation and body gestures/movements) exhibited during self-disclosure tasks based on the human-robot interaction data collected in our previous work. Results from the classification experiments suggest that prosody, head pose, and body postures can be independently used to detect self-disclosure behavior with high accuracy (up to 81%). Moreover, positive emotions, high engagement, self-soothing and positive attitudes behavioral cues were found to be positively correlated to self-disclosure. Insights from our work can help build a self-disclosure detection model that can be used in real time during multi-modal interactions between humans and AI agents.

Intent recognition models, which match a written or spoken input's class in order to guide an interaction, are an essential part of modern voice user interfaces, chatbots, and social robots. However, getting enough data to train these models can be very expensive and challenging, especially when designing novel applications such as real-world human-robot interactions. In this work, we first investigate how much training data is needed for high performance in an intent classification task. We train and evaluate BiLSTM and BERT models on various subsets of the ATIS and Snips datasets. We find that only 25 training examples per intent are required for our BERT model to achieve 94% intent accuracy compared to 98% with the entire datasets, challenging the belief that large amounts of labeled data are required for high performance in intent recognition. We apply this knowledge to train models for a real-world HRI application, character strength recognition during a positive psychology interaction with a social robot, and evaluate against the Character Strength dataset collected in our previous HRI study. Our real-world HRI application results also confirm that our model can produce 76% intent accuracy with 25 examples per intent compared to 80% with 100 examples. In a real-world scenario, the difference is only one additional error per 25 classifications. Finally, we investigate the limitations of our minimal data models and offer suggestions on developing high quality datasets. We conclude with practical guidelines for training BERT intent recognition models with minimal training data and make our code and evaluation framework available for others to replicate our results and easily develop models for their own applications.

More and more voice-user interfaces (VUIs), such as smart speakers like Amazon Alexa or social robots like Jibo or Cozmo, are entering multi-user environments including homes. VUIs can utilize multi-modal cues such as graphics, expressive sounds, and movement to convey social engagement, affecting how users perceive agents as social others. Reciprocal relationships with VUIs, i.e., relationships with give-and-take between the VUI and user, are of key interest as they are more likely to foster rapport and emotional engagement, and lead to successful collaboration. Through an elicitation study with three commercially available VUIs, we explore small group interactions (n = 33 participants) focused on the behaviors participants display to various VUIs to understand (1) reciprocal interactions between VUIs and participants and among small groups and (2) how participants engage with VUIs as the interface's embodiment becomes more socially capable. The discussion explores (1) theories of sociability applied to the users' behaviors seen with the VUIs, and (2) the group contexts where VUIs that build reciprocal relationships with users can become a powerful persuasive technology and a collaborative companion. We conclude the discussion with recommendations for promoting reciprocity from participants and, therefore, fostering rapport and emotional engagement in VUI interactions.

Users can provide valuable insights for designing new technologies like social robots, with the right tools and methodologies. Challenges in inviting users as co-designers of social robots is due to lack of guidelines or methodologies to (1) organize co-design processes and/or (2) engage with people long-term to develop technologies together. The main contribution of this work is to provide guidelines for long- term co-design for how other researchers can adopt long- term co-design, informed by a 12-month co-design with older adults designing a social social robot. We leveraged human- centered, tactile and experiential design activities, including participatory design, based upon the following design principles: scenario specific exploration, long-term lived experiences, supporting multiple design activities, cultivating relationships, and employing divergent and convergent processes. We present seven different sessions across three stages as examples of this methodology that build on each other to engage users as co- designers, successfully deployed in a co-design project of home social robots with 28 older adults. Lastly, we detail 10 long- term divergent-convergent co-design guidelines for designing social robots. We demonstrate the value of leveraging people's lived technology experiences and co-design activities to generate actionable social robot design guidelines, advocating for more applications of the methodology in broader contexts as well.

Artificial curiosity, based on developmental psychology concepts wherein an agent attempts to maximize its learning progress, has gained much attention in recent years. Similarly, social robots are slowly integrating into our daily lives, in schools, factories, and in our homes. In this contribution, we integrate recent advances in artificial curiosity and social robots into a single expressive cognitive architecture. It is composed of artificial curiosity and social expressivity modules and their unique link, i.e., the robot verbally and non-verbally communicates its internally estimated learning progress, or learnability, to its human companion. We implemented this architecture in an interaction where a fully autonomous robot took turns with a child trying to select and solve tangram puzzles on a tablet. During the curious robot’s turn, it selected its estimated most learnable tangram to play, communicated its selection to the child, and then attempted at solving it. We validated the implemented architecture and showed that the robot learned, estimated its learnability, and improved when its selection was based on its learnability estimation. Moreover, we ran a comparison study between curious and non-curious robots, and showed that the robot’s curiosity-based behavior influenced the child’s selections. Based on the artificial curiosity module of the robot, we have formulated an equation that estimates each child’s moment-by-moment curiosity based on their selections. This analysis revealed an overall significant decrease in estimated curiosity during the interaction. However, this drop in estimated curiosity was significantly larger with the non-curious robot, compared to the curious one. These results suggest that the new architecture is a promising new approach to integrate state-of-the-art curiosity-based algorithms to the growing field of social robots.

Prior work in affect-aware educational robots has often relied on a common belief that the relationship between student affect and learning is independent of agent behaviors (child's/robot's) or unidirectional (positive/negative but not both) throughout the entire student-robot interaction. We argue that the student affect-learning relationship should be interpreted in two contexts: (1) social learning paradigm and (2) sub-events within child-robot interaction. In our paper, we examine two different social learning paradigms where children interact with a robot that acts either as a tutor or a tutee. Sub-events within child-robot interaction are defined as task-related events occurring in specific phases of an interaction (e.g., when the child/robot gets a wrong answer). We examine sub-events at a macro level (entire interaction) and a micro level (within specific sub-events). In this paper, we provide an in-depth correlation analysis of children's facial affect and vocabulary learning. We found that children's affective displays became more predictive of their vocabulary learning when children interacted with a tutee robot who did not scaffold their learning. Additionally, children's affect displayed during micro-level events was more predictive of their learning than during macro-level events. Last, we found that the affect-learning relationship is not unidirectional, but rather is modulated by context, i.e., several affective states facilitated student learning when displayed in some sub-events but inhibited learning when displayed in others. These findings indicate that both social learning paradigm and sub-events within interaction modulate student affect-learning relationship.

Automatic speech-based affect recognition of individuals in dyadic conversation is a challenging task, in part because of its heavy reliance on manual pre-processing. Traditional approaches frequently require hand-crafted speech features and segmentation of speaker turns. In this work, we design end-to-end deep learning methods to recognize each person's affective expression in an audio stream with two speakers, automatically discovering features and time regions relevant to the target speaker's affect. We integrate a local attention mechanism into the end-to-end architecture and compare the performance of three attention implementations - one mean pooling and two weighted pooling methods. Our results show that the proposed weighted-pooling attention solutions are able to learn to focus on the regions containing target speaker's affective information and successfully extract the individual's valence and arousal intensity. Here we introduce and use a "dyadic affect in multimodal interaction - parent to child" (DAMI-P2C) dataset collected in a study of 34 families, where a parent and a child (3-7 years old) engage in reading storybooks together. In contrast to existing public datasets for affect recognition, each instance for both speakers in the DAMI-P2C dataset is annotated for the perceived affect by three labelers. To encourage more research on the challenging task of multi-speaker affect sensing, we make the annotated DAMI-P2C dataset publicly available, including acoustic features of the dyads' raw audios, affect annotations, and a diverse set of developmental, social, and demographic profiles of each dyad.

A significant number of college students suffer from mental health issues that impact their physical, social, and occupational outcomes. Various scalable technologies have been proposed in order to mitigate the negative impact of mental health disorders. However, the evaluation for these technologies, if done at all, often reports mixed results on improving users' mental health. We need to better understand the factors that align a user's attributes and needs with technology-based interventions for positive outcomes. In psychotherapy theory, therapeutic alliance and rapport between a therapist and a client is regarded as the basis for therapeutic success. In prior works, social robots have shown the potential to build rapport and a working alliance with users in various settings. In this work, we explore the use of a social robot coach to deliver positive psychology interventions to college students living in on-campus dormitories. We recruited 35 college students to participate in our study and deployed a social robot coach in their room. The robot delivered daily positive psychology sessions among other useful skills like delivering the weather forecast, scheduling reminders, etc. We found a statistically significant improvement in participants' psychological wellbeing, mood, and readiness to change behavior for improved wellbeing after they completed the study. Furthermore, students' personality traits were found to have a significant association with intervention efficacy. Analysis of the post-study interview revealed students' appreciation of the robot's companionship and their concerns for privacy.

Conversational AI agents are proliferating, embodying a range of devices such as smart speakers, smart displays, robots, cars, and more. We can envision a future where a personal conversational agent could migrate across different form factors and environments to always accompany and assist its user to support a far more continuous, personalized and collaborative experience. This opens the question of what properties of a conversational AI agent migrates across forms, and how it would impact user perception. To explore this, we developed a Migratable AI system where a user's information and/or the agent's identity can be preserved as it migrates across form factors to help its user with a task. We validated the system by designing a 2x2 between-subjects study to explore the effects of information migration and identity migration on user perceptions of trust, competence, likeability and social presence. Our results suggest that identity migration had a positive effect on trust, competence and social presence, while information migration had a positive effect on trust, competence and likeability. Overall, users report highest trust, competence, likeability and social presence towards the conversational agent when both identity and information were migrated across embodiments.

Personalized education technologies capable of delivering adaptive interventions could play an important role in addressing the needs of diverse young learners at a critical time of school readiness. We present an innovative personalized social robot learning companion system that utilizes children’s verbal and nonverbal affective cues to modulate their engagement and maximize their long-term learning gains. We propose an affective reinforcement learning approach to train a personalized policy for each student during an educational activity where a child and a robot tell stories to each other. Using the personalized policy, the robot selects stories that are optimized for each child’s engagement and linguistic skill progression. We recruited 67 bilingual and English language learners between the ages of 4–6 years old to participate in a between-subjects study to evaluate our system. Over a three-month deployment in schools, a unique storytelling policy was trained to deliver a personalized story curriculum for each child in the Personalized group. We compared their engagement and learning outcomes to a Non-personalized group with a fixed curriculum robot, and a baseline group that had no robot intervention. In the Personalization condition, our results show that the affective policy successfully personalized to each child to boost their engagement and outcomes with respect to learning and retaining more target words as well as using more target syntax structures as compared to children in the other groups.

PopBots is a hands-on toolkit and curriculum designed to help young children learn about artificial intelligence (AI) by building, programming, training, and interacting with a social robot. Today’s children encounter AI in the forms of smart toys and computationally curated educational and entertainment content. However, children have not yet been empowered to understand or create with this technology. Existing computational thinking platforms have made ideas like sequencing and conditionals accessible to young learners. Going beyond this, we seek to make AI concepts accessible. We designed PopBots to address the specific learning needs of children ages four to seven by adapting constructionist ideas into an AI curriculum. This paper describes how we designed the curriculum and evaluated its effectiveness with 80 Pre-K and Kindergarten children. We found that the use of a social robot as a learning companion and programmable artifact was effective in helping young children grasp AI concepts. We also identified teaching approaches that had the greatest impact on student’s learning. Based on these, we make recommendations for future modules and iterations for the PopBots platform.

We developed a novel early childhood artificial intelligence (AI) platform, PopBots, where preschool children train and interact with social robots to learn three AI concepts: knowledge-based systems, supervised machine learning, and generative AI. We evaluated how much children learned by using AI assessments we developed for each activity. The median score on the cumulative assessment was 70% and children understood knowledge-based systems the best. Then, we analyzed the impact of the activities on children's perceptions of robots. Younger children came to see robots as toys that were smarter than them, but their older counterparts saw them more as people that were not as smart as them. Children who performed worse on the AI assessments believed that robots were like toys that were not as smart as them, however children who did better on the assessments saw robots as people who were smarter than them. We believe early AI education can empower children to understand the AI devices that are increasingly in their lives.

Intelligent tutoring systems (ITS) provide educational benefits through one-on-one tutoring by assessing children's existing knowledge and providing tailored educational content. In the domain of language acquisition, several studies have shown that children often learn new words by forming semantic relationships with words they already know. In this paper, we present a model that uses word semantics (semantics-based model) to make inferences about a child's vocabulary from partial information about their existing vocabulary knowledge. We show that the proposed semantics-based model outperforms models that do not use word semantics (semantics-free models) on average. A subject-level analysis of results reveals that different models perform well for different children, thus motivating the need to combine predictions. To this end, we use two methods to combine predictions from semantics-based and semantics-free models and show that these methods yield better predictions of a child's vocabulary knowledge. Our results motivate the use of semantics-based models to assess children's vocabulary knowledge and build ITS that maximizes children's semantic understanding of words.

Intelligent toys and smart devices are becoming ubiquitous in children's homes. As such, it is imperative to understand how these computational objects impact children's development. Children's attribution of intelligence relates to how they perceive the behavior of these agents [6]. However, their underlying reasoning is not well understood. To explore this, we invited 30 pairs of children (4--10 years old) and their parents to assess the intelligence of mice, robots, and themselves in a maze-solving activity. Participants watched videos of mice and robots solving a maze. Then, they solved the maze by remotely navigating a robot. Solving the maze enabled participants to gain insight into the agent's mind by referencing their own experience. Children and their parents gave similar answers for whether the mouse or the robot was more intelligent and used a wide variety of explanations. We also observed developmental differences in childrens' references to agents' social-emotional attributes, strategies and performance.

Social robots are increasingly being developed for long-term interactions with children in domains such as healthcare, education, therapy, and entertainment. As such, we need to deeply understand how children's relationships with robots develop through time. However, there are few validated assessments for measuring young children's long-term relationships. In this paper, we present a pilot test of four assessments that we have adapted or created for use in this context with children aged 5--6: the Inclusion of Other in Self task, the Social-Relational Interview, the Narrative Description, and the Self-disclosure Task. We show that children can appropriately respond to these assessments with reasonably high internal reliability, and that the proposed assessments are able to capture child-robot relationship adjustments over a long-term interaction. Furthermore, we discuss gender and population differences in children's responses.

While there has been a growing body of work in child-robot interaction, we still have very little knowledge regarding young children's speaking and listening dynamics and how a robot companion should decode these behaviors and encode its own in a way children can understand. In developing a backchannel prediction model based on observed nonverbal behaviors of 4-6 year-old children, we investigate the effects of an attentive listening robot on a child's storytelling. We provide an extensive analysis of young children's nonverbal behavior with respect to how they encode and decode listener responses and speaker cues. Through a collected video corpus of peer-to-peer storytelling interactions, we identify attention-related listener behaviors as well as speaker cues that prompt opportunities for listener backchannels. Based on our findings, we developed a backchannel opportunity prediction (BOP) model that detects four main speaker cue events based on prosodic features in a child's speech. This rule-based model is capable of accurately predicting backchanneling opportunities in our corpora. We further evaluate this model in a human-subjects experiment where children told stories to an audience of two robots, each with a different backchanneling strategy. We find that our BOP model produces contingent backchannel responses that conveys an increased perception of an attentive listener, and children prefer telling stories to the BOP model robot.

Mindset has been shown to have a large impact on people's academic, social, and work achievements. A growth mindset, i.e., the belief that success comes from effort and perseverance, is a better indicator of higher achievements as compared to a fixed mindset, i.e., the belief that things are set and cannot be changed. Interventions aimed at promoting a growth mindset in children range from teaching about the brain's ability to learn and change, to playing computer games that grant brain points for effort rather than success. This work explores a novel paradigm to foster a growth mindset in young children where they play a puzzle solving game with a peer-like social robot. The social robot is fully autonomous and programmed with behaviors suggestive of it having either a growth mindset or a neutral mindset as it plays puzzle games with the child. We measure the mindset of children before and after interacting with the peer-like robot, in addition to measuring their problem solving behavior when faced with a challenging puzzle. We found that children who played with a growth-mindset robot 1) self-reported having a stronger growth mindset and 2) tried harder during a challenging task, as compared to children who played with the neutral-mindset robot. These results suggest that interacting with peer-like social robot with a growth mindset can promote the same mindset in children.

This paper investigates how a robot that can produce contingent listener response, i.e., backchannel, can deeply engage children as a storyteller. We propose a backchannel opportunity prediction (BOP) model trained from a dataset of children's dyad storytelling and listening activities. Using this dataset, we gain better understanding of what speaker cues children can decode to find backchannel timing, and what type of nonverbal behaviors they produce to indicate engagement status as a listener. Applying our BOP model, we conducted two studies, within- and between-subjects, using our social robot platform, Tega. Behavioral and self-reported analyses from the two studies consistently suggest that children are more engaged with a contingent backchanneling robot listener. Children perceived the contingent robot as more attentive and more interested in their story compared to a non-contingent robot. We find that children significantly gaze more at the contingent robot while storytelling and speak more with higher energy to a contingent robot.

A robot companion should adapt to its user's needs by learning to perform new tasks. In this paper, we present a robot playmate that learns and adapts to tasks chosen by the child on a touchscreen tablet. We aim to solve the task learning problem using an experience-based learning framework that stores human demonstrations as task instances. These instances are retrieved when confronted with a similar task in which the system generates predictions of task behaviors based on prior actions. In order to automate the processes of instance encoding, acquisition, and retrieval, we have developed a framework that gathers task knowledge through interaction with human teachers. This approach, further referred to as interactive instance-based learning (IIBL), utilizes limited information available to the robot to generate similarity metrics for retrieving instances. In this paper, we focus on introducing and evaluating a new hybrid IIBL framework using sensitivity analysis with artificial neural networks and discuss its advantage over methods using k-NNs and linear regression in retrieving instances.

This paper introduces a bounding gait control algorithm that allows a variable-speed running in the MIT Cheetah 2. A simple impulse planning algorithm is proposed to design vertical and horizontal force profiles which make net impulse on the system during one cycle zero. This design of force profiles leads to the conservation of linear momentum over a complete step, providing periodicity in horizontal and vertical velocity. When designed profiles are applied to the system, periodic orbits with an ability to change running speed are obtained. A virtual compliance control in the horizontal and vertical direction has been added onto the designed force profiles to stabilize the periodic orbits. The experimental results show that the algorithm successfully achieved untethered 3D running of the MIT Cheetah 2, with speeds ranging from 0 m/sec to 4.5 m/sec on treadmills as well as on grassy fields.

One of the key elements for building a long-term robotic companion is incorporating the ability for a robot to continuously learn and engage in new tasks. Utilizing a defined workspace that provides various shared content between human and robot could assist in this learning process. Here, we propose integrating a touchscreen tablet and a robot learner for engaging the user during human-robot interaction scenarios. The robot learner's domain-independent core reasoner follows the structure of instance-based learning which addresses the issues of acquiring knowledge, encoding cases, and learning a retrieval metric. The system utilizes demonstrations provided by the user to auto-populate the knowledge base through natural interaction methods, encodes cases based on the feature structure provided by the user, and uses an adaptive-weighting technique to design a retrieval metric with linear regression in the feature-distance space. Through a tablet environment, the user teaches a task to the robot in a shared workspace and intuitively monitors the robot's behavior and progress in real time. In this setting, the user is able to interrupt the robot and provide necessary demonstrations at the moment learning is taking place, thus providing a means to continuously engage both the participant and the robot in the learning cycle.

This paper introduces a bounding gait control algorithm that allows a successful implementation of duty cycle modulation in the MIT Cheetah 2. Instead of controlling leg stiffness to emulate a ‘springy leg’ inspired from the Spring-Loaded-Inverted-Pendulum (SLIP) model, the algorithm prescribes vertical impulse by generating scaled ground reaction forces at each step to achieve the desired stance and total stride duration. Therefore, we can control the duty cycle: the percentage of the stance phase over the entire cycle. By prescribing the required vertical impulse of the ground reaction force at each step, the algorithm can adapt to variable duty cycles attributed to variations in running speed. Following linear momentum conservation law, in order to achieve a limit-cycle gait, the sum of all vertical ground reaction forces must match vertical momentum created by gravity during a cycle. In addition, we added a virtual compliance control in the vertical direction to enhance stability. The stiffness of the virtual compliance is selected based on the eigenvalue analysis of the linearized Poincaré map and the chosen stiffness is 700 N/m, which corresponds to around 12% of the stiffness used in the previous trotting experiments of the MIT Cheetah, where the ground reaction forces are purely caused by the impedance controller with equilibrium point trajectories. This indicates that the virtual compliance control does not significantly contributes to generating ground reaction forces, but to stability. The experimental results show that the algorithm successfully prescribes the duty cycle for stable bounding gaits. This new approach can shed a light on variable speed running control algorithm.

The objective of this doctoral research is to design multimodal task-learning policies for a robotic system that targets the exchange of task rules between humans and robots. This objective is achieved through a collaborative task application during human-robot interaction where the two partners learn a task from each other and accomplish a shared goal. As a first step, a method to model human-action primitives using a pattern-recognition technique is presented. Next, algorithms are developed to generate turn-taking strategies in response to human task behaviors. The contribution of this work is in engaging robots with humans in collaborative play task by modeling statistical patterns of play behaviors and reusing previously learned knowledge to reduce the decision process. Here, results of previous work are presented, and remaining works including deploying a physically embodied agent and developing an evaluation platform are outlined.

In this paper, we discuss a methodology to build a system for a robot playmate that extracts and sequences low-level play primitives during a robot-child interaction scenario. The motivation is to provide a robot with basic knowledge of how to manipulate toys in an equivalent manner as a human does - as a first step in engaging children in cooperative play. Our approach involves the extraction of play primitives based on observation of motion gradient vectors computed from the image sequence. Hidden Markov Models (HMMs) are then used to recognize 14 different play primitives during play. Experimental results from a data set of 100 play scenarios including child subjects demonstrate 86.88% accuracy recognizing and sequencing the play primitives.

When young children play, they often manipulate toys that have been specifically designed to accommodate and stimulate their perceptual-motor skills. Robotic playmates capable of physically manipulating toys have the potential to engage children in therapeutic play and augment the beneficial interactions provided by overtaxed care givers and costly therapists. To date, assistive robots for children have almost exclusively focused on social interactions and teleoperative control. Within this paper we present progress towards the creation of robots that can engage children in manipulative play. First, we present results from a survey of popular toys for children under the age of 2 which indicates that these toys share simplified appearance properties and are designed to support a relatively small set of coarse manipulation behaviors. We then present a robotic control system that autonomously manipulates several toys by taking advantage of this consistent structure. Finally, we show results from an integrated robotic system that imitates visually observed toy playing activities and is suggestive of opportunities for robots that play with toys.

In this paper, we discuss a methodology to extract play primitives, defined as a sequence of low-level motion behaviors identified during a playing action, such as stacking or inserting a toy. Our premise is that if a robot could interpret the basic movements of a humanpsilas play, it will be able to interact with many different kinds of toys, in conjunction with its human playmate. As such, we present a method that combines motion behavior analysis and behavior sequencing, which capitalizes on the inherent characteristics found in the dynamics of play such as the limited domain of the objects and manipulation skills required. In this paper, we give details on the approach and present results from applying the methodology to a number of play scenarios.