This paper describes the development of algorithms that decide when to move, where to move, and how to look for people in a home environment. We introduce a design framework as a tool to guide the development of a social robot to proactively be with people for companionship and assistance in the home. Through a series of experiments ranging from simulations to longitudinal A/B studies, we demonstrate how to utilize the design framework to help guide the evaluation and selection of solutions. We deployed our autonomous robot in a long-term in-situ study and found our proposed approach to be more capable of being co-present with its household members compared to a baseline approach. Conducted in an industry setting, our research approach departs from typical academic practices as the motivations are inherently different. We share our perspective on the differences of industry research when developing a social robot as a commercial product.

This paper defines a dual computational framework to nonverbal communication for human-robot interactions. We use a Bayesian Theory of Mind approach to model dyadic storytelling interactions where the storyteller and the listener have distinct roles. The role of storytellers is to influence and infer the attentive state of listeners using speaker cues, and we computationally model this as a POMDP planning problem. The role of listeners is to convey attentiveness by influencing perceptions through listener responses, which we computational model as a DBN with a myopic policy. Through a comparison of state estimators trained on human-human interaction data, we validate our storyteller model by demonstrating how it outperforms current approaches to attention recognition. Then through a human-subjects experiment where children told stories to robots, we demonstrate that a social robot using our listener model more effectively communicates attention compared to alternative approaches based on signaling.

Understanding social-emotional behaviors in storytelling interactions plays a critical role in the development of interactive educational technologies for children. A challenge when designing for such interactions using technology like social robots, virtual agents, and tablets is understanding the social-emotional behaviors pertinent to storytelling-especially when emulating a natural peer-to-peer relation between the child and the technology. We present P2PSTORY, a dataset of young children (5-6 years old) engaging in natural peer-to-peer storytelling interactions with fellow classmates. The dataset consists of rich social behaviors of children without adult supervision, with each participant demonstrating being a storyteller and a listener. The dataset contains 58 video recorded sessions along with a diverse set of behavioral annotations as well as developmental and demographic profiles of each child participant. We describe the main characteristics of the dataset in addition to findings that reveal perceptual differences between adults and children when evaluating the attentiveness of listeners.

While there has been a growing body of work in child-robot interaction, we still have very little knowledge regarding young children's speaking and listening dynamics and how a robot companion should decode these behaviors and encode its own in a way children can understand. In developing a backchannel prediction model based on observed nonverbal behaviors of 4-6 year-old children, we investigate the effects of an attentive listening robot on a child's storytelling. We provide an extensive analysis of young children's nonverbal behavior with respect to how they encode and decode listener responses and speaker cues. Through a collected video corpus of peer-to-peer storytelling interactions, we identify attention-related listener behaviors as well as speaker cues that prompt opportunities for listener backchannels. Based on our findings, we developed a backchannel opportunity prediction (BOP) model that detects four main speaker cue events based on prosodic features in a child's speech. This rule-based model is capable of accurately predicting backchanneling opportunities in our corpora. We further evaluate this model in a human-subjects experiment where children told stories to an audience of two robots, each with a different backchanneling strategy. We find that our BOP model produces contingent backchannel responses that conveys an increased perception of an attentive listener, and children prefer telling stories to the BOP model robot.

This paper investigates how a robot that can produce contingent listener response, i.e., backchannel, can deeply engage children as a storyteller. We propose a backchannel opportunity prediction (BOP) model trained from a dataset of children's dyad storytelling and listening activities. Using this dataset, we gain better understanding of what speaker cues children can decode to find backchannel timing, and what type of nonverbal behaviors they produce to indicate engagement status as a listener. Applying our BOP model, we conducted two studies, within- and between-subjects, using our social robot platform, Tega. Behavioral and self-reported analyses from the two studies consistently suggest that children are more engaged with a contingent backchanneling robot listener. Children perceived the contingent robot as more attentive and more interested in their story compared to a non-contingent robot. We find that children significantly gaze more at the contingent robot while storytelling and speak more with higher energy to a contingent robot.

Though substantial research has been dedicated towards using technology to improve education, no current methods are as effective as one-on-one tutoring. A critical, though relatively understudied, aspect of effective tutoring is modulating the student's affective state throughout the tutoring session in order to maximize long-term learning gains. We developed an integrated experimental paradigm in which children play a second-language learning game on a tablet, in collaboration with a fully autonomous social robotic learning companion. As part of the system, we measured children's valence and engagement via an automatic facial expression analysis system. These signals were combined into a reward signal that fed into the robot's affective reinforcement learning algorithm. Over several sessions, the robot played the game and personalized its motivational strategies (using verbal and non-verbal actions) to each student. We evaluated this system with 34 children in preschool classrooms for a duration of two months. We saw that (1) children learned new words from the repeated tutoring sessions, (2) the affective policy personalized to students over the duration of the study, and (3) students who interacted with a robot that personalized its affective feedback strategy showed a significant increase in valence, as compared to students who interacted with a non-personalizing robot. This integrated system of tablet-based educational content, affective sensing, affective policy learning, and an autonomous social robot holds great promise for a more comprehensive approach to personalized tutoring.

We deployed an autonomous social robotic learning companion in three preschool classrooms at an American public school for two months. Before and after this deployment, we asked the teachers and teaching assistants who worked in the classrooms about their views on the use of social robots in preschool education. We found that teachers' expectations about the experience of having a robot in their classrooms often did not match up with their actual experience. These teachers generally expected the robot to be disruptive, but found that it was not, and furthermore, had numerous positive ideas about the robot's potential as a new educational tool for their classrooms. Based on these interviews, we provide a summary of lessons we learned about running child-robot interaction studies in preschools. We share some advice for future researchers who may wish to engage teachers and schools in the course of their own human-robot interaction work. Understanding the teachers, the classroom environment, and the constraints involved is especially important for microgenetic and longitudinal studies, which require more of the school's time-as well as more of the researchers' time-and is a greater opportunity investment for everyone involved.

Tega is a new expressive “squash and stretch”, Android-based social robot platform, designed to enable long-term interactions with children.

This paper describes an extended (6-session) interaction between an ethnically and geographically diverse group of 26 first-grade children and the DragonBot robot in the context of learning about healthy food choices. We find that children demonstrate a high level of enjoyment when interacting with the robot, and a statistically significant increase in engagement with the system over the duration of the interaction. We also find evidence of relationship-building between the child and robot, and encouraging trends towards child learning. These results are promising for the use of socially assistive robotic technologies for long-term one-on-one educational interventions for younger children.

People are increasingly working with robots in teams and recent research has focused on how human-robot teams function, but little attention has yet been paid to the role of social signaling behavior in human-robot teams. In a controlled experiment, we examined the role of backchanneling and task complexity on team functioning and perceptions of the robots' engagement and competence. Based on results from 73 participants interacting with autonomous humanoid robots as part of a human-robot team (one participant, one confederate, and three robots), we found that when robots used backchanneling team functioning improved and the robots were seen as more engaged. Ironically, the robots using backchanneling were perceived as less competent than those that did not. Our results suggest that backchanneling plays an important role in human-robot teams and that the design and implementation of robots for human-robot teams may be more effective if backchanneling capability is provided.