With robotics rapidly advancing, more effective human–robot interaction is increasingly needed to realize the full potential of robots for society. While spoken language must be part of the solution, our ability to provide spoken language interaction capabilities is still very limited. In this article, based on the report of an interdisciplinary workshop convened by the National Science Foundation, we identify key scientific and engineering advances needed to enable effective spoken language interaction with robotics. We make 25 recommendations, involving eight general themes: putting human needs first, better modeling the social and interactive aspects of language, improving robustness, creating new methods for rapid adaptation, better integrating speech and language with other communication modalities, giving speech and language components access to rich representations of the robot’s current knowledge and state, making all components operate in real time, and improving research infrastructure and resources. Research and development that prioritizes these topics will, we believe, provide a solid foundation for the creation of speech-capable robots that are easy and effective for humans to work with.

"The only thing that scales with undergrads is undergrads". As Computer Science course enrollments have grown, there has been a necessary increase in the number of undergraduate and graduate teaching assistants (TAs, and UTAs). TA duties often extend far beyond grading, including designing and leading lab or recitation sections, holding office hours and creating assignments. Though advanced students, TAs need proper pedagogical training to be the most effective in their roles. Training strategies have widely varied from no training at all, to semester-long prep courses. We will explore the challenges of TA training across both large and small departments. While much of the effort has focused on teams of undergraduates, most presenters have used the same tools and strategies with their graduate students. Training for TAs should not just include the mechanics of managing a classroom, but culturally relevant pedagogy. The panel will focus on the challenges of providing "just in time", and how we manage both intra-course training and department or campus led courses.

The MaGE Training curriculum prepares computer science students for the task of inclusive peer mentoring and teaching. The curriculum raises awareness of the role of social identity in learning, emphasizes active learning within computer science, and provides preparation for technical code review. This article presents an overview of the MaGE Training curriculum where it has been used to train six cohorts of near-peer mentors and its impact on more than 500 students at a liberal arts college. While rapid growth in course enrollments has presented many challenges, results suggest that the MaGE curriculum has helped to address some of these challenges by maintaining high quality feedback to, and close interaction with, introductory students. Effectiveness is evidenced through increases in mentor self-efficacy, positive impact on student belongingness and continued enrollment, and reports of buffering the instructor workload. The flexibility of the curriculum is supported through a set of modules that can be engaged with via in-person discussions or viewed remotely. This enables easier adoption of the curriculum for use at other institutions.

In this paper, we apply the contribution model of grounding to a corpus of humanhuman peer-mentoring dialogues.From this analysis, we propose effective turntaking strategies for human-robot interaction with a teachable robot.Specifically, we focus on (1) how robots can encourage humans to present and (2) how robots can signal that they are going to begin a new presentation.We evaluate the strategies against a corpus of human-robot dialogues and offer three guidelines for teachable robots to follow to achieve more humanlike collaborative dialogue.

Undergraduates are widely used in support of Computer Science (CS) departments' teaching missions as teaching assistants, peer mentors, section leaders, course assistants, and tutors. Those undergraduates engaged in teaching have the opportunity to deeply engage with CS concepts and develop key communication and social competencies. As enrollments surge, undergraduate teaching assistants (UTAs) play a larger role in student experience and outcomes. While faculty and graduate student instructional support does not necessarily increase with the number of students in our courses, the number of qualified undergraduate teaching assistants for introductory CS courses should scale with the number of students in our courses. With large courses, the significance of the UTAs' role in students' learning likely also increases. Students have relatively little interaction with the instructor, and faculty may have more challenges monitoring and supporting individual UTAs. UTAs have a major role in affecting climate in computer science courses. The climate in large courses has substantial implications for students from groups traditionally underrepresented in computing. This panel will discuss how undergraduate teaching assistants can serve as a scalable effective teaching resource that benefits both the students in the course and the UTAs themselves.

With a growing number of applications involving social human-robot interactions, there is an increasingly important role for socially responsive speech interfaces that can effectively engage the user. For example, learning companions provide both task-related feedback and motivational support for students with the goal of improving learning. As a learning companion's ability to be socially responsive increases, so do learning outcomes. This paper presents a socially responsive speech interface for an embodied, robotic learning companion. We explore two methods of social responsiveness. The first method introduces social responses into the dialogue, while the second method augments these responses with voice-adaptation based on acoustic-prosodic entrainment. We evaluate the effect of a social, voice-adaptive robotic learning companion on social variables such as social presence and rapport, and we compare this to a companion with only social dialogue and one with neither social dialogue nor voice-adaptions. We contrast the effects against those of individual factors, such as gender. We find (1) that social presence is significantly higher with a social voice-adaptive speech interface than with purely social dialogue, and (2) that females feel significantly more rapport and are significantly more persistent in interactions with a robotic learning companion than males.

The Megas and Gigas Educate (MaGE) program is a peer mentoring program being developed at Mount Holyoke College, a liberal arts college for women, for the introductory CS curriculum. Consistent with national trends, interest in CS is rising rapidly; current resources cannot meet demand while maintaining quality feedback and pedagogy. Supported by a Google Computer Science Capacity Award, MaGE has three main objectives: (1) to triple enrollment capacity over 3 years in introductory courses; (2) to increase enrollment and retention for women and other underrepresented groups; (3) to train students to educate, mentor, and support others in inclusive ways. Trained undergraduate students act as peer mentors to beginner students, providing close interaction and assisting with feedback. MaGE is currently being piloted in the introductory CS1 course. Enrolled students bring varying interests, including Art, History, Biochemistry, Economics and Engineering; most students have no prior programming experience. Each CS1 student is assigned a peer Giga Education Mentor (or GEM) in a 9:1 ratio. GEMs have undergone a rigorous training course that raises awareness of the role of social identity in learning, emphasizes active learning within computer science, and provides preparation for being technical peer mentors. While research supports the need for culturally-sensitive, inclusive training as part of the curriculum, we know of few peer-based models in CS that explicitly include this education. By building a diverse set of peer role models and connecting with the pre-existing co-curricular Megas and Gigas mentoring program, MaGE seeks to effectively engage underrepresented students in computing.

Observed frequently in human-human interactions, entrainment is a social phenomenon in which speakers become more like each other over the course of a conversation. Acoustic-prosodic entrainment occurs when individuals adapt their acoustic-prosodic speech features, such as pitch and intensity. Correlated with communicative success, naturalness, and conversational flow as well as social variables such as rapport, a dialogue system which automatically entrains has the potential to improve verbal interactions by increasing rapport, naturalness, and conversational flow. In an application like the learning companion, such a socially responsive dialogue system may improve learning and motivation. However, it is not clear how to produce entrainment in an automatic dialogue system in ways that produce the effects seen in human-human dialogue. In this paper, we take the first steps towards implementing a spoken dialogue system which can entrain. We propose three methods of pitch adaptation based on analysis of human entrainment, and design and implement a system which can manipulate the pitch of text-to-speech output adaptively. We find a clear relationship between perceptions of rapport and different forms of pitch adaptations. Certain adaptations are perceived as significantly more natural and rapport-like. Ultimately, adapting by shifting the pitch contour of the text-to-speech output by the mean pitch of the user results in the highest reported measures of rapport and naturalness.

Today, people are just as likely to have a business meeting remotely as they are face-to-face. Individuals obtain college degrees remotely and sick patients can visit the doctor from home. Especially important in light of this popularity, remote settings are posing communication challenges that are not present in face-to-face settings. Visual cues such as facial expressions and body language are either degraded or nonexistent. In this paper, we are interested in how remote settings affect spoken dialogue when compared to face-to-face settings. We focus on entrainment, a phenomenon of conversation where individuals adapt to each other during the interaction. Specifically, we investigate acoustic-prosodic entrainment, where individuals become more similar in their pitch, loudness, or speaking rate. We explore three different measures of acoustic-prosodic entrainment, comparing remote settings to face-to-face settings on a turn-by-turn basis. Our results indicate that the two settings do differ for different forms of entrainment, suggesting that the presence or absence of visual cues such as facial expressions and body language has an impact on the degree of entrainment.

We describe an experiment that investigates whether sub-utterance prosodic features can be used to detect uncertainty at the wordlevel. That is, given an utterance that is classified as uncertain, we want to determine which word or phrase the speaker is uncertain about. We have a corpus of utterances spoken under varying degrees of certainty. Using combinations of sub-utterance prosodic features we train models to predict the level of certainty of an utterance. On a set of utterances that were perceived to be uncertain, we compare the predictions of our models for two candidate target word segmentations: (a) one with the actual word causing uncertainty as the proposed target word, and (b) one with a control word as the proposed target word. Our best model correctly identifies the word causing the uncertainty rather than the control word 91% of the time.

Abstract We explore the relationship between question answering and constraint relaxation in spoken dialogue systems and develop dialogue strategies for selecting and presenting information succinctly. In particular, we describe methods for dealing with the results of database queries in information-seeking dialogues. Our goal is to structure the dialogue in such a way that the user is neither overwhelmed with information nor left uncertain as to how to refine the query further. We present two sets of evaluation results for a restaurant selection task: one is a system performance evaluation experiment involving twenty subjects, the other is an experimental evaluation of the use of suggestions involving sixteen subjects.

We present a project aimed at understanding the acoustic and prosodic correlates of confidence and uncertainty in spoken lan-guage. We elicited speech produced under varying levels of cer-tainty and performed perceptual and statistical analyses on the speech data to determine which prosodic features (e.g., pitch, energy, timing) are associated with a speaker’s level of certainty and where these prosodic manifestations occur relative to the lo-cation of the word or phrase that the speaker is confident or un-certain about. Our findings suggest that prosodic manifestations of confidence and uncertainty occur both in the local region that causes the uncertainty as well as in its surrounding context. Index Terms: prosody, human speech perception, emotion de-tection, paralinguistic cues

In this paper we present a framework for managing information presentation in spoken dialogue systems. We describe a content optimization module that makes use of ontological relationships in information-seeking dialogues in order to organize knowledge base items and perform adjustments such as relaxing or tightening user constraints. We present the results of an experimental evaluation comparing two response strategies: (a) one that uses the content optimization module to offer suggestions and (b) one that gives no suggestions. The results indicate that giving such suggestions is preferred when a user query matches either no items or many items in the knowledge base, and may also lead to more efficient dialogues. Index Terms: spoken dialogue systems, content management 1.

Spoken dialogue interfaces, mostly command-and-control, become more visible in applications where attention needs to be shared with other tasks, such as driving a car. The deployment of the simple dialog systems, instead of more sophisticated ones, is partly because the computing platforms used for such tasks have been less powerful and partly because certain issues from these cognitively challenging tasks have not been well addressed even in the most advanced dialog systems. This paper reports the progress of our research effort in developing a robust, wide-coverage, and cognitive load-sensitive spoken dialog interface called CHAT: Conversational Helper for Automotive Tasks. Our research in the past few years has led to promising results, including high task completion rate, dialog efficiency, and improved user experience. Index Terms: dialog systems, cognitive load, robustness

Contextualizing learning in an intelligent tutoring system is difficult for many reasons. Goals such as presenting material in an understandable manner, minimizing confusion and frustration, and helping the student reason about their actions all need to be balanced. Previous research has shown reflective discussions (with human tutors) occurring after problem-solving to be effective in helping students reason about their own actions (S. Katz et. al, 2003). However, leading a reflective discussion makes it difficult to present information in an understandable manner, and without contextualization, it is easy to create student confusion and frustration. This raises the question: how can intelligent tutoring systems effectively contextualize learning in a reflective discussion? In this paper, we describe the tutorial architecture of SCoT, a spoken conversational tutor that uses flexible, adaptive planning and multi-modal task modeling to support the contextualization of learning in reflective dialogues.