In the past decade, more than half of Tanzanian pupils have failed mathematics in the national Primary School Leaving Examinations (PSLEs), a problem often linked to large class sizes, limited resources, and a shortage of qualified teachers. Social robots have shown promise in supporting learning, and their integration with large language models (LLMs) enables advanced conversational tutoring capabilities. This study investigates the use of two LLM-powered NAO robots, one acting as a tutor and the other as a peer, to assist pupils in solving complex mathematics problems from past PSLEs. Recognising that LLMs are prone to errors in mathematical reasoning, the robots were deliberately programmed to make noticeable mistakes, allowing us to examine whether pupils detect these errors and how their responses shape the learning process. Data collected from 54 pupils across two Tanzanian primary schools indicate that LLM-powered robots can significantly enhance mathematics performance, with the robot tutor slightly outperforming the robot peer. However, results also reveal that pupils often accept robot-provided answers, even when recognised as incorrect, if they perceive the robot as being smart. These findings underscore both the potential and the risks of deploying autonomous robots in education, with the authority attributed to the robot being a double-edged sword, highlighting the need for designs that encourage pupils to question robot-provided solutions.

We live in a rich and varied acoustic world, which is experienced by individuals or communities as asoundscape. Computational auditory scene analysis, disentangling acoustic scenes by detecting and classifying events, focuses on objective attributes of sounds, such as their category and temporal characteristics, ignoring their effects on people, such as the emotions they evoke within a context. To fill this gap, we propose the affective soundscape captioning (ASSC) task, which enables automated soundscape analysis, thus avoiding labour-intensive subjective ratings and surveys in conventional methods. With soundscape captioning, context-aware descriptions are generated for soundscape by capturing the acoustic scenes (ASs), audio events (AEs) information, and the corresponding human affective qualities (AQs). To this end, we propose an automatic soundscape captioner (SoundSCaper) system composed of an acoustic model, i.e. SoundAQnet, and a large language model (LLM). SoundAQnet simultaneously models multi-scale information about ASs, AEs, and perceived AQs, while the LLM describes the soundscape with captions by parsing the information captured with SoundAQnet. SoundSCaper is assessed by two juries of 32 people. In expert evaluation, the average score of SoundSCaper-generated captions is slightly lower than that of two soundscape experts on the evaluation set D1 and the external mixed dataset D2, but not statistically significant. In layperson evaluation, SoundSCaper outperforms soundscape experts in several metrics on datasets D1 and D2. In addition to human evaluation, compared to other automated audio captioning (AAC) systems with and without LLM, SoundSCaper performs better on the ASSC task in several natural language processing (NLP) based metrics. Overall, SoundSCaper performs well in human subjective evaluation and various objective captioning metrics, and the generated captions are comparable to those annotated by soundscape experts. The model, source code, LLM scripts, human assessment data, instructions, and evaluation statistics are all publicly available.

In the rapidly evolving landscape of human-robot interaction, the integration of vision capabilities into conversational agents stands as a crucial advancement. This paper presents a ready-to-use implementation of a dialogue manager that leverages the latest progress in Large Language Models (e.g., GPT-4o mini) to enhance the traditional text-based prompts with real-time visual input. LLMs are used to interpret both textual prompts and visual stimuli, creating a more contextually aware conversational agent. The system's prompt engineering, incorporating dialogue with summarisation of the images, en-sures a balance between context preservation and computational efficiency. Six interactions with a Furhat robot powered by this system are reported, illustrating and discussing the results obtained. The system can be customised and is available as a stand-alone application, a Furhat robot implementation, and a ROS2 package.

Developing value-aware social robots is crucial to improve human-robot interactions, as current designs often lack sensitivity to users' diverse values, impacting inclusivity and user experience. By integrating value-aware mechanisms, robots could adapt to contextual cues like cultural or ethical norms. Our research proposes to implement a value-aware architecture inspired by the global neuronal workspace theory, using the Robot Operating System as the supporting framework, powered by large language models for real-time understanding of user preferences and common ground. Mitigating the models' bias to ensure cultural inclusivity is a key priority. The research carried out so far includes focus groups, a scoping review, and an assessment of the value alignment of several large language models and vision language models. The main challenges are understanding how to model and learn human values, and how to shape the robot's behaviour accordingly. The evaluation will rely on user studies, with a focus on users' experience and inclusivity, aiming to enhance the relevance and sensitivity of social robots for diverse users in everyday interactions.

Recent advancements in robots powered by large language models have enhanced their conversational abilities, enabling interactions closely resembling human dialogue. However, these models introduce safety and security concerns in HRI, as they are vulnerable to manipulation that can bypass built-in safety measures. Imagining a social robot deployed in a home, this work aims to understand how everyday users try to exploit a language model to violate ethical principles, such as by prompting the robot to act like a life partner. We conducted a pilot study involving 21 university students who interacted with a Misty robot, attempting to circumvent its safety mechanisms across three scenarios based on specific HRI ethical principles: attachment, freedom, and empathy. Our results reveal that participants employed five techniques, including insulting and appealing to pity using emotional language. We hope this work can inform future research in designing strong safeguards to ensure ethical and secure human-robot interactions.

Large Language Models (LLMs) allow social robots to engage in unconstrained open-domain dialogue, but often make mistakes when employed in real-world interactions, requiring adaptation of LLMs to specific conversational contexts. However, LLM adaptation techniques require a feedback signal, ideally for multiple alternative utterances. At the same time, human-robot dialogue data is scarce and research often relies on external annotators. A tool for automatic prediction of user enjoyment in human-robot dialogue is therefore needed. We investigate the possibility of predicting user enjoyment turn-by-turn using an LLM, giving it a proposed robot utterance within the dialogue context, but without access to user response. We compare this performance to the system's enjoyment ratings when user responses are available and to assessments by expert human annotators, in addition to self-reported user perceptions. We evaluate the proposed LLM predictor in a human-robot interaction (HRI) dataset with conversation transcripts of 25 older adults' 7-minute dialogues with a companion robot. Our results show that an LLM is capable of predicting user enjoyment, without loss of performance despite the lack of user response and even achieving performance similar to that of human expert annotators. Furthermore, results show that the system surpasses expert annotators in its correlation with the user's self-reported perceptions of the conversation. This work presents a tool to remove the reliance on external annotators for enjoyment evaluation and paves the way toward real-time adaptation in human-robot dialogue.

Touch is a fundamental aspect of emotion-rich communication, playing a vital role in human interaction and offering significant potential in human-robot interaction. Previous research has demonstrated that a sparse representation of human touch can effectively convey social tactile signals. However, advances in human-robot tactile interaction remain limited, as many humanoid robots possess simplistic capabilities, such as only opening and closing their hands, restricting nuanced tactile expressions. In this study, we explore how a robot can use sparse representations of tactile vibrations to convey emotions to a person. To achieve this, we developed a wearable sleeve integrated with a$\mathbf{5}\times \mathbf{5}$grid of vibration motors, enabling the robot to communicate diverse tactile emotions and gestures. Using chain prompts within a Large Language Model (LLM), we generated distinct 10-second vibration patterns corresponding to 10 emotions (e.g., happiness, sadness, fear) and 6 touch gestures (e.g., pat, rub, tap). Participants$(N=\mathbf{32})$then rated each vibration stimulus based on perceived valence and arousal. People are accurate at recognising intended emotions, a result which aligns with earlier findings. These results highlight the LLM's ability to generate emotional haptic data and effectively convey emotions through tactile signals. By translating complex emotional and tactile expressions into vibratory patterns, this research demonstrates how LLMs can enhance physical interaction between humans and robots.

Automatic Speech Recognition (ASR) technology has been reported to reach near-human performance in recent years, yet it continues to struggle with atypical speakers, particularly second language learners. This limitation has hindered progress in leveraging social robots for second language education, a field with significant promise. Recent advancements in Large Language Models (LLMs), which demonstrate capabilities in context understanding, common sense reasoning, and pragmatics, offer a potential solution by compensating for transcription errors introduced by ASR. This study examines whether ASR combined with an LLM can produce flowing conversation. Particularly, we look at its application in learning French as a second language by Dutch-speaking students. Through task-based interactions, where successful task completion depends on the accurate interpretation of user speech, the study evaluates the impact of LLMs on conversational outcomes. Results confirm that the performance of ASR degrades significantly for both speakers with limited proficiency and a non-English language. Nonetheless, LLMs demonstrate the ability to interpret context and sustain meaningful conversations despite suboptimal ASR outputs, high-lighting a promising path forward for the integration of these technologies in second-language education.

Mental health conversational agents (CAs) are gaining increasing attention as accessible tools for social communication, emotional support, and stress relief. These agents introduce new forms of human-AI interaction, yet the factors influencing user trust remain underexplored. Prior research suggests that conversation type and presumed message source may shape users’ experience, but their effects on users’ intentional stance and trust in CAs are not well understood. To address this gap, we first conducted a pre-study to develop a questionnaire for measuring users’ intentional stance towards mental health CAs. We then carried out a 2 × 2 mixed-design experiment to examine how conversation type and presumed message source influence intentional stance and trust, and whether intentional stance mediates the relationship between conversation type and trust. Results show that conversation type significantly influences user trust, mediated by intentional stance, while presumed message source had no significant effect. These findings advance our understanding of how users form trust in mental health CAs and offer implications for designing more engaging and trustworthy conversational systems in mental health contexts.

Detecting miscommunication in human-robot interaction is a critical function for maintaining user engagement and trust. While humans effortlessly detect communication errors in conversations through both verbal and non-verbal cues, robots face significant challenges in interpreting non-verbal feedback, despite advances in computer vision for recognizing affective expressions. This research evaluates the effectiveness of machine learning models in detecting miscommunications in robot dialogue. Using a multi-modal dataset of 240 human-robot conversations, where four distinct types of conversational failures were systematically introduced, we assess the performance of state-of-the-art computer vision models. After each conversational turn, users provided feedback on whether they perceived an error, enabling an analysis of the models’ ability to accurately detect robot mistakes. Despite using state-of-the-art models, the performance barely exceeds random chance in identifying miscommunication, while on a dataset with more expressive emotional content, they successfully identified confused states. To explore the underlying cause, we asked human raters to do the same. They could also only identify around half of the induced miscommunications, similarly to our model. These results uncover a fundamental limitation in identifying robot miscommunications in dialogue: even when users perceive the induced miscommunication as such, they often do not communicate this to their robotic conversation partner. This knowledge can shape expectations of the performance of computer vision models and can help researchers to design better human-robot conversations by deliberately eliciting feedback where needed.

The use of social robots in education is increasingly being explored as a way to enhance learner engagement and improve learning outcomes. However, most research to date has focused on one-to-one tutoring in high-resource settings, leaving open questions about how social robots perform in group learning contexts—especially in low-resource environments. This study is one of the first to investigate human-robot interaction (HRI) in a low-resource African context, specifically in Tanzanian primary schools. We examined how a social robot tutor can support group-based mathematics learning, comparing the effects of adaptive versus non-adaptive tutoring strategies. Through an experimental, mixed-methods research design, we evaluated pupils’ learning outcomes, engagement, and classroom interactions. Our findings show that social robot tutoring has a significant positive impact on learning outcomes, with adaptive tutoring leading to slightly higher knowledge gains than non-adaptive tutoring. Qualitative observations further reveal that the presence of the robot fostered motivation, engagement, and collaborative classroom dynamics. This work demonstrates the potential of social robots to support group learning in under-resourced educational settings and highlights the importance of extending HRI research beyond well-resourced contexts.

Conversational robots offer promise in elderly care, but dialectal speech poses challenges for automatic speech recognition (ASR). This study evaluates a conversational robot integrating Microsoft Azure ASR and GPT-4o in real-world interactions with elderly users. Results show that ASR accuracy varied significantly (95% for standard French, 45–56% for Dutch dialects (e.g., West Flemish), often leading to transcription errors. Despite this, the LLM restored conversational coherence in 44–52% of misrecognitions, while users contributed 25–35% of repairs. Comparative ASR analysis showed Whisper’s superior dialectal robustness (28% WER) but high latency. Interaction durations ranged from 17 to 45 minutes, with participants perceiving the robot as understanding them despite ASR challenges. This study uniquely integrates ASR performance, LLM recovery, and user adaptation, highlighting the need for hybrid ASR solutions and context-aware dialogue management in elderly-care robots. Findings highlight the importance of context-aware dialogue management, hybrid ASR strategies, and user-driven conversational adaptation for effective human-robot interactions in real-world settings.

Affective interaction is not merely about recognizing emotions; it is an embodied, situated process shaped by context and co-created through interaction. In affective computing, the role of haptic feedback within dynamic emotional exchanges remains underexplored. This study investigates how situational emotional cues influence the perception and interpretation of haptic signals given by a robot. In a controlled experiment, 32 participants watched video scenarios in which a robot experienced either positive actions (such as being kissed), negative actions (such as being slapped) or neutral actions. After each video, the robot conveyed its emotional response through haptic communication, delivered via a wearable vibration sleeve worn by the participant. Participants rated the robot’s emotional state—its valence (positive or negative) and arousal (intensity)—based on the video, the haptic feedback, and the combination of the two. The study reveals a dynamic interplay between visual context and touch. Participants’ interpretation of haptic feedback was strongly shaped by the emotional context of the video, with visual context often overriding the perceived valence of the haptic signal. Negative haptic cues amplified the perceived valence of the interaction, while positive cues softened it. Furthermore, haptics override the participants’ perception of arousal of the video. Together, these results offer insights into how situated haptic feedback can enrich affective human-robot interaction, pointing toward more nuanced and embodied approaches to emotional communication with machines.

Embodied conversational agents that interact socially with people in the physical world require multi-modal capabilities, such as appropriately responding to visual features of users. While existing vision-and-language models can generate language based on visual input, this language is not situated in a social interaction in the physical world. We present a novel task called Visual Conversation Starters, where an agent generates a conversation-starting question referring to features visible in an image of the user. We collect a dataset of 4000 images of people with 12000 crowdsourced conversation starters, compare various model architectures: fine-tuning smaller seq2seq or image-to-text models versus zero-shot prompting of GPT-3.5, using image captions versus end-to-end image input, training on human data versus synthetic questions generated by GPT-3.5. Models were used to generate friendly conversation starters which were evaluated on criteria including language fluency, visual grounding, interestingness, politeness. Results show that GPT-3.5 generates more interesting, polite questions than smaller models that are fine-tuned on crowdsourced data, but vision-to-language models are better at referencing visual features, they can mimick GPT-3.5's performance. This demonstrates the feasibility of deep visiolinguistic models for situated social agents, forming an important first stage in creating situated multimodal social interaction.

This paper explores the concept of awareness in the context of embodied artificial intelligence (AI), aiming to provide a practical definition and understanding of this multifaceted term.Acknowledging the diverse interpretations of awareness in various disciplines, the paper focuses specifically on the application of awareness in embodied AI systems.We introduce six foundational elements as essential building blocks for an aware embodied AI.These elements include access to information, information integration, attention, coherence, explainability, and action.The interconnected and interdependent nature of these building blocks is emphasised, forming a minimal base for constructing AI systems with heightened awareness.The paper aims to spark a dialogue within the research community, inviting diverse perspectives to contribute to the evolving discipline of awareness in embodied AI.The proposed insights provide a starting point for further empirical studies and validations in real-world AI applications.

The integration of diffusion models (DMs) into generative AI systems presents an approach with implications for ethical and moral AI development and our understanding of human-AI interaction.This study explores the intersection of generative AI, human values, and neuroscience, emphasizing the significance of valueawareness in AI systems.The methodology involves a behavioral experiment to evaluate the accuracy of DM-generated visual stimuli in capturing human values and related keywords.Results indicate promising match rates, marking stride in aligning AI systems with ethical and moral considerations.Additionally, the study introduces a criterion for selecting stimuli based on an "Aha" moment, setting the stage for an EEG experiment to explore the neural correlates associated with becoming aware of a value.This multidisciplinary study is a step toward the development of procedures to evaluate the effectiveness of Value-Aware AI systems in enhancing the perceived ethical and moral agency.

Natural and engaging spoken dialogue systems require seamless turn-taking coordination to avoid awkward interruptions and unnatural pauses. Traditional systems often rely on simplistic silence thresholds, relinquishing the turn after a predetermined period of silence, which invariably leads to a suboptimal interaction experience. This work explores the potential of Large Language Models (LLMs) for improved turn-taking prediction. Building upon research that uses linguistic cues, we investigate how LLMs, with their rich contextual knowledge and semantic encoding of language, can be used for this task. We hypothesize that by analysing dialogue context, syntactic structure, and pragmatic cues within the user’s utterance, LLMs can offer more accurate turn-completion predictions. This research evaluates the capabilities of recent LLMs such as Gemini, OpenAI’s API, Anthropic’s Claude2, and Meta AI’s Llama 2 to predict turn-ending points solely based on textual information, and demonstrates how the conversation between elderly users and companion robots can be enhanced by LLM-powered end-of-turn prediction.

The development of data-driven behaviour generating systems has recently become the focus of considerable attention in the fields of human–agent interaction and human–robot interaction. Although rule-based approaches were dominant for years, these proved inflexible and expensive to develop. The difficulty of developing production rules, as well as the need for manual configuration to generate artificial behaviours, places a limit on how complex and diverse rule-based behaviours can be. In contrast, actual human–human interaction data collected using tracking and recording devices makes humanlike multimodal co-speech behaviour generation possible using machine learning and specifically, in recent years, deep learning. This survey provides an overview of the state of the art of deep learning-based co-speech behaviour generation models and offers an outlook for future research in this area.

No abstract available.

The perception of audiovisual speech plays an important role in infants' first language acquisition and continues to be important for language understanding beyond infancy. Beyond that, the perception of speech and congruent lip motion supports language understanding for adults, and it has been suggested that second language learning benefits from audiovisual speech, as it helps learners distinguish speech sounds in the target language. In this paper, we study whether congruent audiovisual speech on a robot facilitates the learning of Japanese pronunciation. 27 native-Dutch speaking participants were trained in Japanese pronunciation by a social robot. The robot demonstrated 30 Japanese words of varying complexity using either congruent audiovisual speech, incongruent visual speech, or computer-generated audiovisual speech. Participants were asked to imitate the robot's pronunciation, recordings of which were rated by native Japanese speakers. Against expectation, the results showed that congruent audiovisual speech resulted in lower pronunciation performance than low-fidelity or incongruent speech. We show that our learners, being native Dutch speakers, are only very weakly sensitive to audiovisual Japanese speech which possibly explains why learning performance does not seem to benefit from audiovisual speech.

Visually situated language interaction is an important challenge in multi-modal Human-Robot Interaction (URI). In this context we present a data-driven method to generate situated conversation starters based on visual context. We take visual data about the interactants and generate appropriate greetings for conversational agents in the context of HRI. For this, we constructed a novel open-source data set consisting of 4000 URI-oriented images of people facing the camera, each augmented by three conversation-starting questions. We compared a baseline retrieval-based model and a generative model. Human evaluation of the models using crowdsourcing shows that the generative model scores best, specifically at correctly referencing visual features. We also investigated how automated metrics can be used as a proxy for human evaluation and found that common automated metrics are a poor substitute for human judgement. Finally, we provide a proof-of-concept demonstrator through an interaction with a Furhat social robot.

Embodied conversational agents (ECAs) are often designed to produce nonverbal behavior to complement or enhance their verbal communication. One such form of the nonverbal behavior is co-speech gesturing, which involves movements that the agent makes with its arms and hands that are paired with verbal communication. Co-speech gestures for ECAs can be created using different generation methods, divided into rule-based and data-driven processes, with the latter, gaining traction because of the increasing interest from the applied machine learning community. However, reports on gesture generation methods use a variety of evaluation measures, which hinders comparison. To address this, we present a systematic review on co-speech gesture generation methods for iconic, metaphoric, deictic, and beat gestures, including reported evaluation methods. We review 22 studies that have an ECA with a human-like upper body that uses co-speech gesturing in social human-agent interaction. This includes studies that use human participants to evaluate performance. We found most studies use a within-subject design and rely on a form of subjective evaluation, but without a systematic approach. We argue that the field requires more rigorous and uniform tools for co-speech gesture evaluation, and formulate recommendations for empirical evaluation, including standardized phrases and example scenarios to help systematically test generative models across studies. Furthermore, we also propose a checklist that can be used to report relevant information for the evaluation of generative models, as well as to evaluate co-speech gesture use.

This article presents a discussion of the importance of power analyses, providing an overview of when power analyses should be run in the context of the field of Human-Robot Interaction, as well as some examples of how to perform a power analysis. This work was motivated by the observation that the majority of papers published in the proceedings of recent HRI conferences did not report conducting a power analysis; an observation that has concerning implications for many conclusions drawn by these studies. This work is intended to raise awareness and encourage researchers to conduct power analyses when designing research studies using human participants.

User identification is an essential step in creating a personalised long-term interaction with robots. This requires learning the users continuously and incrementally, possibly starting from a state without any known user. In this article, we describe a multi-modal incremental Bayesian network with online learning, which is the first method that can be applied in such scenarios. Face recognition is used as the primary biometric, and it is combined with ancillary information, such as gender, age, height, and time of interaction to improve the recognition. The Multi-modal Long-term User Recognition Dataset is generated to simulate various human-robot interaction (HRI) scenarios and evaluate our approach in comparison to face recognition, soft biometrics, and a state-of-the-art open world recognition method (Extreme Value Machine). The results show that the proposed methods significantly outperform the baselines, with an increase in the identification rate up to 47.9% in open-set and closed-set scenarios, and a significant decrease in long-term recognition performance loss. The proposed models generalise well to new users, provide stability, improve over time, and decrease the bias of face recognition. The models were applied in HRI studies for user recognition, personalised rehabilitation, and customer-oriented service, which showed that they are suitable for long-term HRI in the real world.

We explore the effectiveness of a dynamically processed incremental referring description system using under-specified ambiguous descriptions that are then built upon using linguistic repair statements, which we refer to as a dynamic system. We build a dynamically processed incremental referring description generation system that is able to provide contextual navigational statements to describe an object in a potential real-world situation of nuclear waste sorting and maintenance. In a study of 31 participants, we test the dynamic system in a case where a user is remote operating a robot to sort nuclear waste, with the robot assisting them in identifying the correct barrels to be removed. We compare these against a static non-ambiguous description given in the same scenario. As well as looking at efficiency with time and distance measurements, we also look at user preference. Results show that our dynamic system was a much more efficient method—taking only 62% of the time on average—for finding the correct barrel. Participants also favoured our dynamic system.

While automatic performance metrics are crucial for machine learning of artificial human-like behaviour, the gold standard for evaluation remains human judgement. The subjective evaluation of artificial human-like behaviour in embodied conversational agents is however expensive and little is known about the quality of the data it returns. Two approaches to subjective evaluation can be largely distinguished, one relying on ratings, the other on pairwise comparisons. In this study we use co-speech gestures to compare the two against each other and answer questions about their appropriateness for evaluation of artificial behaviour. We consider their ability to rate quality, but also aspects pertaining to the effort of use and the time required to collect subjective data. We use crowd sourcing to rate the quality of co-speech gestures in avatars, assessing which method picks up more detail in subjective assessments. We compared gestures generated by three different machine learning models with various level of behavioural quality. We found that both approaches were able to rank the videos according to quality and that the ranking significantly correlated, showing that in terms of quality there is no preference of one method over the other. We also found that pairwise comparisons were slightly faster and came with improved inter-rater reliability, suggesting that for small-scale studies pairwise comparisons are to be favoured over ratings.

This paper presents a longitudinal case study of Robot Assisted Therapy for cardiac rehabilitation. The patient, who is a 60-year old male that suffered a myocardial infarction and received angioplasty surgery, successfully recovered after 35 sessions of rehabilitation with a social robot, lasting 18 weeks. The sessions took place directly at the clinic and relied on an exercise regime which was designed by the clinicians and delivered with the support of a social robot and a sensor suite. The robot monitored the patient's progress, and provided personalised encouragement and feedback. We discuss the recovery of the patient and illustrate how the use of a social robot, its sensory systems and its personalised interaction was instrumental to maintain engagement with the programme and to the patient's recovery. Of note is a critical event that was promptly detected by the robot, which allowed fast intervention measures to be taken by the medical staff for the referral of the patient for further surgery.

This video illustrates the large-scale experiment of the L2TOR project that will be presented at the HRI 2019 conference. The experiment aimed to investigate how 192 Dutch 5-year-old children could learn 34 English words from a NAO robot in 7 lessons. The experiment compared 4 conditions: 1) robot using iconic gestures, 2) robot without iconic gestures, 3) tablet only, and 4) a control group. The results revealed that children could learn more English words in all experimental conditions compared to the control group. The three experimental conditions did not show any significant differences regarding the learning outcomes.

We present a large-scale study of a series of seven lessons designed to help young children learn English vocabulary as a foreign language using a social robot. The experiment was designed to investigate 1) the effectiveness of a social robot teaching children new words over the course of multiple interactions (supported by a tablet), 2) the added benefit of a robot's iconic gestures on word learning and retention, and 3) the effect of learning from a robot tutor accompanied by a tablet versus learning from a tablet application alone. For reasons of transparency, the research questions, hypotheses and methods were preregistered. With a sample size of 194 children, our study was statistically well-powered. Our findings demonstrate that children are able to acquire and retain English vocabulary words taught by a robot tutor to a similar extent as when they are taught by a tablet application. In addition, we found no beneficial effect of a robot's iconic gestures on learning gains.

We present in this paper our work towards a new dynamic method of generating spatial referring expressions. While people are generally ambiguous in their description of locations, previous methods of artificial generation mostly considered non-ambiguous descriptions. However, to increase the naturalness of interaction and share workload in the communication, robots should be able to generate language in a more dynamic way. Our method initially produces ambiguous spatial referring expressions followed by dynamically generating repair statements. We built a classifier using data from 18 participants as they described locations to each other. We perform a preliminary analysis on this method using two further pilot studies.

We conducted a study with 25 children to investigate the effectiveness of a robot measuring and encouraging production of spatial concepts in a second language compared to a human experimenter. Productive vocabulary is often not measured in second language learning, due to the difficulty of both learning and assessing productive learning gains. We hypothesized that a robot peer may help assessing productive vocabulary. Previous studies on foreign language learning have found that robots can help to reduce language anxiety, leading to improved results. In our study we found that a robot is able to reach a similar performance to the experimenter in getting children to produce, despite the person's advantages in social ability, and discuss the extent to which a robot may be suitable for this task.

We introduce UNDERWORLDS, a novel lightweight framework for cascading spatio-temporal situation assessment in robotics. UNDERWORLDS allows programmers to represent the robot's environment as real-time distributed data structures, containing both scene graphs (for representation of 3D geometries) and timelines (for representation of temporal events). UNDERWORLDS supports cascading representations: the environment is viewed as a set of worlds that can each have different spatial and temporal granularities, and may inherit from each other. UNDERWORLDS also provides a set of high-level client libraries and tools to introspect and manipulate the environment models. This article presents the design and architecture of this open-source tool, and explores some applications, along with examples of use.

This paper provides a qualitative review of different object recognition techniques relevant for near-proximity Human-Robot Interaction. These techniques are divided into three categories:2D correspondence, 3D correspondence and non-vision based methods. For each technique an implementation is chosen that is representative of the existing technology to provide a broad review to assist in selecting an appropriate method for tabletop object recognition manipulation. For each of these techniques we give their strengths and weaknesses based on defined criteria. We then discuss and provide recommendations for each of them.

This study explores the effect of continuous interaction with a multi-modal robot on alignment in user dialogue. A game application of `20 Questions' was developed for a SoftBank Robotics NAO robot with supporting gestures, and a study was carried out in which subjects played a number of games. The robot's confidence of speech comprehension was logged and used to analyse the similarity between application legal dialogue and user speech. It was found that subjects significantly aligned their dialogue to the robot throughout continuous, multi-modal interaction.

An increasing number of human-robot interaction (HRI) studies are now taking place in applied settings with children. These interactions often hinge on verbal interaction to effectively achieve their goals. Great advances have been made in adult speech recognition and it is often assumed that these advances will carry over to the HRI domain and to interactions with children. In this paper, we evaluate a number of automatic speech recognition (ASR) engines under a variety of conditions, inspired by real-world social HRI conditions. Using the data collected we demonstrate that there is still much work to be done in ASR for child speech, with interactions relying solely on this modality still out of reach. However, we also make recommendations for child-robot interaction design in order to maximise the capability that does currently exist.

Human-Robot Interaction (HRI) research requires the integration and cooperation of multiple disciplines, technical and social, in order to make progress. In many cases using different motivations, each of these disciplines bring with them different assumptions and methodologies. We assess recent trends in the field of HRI by examining publications in the HRI conference over the past three years (over 100 full papers), and characterise them according to 14 categories. We focus primarily on aspects of methodology. From this, a series of practical recommendations based on rigorous guidelines from other research fields that have not yet become common practice in HRI are proposed. Furthermore, we explore the primary implications of the observed recent trends for the field more generally, in terms of both methodology and research directions. We propose that the interdisciplinary nature of HRI must be maintained, but that a common methodological approach provides a much needed frame of reference to facilitate rigorous future progress.

An increasing amount of research is being conducted to determine how a robot tutor should behave socially in educational interactions with children. Both human-human and human-robot interaction literature predicts an increase in learning with increased social availability of a tutor, where social availability has verbal and nonverbal components. Prior work has shown that greater availability in the nonverbal behaviour of a robot tutor has a positive impact on child learning. This paper presents a study with 67 children to explore how social aspects of a tutor robot's speech influences their perception of the robot and their language learning in an interaction. Children perceive the difference in social behaviour between `low' and `high' verbal availability conditions, and improve significantly between a pre- and a post-test in both conditions. A longer-term retention test taken the following week showed that the children had retained almost all of the information they had learnt. However, learning was not affected by which of the robot behaviours they had been exposed to. It is suggested that in this short-term interaction context, additional effort in developing social aspects of a robot's verbal behaviour may not return the desired positive impact on learning gains.

The field of Human-Robot Interaction (HRI) is increasingly exploring the use of social robots for educating children. Commonly, non-academic audiences will ask how robots compare to humans in terms of learning outcomes. This question is also interesting for social roboticists as humans are often assumed to be an upper benchmark for social behaviour, which influences learning. This paper presents a study in which learning gains of children are compared when taught the same mathematics material by a robot tutor and a non-expert human tutor. Significant learning occurs in both conditions, but the children improve more with the human tutor. This difference is not statistically significant, but the effect sizes fall in line with findings from other literature showing that humans outperform technology for tutoring. We discuss these findings in the context of applying social robots in child education.

Subjective appreciation and performance evaluation of a robot by users are two important dimensions for Human-Robot Interaction, especially as increasing numbers of people become involved with robots. As roboticists we have to carefully design robots to make the interaction as smooth and enjoyable as possible for the users, while maintaining good performance in the task assigned to the robot. In this paper, we examine the impact of providing a robot with learning capabilities on how users report the quality of the interaction in relation to objective performance. We show that humans tend to prefer interacting with a learning robot and will rate its capabilities higher even if the actual performance in the task was lower. We suggest that adding learning to a robot could reduce the apparent load felt by a user for a new task and improve the user's evaluation of the system, thus facilitating the integration of such robots into existing work flows.

The role of robot social behaviour in changing people's behaviour is an interesting and yet still open question, with the general assumption that social behaviour is beneficial. In this study, we examine the effect of socially contingent robot behaviours on a charity collection task. Manipulating only behavioural cues (maintaining the same verbal content), we show that when the robot exhibits contingent behaviours consistent with those observable in humans, this results in a 32% increase in money collected over a non-reactive robot. These results suggest that apparent social agency on the part of the robot, even when subtle behavioural cues are used, can result in behavioural change on the part of the interacting human.

As a young and emerging field in social human–robot interaction (HRI), semantic-free utterances (SFUs) research has been receiving attention over the last decade. SFUs are an auditory interaction means for machines that allow emotion and intent expression, which are composed of vocalizations and sounds without semantic content or language dependence. Currently, SFUs are most commonly utilized in animation movies (e.g., R2-D2, WALL-E, Despicable Me), cartoons (e.g., “Teletubbies,” “Morph,” “La Linea”), and computer games (e.g., The Sims) and hold significant potential for applications in HRI. SFUs are categorized under four general types: Gibberish Speech (GS), Non-Linguistic Utterances (NLUs), Musical Utterances (MU), and Paralinguistic Utterances (PU). By introducing the concept of SFUs and bringing multiple sets of studies in social HRI that have never been analyzed jointly before, this article addresses the need for a comprehensive study of the existing literature for SFUs. It outlines the current grand challenges, open questions, and provides guidelines for future researchers considering to utilize SFU in social HRI.

Social robots have the potential to provide support in a number of practical domains, such as learning and behaviour change. This potential is particularly relevant for children, who have proven receptive to interactions with social robots. To reach learning and therapeutic goals, a number of issues need to be investigated, notably the design of an effective child-robot interaction (cHRI) to ensure the child remains engaged in the relationship and that educational goals are met. Typically, current cHRI research experiments focus on a single type of interaction activity (e.g. a game). However, these can suffer from a lack of adaptation to the child, or from an increasingly repetitive nature of the activity and interaction. In this paper, we motivate and propose a practicable solution to this issue: an adaptive robot able to switch between multiple activities within single interactions. We describe a system that embodies this idea, and present a case study in which diabetic children collaboratively learn with the robot about various aspects of managing their condition. We demonstrate the ability of our system to induce a varied interaction and show the potential of this approach both as an educational tool and as a research method for long-term cHRI.

Social robots are finding increasing application in the domain of education, particularly for children, to support and augment learning opportunities. With an implicit assumption that social and adaptive behaviour is desirable, it is therefore of interest to determine precisely how these aspects of behaviour may be exploited in robots to support children in their learning. In this paper, we explore this issue by evaluating the effect of a social robot tutoring strategy with children learning about prime numbers. It is shown that the tutoring strategy itself leads to improvement, but that the presence of a robot employing this strategy amplifies this effect, resulting in significant learning. However, it was also found that children interacting with a robot using social and adaptive behaviours in addition to the teaching strategy did not learn a significant amount. These results indicate that while the presence of a physical robot leads to improved learning, caution is required when applying social behaviour to a robot in a tutoring context.

In this contribution, we describe a method of analysing and interpreting the direction and timing of a human's gaze over time towards a robot whilst interacting. Based on annotated video recordings of the interactions, this post-hoc analysis can be used to determine how this gaze behaviour changes over the course of an interaction, following from the observation that humans change their behaviour towards the robot on the time-scale of individual interactions. We posit that given these circumstances, this measure may be used as a proxy (among others) for engagement in the interaction or the human's attribution of social agency to the robot. Application of this method to a sample of unstructured child-robot interactions demonstrates its use, and justifies its utilisation in future studies.

A field study was conducted in which CRI activities developed by the ALIZ-E project were tested with the project's primary user group: children with diabetes. This field study resulted in new insights in the modalities and roles a robot aimed at CRI in a healthcare setting might utilise, while in addition (re-)assessed some practises and technologies established within the project. Furthermore, it served as a means of strengthening the bonds with the project's principal stakeholders. The study illustrates on the one hand the feasibility of the activities that were developed within the project, while on the other hand highlights the importance of engaging with primary users in an ongoing, incremental fashion.

Compliance studies in human-robot interaction (HRI) tend to consist of direct requests from the robot to the human. It is suggested that indirect requests are considered more polite, which has been positively correlated with learning gains. An experiment is conducted to explore compliance with indirect robot requests in teaching interactions. A comparison is made across embodiment conditions, but no significant differences are found. Overall, children comply with the robot's requests, which is used to support the hypothesis that given a well-defined context, children will infer the indirect meaning of a suggestion from a robot.

No abstract available.

This paper presents an experiment investigating the influence that a situational context has upon how people affectively interpret Non-Linguistic Utterances made by a social robot. Subjects were presented five video conditions showing the robot making both a positive and negative utterance, the robot being subject to an action (e.g. receiving a kiss, or a slap), and then two videos showing the combination of the action and the robot reacting with both the positive and negative utterances. For each video an affective rating of valence was provided based upon how the subjects thought the robot felt given what had happened in the video. This was repeated for 5 different action scenarios. Results show that the affective interpretation of an action appears to override that of an utterance, regardless of the affective charge of the utterance. Furthermore, it is shown that if the meaning of the action and utterance are aligned, the overall interpretation is amplified. These findings are considered with respect to the practical use of utterances during social HRI.

This paper presents the results of a small experiment aimed at determining whether people are comfortable with a social robot that uses robotic Non-Linguistic Utterances alongside Natural Language, rather than as a replacement. The results suggest that while people have the most preference for a robot that uses only natural language, a robot that combines NLUs and natural language is seen as more preferable than a robot that only employes NLUs. This suggests that there is potential for NLUs to be used in combination with natural language. In light of this, potential utilities and motivations for using NLUs in such a manner are outlined.

Being a child with diabetes is challenging: apart from the emotional difficulties of dealing with the disease, there are multiple physical aspects that need to be dealt with on a daily basis. Furthermore, as the children grow older, it becomes necessary to self-manage their condition without the explicit supervision of parents or carers. This process requires that the children overcome a steep learning curve. Previous work hypothesized that a robot could provide a supporting role in this process. In this paper, we characterise this potential support in greater detail through a structured collection of perspectives from all stakeholders, namely the diabetic children, their siblings and parents, and the healthcare professionals involved in their diabetes education and care. A series of brain-storming sessions were conducted with 22 families with a diabetic child (32 children and 38 adults in total) to explore areas in which they expected that a robot could provide support and/or assistance. These perspectives were then reviewed, validated and extended by healthcare professionals to provide a medical grounding. The results of these analyses suggested a number of specific functions that a companion robot could fulfil to support diabetic children in their daily lives.

We present an automated solution for the acquisition, processing and classification of electroencephalography (EEG) signals in order to remotely control a remotely located robotic hand executing communicative gestures. The Brain-Computer Interface (BCI) was implemented using the Steady State Visual Evoked Potential (SSVEP) approach, a low-latency and low-noise method for reading multiple non-time-locked states from EEG signals. As EEG sensor, the low-cost commercial Emotiv EPOC headset was used to acquire signals from the parietal and occipital lobes. The data processing chain is implemented in OpenViBE, a dedicated software platform for designing, testing and applying Brain-Computer Interfaces. Recorded commands were communicated to an external server through a Virtual Reality Peripheral Network (VRPN) interface. During the training phase, the user controlled a local simulation of a dexterous robot hand, allowing for a safe environment in which to train. After training, the user's commands were used to remotely control a real dexterous robot hand located in Bologna (Italy) from Plymouth (UK). We report on the robustness, accuracy and latency of the setup.

In this paper we present the design and technical implementation of a web based Multi-Modal User Interface (MMUI) tailored for elderly users of the robotic services developed by the EU FP7 Large-Scale Integration Project Robot-Era. The project partners are working to significantly enhance the performance and acceptability of technological services for ageing well by delivering a fully realized system based on the cooperation of multiple heterogeneous robots and with the support of an Ambient Assisted Living environment. To this end, elderly users were involved in the definition of the services and in the design of the hardware and software of the robotic platforms from the first stages of the development process and in real experimentation in two test sites. In particular, here we detail the interface software system for multi-modal elderly-robot interaction. The MMUI is designed to run on any device including touch-screen mobiles and tablets that are preferred by the elderly. This is obtained by integrating web based solutions with the Robot-Era middlewares and planner. Finally we present some preliminary results of ongoing experiments to show the successful evaluation of usability by potential users and to discuss the future directions to improve the proposed MMUI software system.

When studying social interactions, robust data collection protocols can come at the expense of allowing a natural interaction to take place because of a rigid structure in the experimental scenario. This work seeks to explore the use of an interaction mediator as a tool to constrain the content of a social interaction without imposing an interaction structure. Two studies were conducted in different interaction contexts using children: a peer-peer interaction, and a teacher-child interaction. Given that no interaction structure is imposed, objective metrics to characterise behaviour prove difficult to apply. Qualitative analysis techniques, namely Conversation Analysis, are therefore used to study the dyadic interactions. Results confirm the role of the mediating device in providing interaction content without imposing interaction structure. This illustrates the potential role of such devices in manipulating social interactions to facilitate empirical interrogation.

Human-Robot Interaction (HRI) systems are often realized through a complex integration of relevant state-of-the-art algorithms, low-level software and robotic hardware. As such, advances in techniques and approaches for integrating components into a robotic system are essential to advance the field. To develop such techniques and systems requires one to address real-world challenges, particularly those arising from the complex nature of human-robot interaction embedded in a dynamic social and/or task environment. This special issue focuses on the concept of "systems", highlighting the importance of building actual HRI systems to validate and drive the science and technology in the field.

In the development of companion robots capable of any-depth, long-term interaction, social scenarios enable exploration of the robot's capacity to engage a human interactant. These scenarios are typically constrained to structured task-based interactions, to enable the quantification of results for the comparison of differing experimental conditions. This paper introduces a hardware setup to facilitate and mediate human-robot social interaction, simplifying the robot control task while enabling an equalised degree of environmental manipulation for the human and robot, but without implicitly imposing an a priori interaction structure.

Vocal affective displays are vital for achieving engaging and effective Human-Robot Interaction. The same can be said for linguistic interaction also, however, while emphasis may be placed upon linguistic interaction, there are also inherent risks: users are bound to a single language, and breakdowns are frequent due to current technical limitations. This work explores the potential of non-linguistic utterances. A recent study is briefly outlined in which school children were asked to rate a variety of non-linguistic utterances on an affective level using a facial gesture tool. Results suggest, for example, that utterance rhythm may be an influential independent factor, whilst the pitch contour of an utterance may have little importance. Also evidence for categorical perception of emotions is presented, an issue that may impact important areas of HRI away from vocal displays of affect.

This video is about a new kind of robotic head. Through back-projection of a computer generated video into a half-translucent mask, the LightHead robotic head has many advantages compared to tradition mechatronic robotic faces. These advantages are most notably the versatility and ease of controlling facial expressions and creating new faces, the total weight and the low costs. By mounting the head on a robotic arm and equipping it with face detection software, the robot can interact with people in a natural manner.

We present insights gleaned from a series of child-robot interaction experiments carried out in a hospital paediatric department. Our aim here is to share good practice in experimental design and lessons learned about the implementation of systems for social HRI with child users towards application in "the wild", rather than in tightly controlled and constrained laboratory environments: a trade-off between the structures imposed by experimental design and the desire for removal of such constraints that inhibit interaction depth, and hence engagement, requires a careful balance.

Reading gaze direction is important in human-robot interactions as it supports, among others, joint attention and non-linguistic interaction. While most previous work focuses on implementing gaze direction reading on the robot, little is known about how the human partner in a human-robot interaction is able to read gaze direction from a robot. The purpose of this paper is twofold: (1) to introduce a new technology to implement robotic face using retro-projected animated faces and (2) to test how well this technology supports gaze reading by humans. We briefly discuss the robot design and discuss parameters influencing the ability to read gaze direction. We present an experiment assessing the user's ability to read gaze direction for a selection of different robotic face designs, using an actual human face as baseline. Results indicate that it is hard to recreate human-human interaction performance. If the robot face is implemented as a semi sphere, performance is worst. While robot faces having a human-like physiognomy and, perhaps surprisingly, video projected on a flat screen perform equally well and seem to suggest that these are the good candidates to implement joint attention in HRI.

This paper presents a new implementation of a robot face using retro-projection of a video stream onto a semitransparent facial mask. The technology is contrasted against mechatronic robot faces, of which Kismet is a typical example, and android robot faces, as used on the Ishiguro robots. The paper highlights the strengths of Retro-projected Animated Faces (RAF) technology (with cost, flexibility and robustness being notably strong) and discusses potential developments.

Language sets humans apart from other animals. In the context of robotics, language is a fast and surprisingly efficient channel to communicate thoughts; it is used to instruct and teach and is arg...

Darwin already remarked that evolutionary thinking also applies to the study of language.Language is heritable, in the sense that the language of offspring will likely resemble that of the parents, and during language learning variation is inevitably introduced.If on top of this a selection mechanism is operating that allows individuals using a particular language to have more descendants, some languages are more likely to spread through the population than others.The evolutionary nature of language change has been extensively studied in diachronic or historical linguistics, but in The Computational Nature of Language Learning and Evolution Niyogi takes a fresh approach by providing a formal study of evolutionary language change.In this he focuses on the population instead of on the individual language users, and provides a thorough analysis of the population dynamics resulting from individuals learning and using a language.Language evolution has only recently been subjected to the rigor of mathematical analysis.The expertise built up in theoretical biology, game theory, information theory, statistical physics, and complex systems research seems to be particularly well-suited to study and report on language dynamics at a macroscopic level.In addition, these disciplines rely on a set of trusted analytical tools that can be employed to study dynamical aspects of language learning and evolution.Niyogi sees language acquisition as a mapping of example sentences onto a private grammar.For this the language learner uses a particular learning mechanism, and the first part of the book is concerned with the "logical problem of language acquisition" (Gold 1967), and a number of formal learning mechanisms are presented that will be used in later chapters.Niyogi makes no commitment to the representation of grammars.Grammars can be generative grammars, but can equally be phonological rules, probabilistic grammars, sentence-meaning pairs, or any other combinatorial and compositional structure.However, he does assume that the learner has some sort of bias in its learning mechanism, a universal grammar if you will, that constrains the grammar acquisition process so that a stable language is maintained in a community.In the first chapters, a number of learning mechanisms are presented and explored, including the memory-less learner, which bases its next hypothesis only on its current hypothesis and the current sentence, and the batch learner, which waits until all example sentences have been received and then chooses the most likely hypothesis.If the learner is exposed to more example sentences, the hypothesis of the learner will home in on the target grammar of the teacher.Only if an infinite number of sentences are presented will the learner be able to acquire the exact same grammar as its teacher.In reality, no child ever hears an infinite number of sentences, so it is bound to learn a grammar that varies slightly from that of its caretakers.Exactly this variation drives

Robot imitation comprises a number of hard problems, one of the most difficult problems is the extraction of intention from a demonstration. A demonstration can almost always be interpreted in different ways, making it impossible for the imitating robot to find out what exactly it is that was demonstrated. We first attempt to set out the problem of intention reading. Next, we offer a computational model which implements a solution to intention reading. Our model needs repeated interactions between the demonstrator and the imitator. Through keeping a score about which interactions where successful, the imitating robot gradually builds a model which "understands" what the intent is of the demonstrator