Storytelling has accompanied humans since the beginning of mankind, and the reception of stories has become one of the most popular leisure activities. With their multimodal abilities, social robots bear great potential as storytellers. In this contribution, we present the Automatic Multimodal Robotic Storyteller – a system implemented for the Pepper robot which allows to automatically annotate a given story with emotions and perform it by the robot using emotional body language, emotion-inducing music, and colored lights. The system is empirically based on the results of several studies that are briefly summarized in this work. In an encompassing study, we compared the final Automatic Multimodal Robotic Storyteller to traditional storytelling media formats, namely text and audio book. While reading a story is preferred for the perceived control it offers, robotic storytellers are as well received as today’s traditional storytelling media in most aspects of storytelling experience, and even surpass the audio book in terms of emotion induction. Given this potential, our Automatic Multimodal Robotic Storyteller system could help robotic storytellers become a common storytelling medium in the future by making them accessible to a wide range of stories, by simply applying our pipeline to any given story in text format. To do so, the system will be made publicly available upon acceptance of this manuscript. The approach can also serve as guidance for other robotic storytellers.

Storytelling is a long-established human tradition for entertainment and knowledge transfer. Social robots are emerging as a new storytelling medium, being able to imitate human storytelling using gestures but also extend it by adding, e.g., sound effects to the experience. Due to COVID-19 restrictions, we conducted an online video-based study to investigate the effects of congruent respectively incongruent or no gesture usage in combination with additional non-speech sounds, i.e. sound effects and background music, on recipients' transportation into the story told, emotion induction, and perception of the robot. Results indicate no effect of additional non-speech sound integration on the variables listed above. Contradicting with related findings from in-person studies, we found a no significant differences between congruent, incongruent and no gesture usage. Last, no interplay of additional sounds and gesture congruence was identified. Future studies should provide deeper insights into the importance of multimodal congruence in video-taped robots and the possible advantages of adding non-speech sounds to online but also in-person robotic storytelling as well as their interplay in in-person HRI.

Virtual environments (VEs) can be designed to evoke specific emotions for example by using colored light, not only applicable for games but also for virtual storytelling with a single storyteller. Social robots are perfectly suited as storytellers due to their multimodality. However, there is no research yet on the transferability of robotic storytelling to virtual reality (VR). In addition, the transfer of concepts from VE design such as adaptive room illumination to robotic storytelling has yet not been tested. Thus, we conducted a study comparing the same robotic storytelling with a physically embodied robotic storyteller and in VR to investigate the transferability of robotic storytelling to VR. As a second factor, we manipulated the room light following design guidelines for VEs or kept it constant. Results show that a virtual robotic storyteller is not perceived worse than a physically embodied storyteller, suggesting the applicability of virtual static robotic storytellers. Regarding emotion-driven lighting, no significant effect of colored lights on self-reported emotions was found, but adding colored light increased the social presence of the robot and its’ perceived competence in both VR and reality. As our study was limited by a static robotic storyteller not using bodily expressiveness future work is needed to investigate the interaction between well-researched robot modalities and the rather new modality of colored light based on our results.

Storytelling is an important method not only in entertainment but also for education or therapy. Humor can improve storytelling due to its social functions. We investigate whether the humorous function of laughter can also be beneficial for robotic storytelling. We conducted a study focusing on the effects of voice type (human vs. synthetic) and laughter (presence vs. absence) on the robot’s perceived humorous appeal as well as recipients’ transportation, i.e. their absorption into the story, and emotions. No significant differences were found regarding the manipulations. The type of voice and use of laughter are therefore less relevant than assumed. However, it is shown that negative emotions are significantly reduced by the humorous story presented by the robot. Furthermore, the story was perceived as more funny and elicited increased transportation when being received by the robot compared to reading the story. In conclusion, this study indicates that robots are generally suitable for humorous storytelling and interesting results for future research on humorous robotic storytelling.

Social robots can use biomimetic modalities such as gestures to convey emotions. The use of colored light for emotion expression is also possible, but rarely explored. In this paper, colored LEDs are used in addition to contextual gestures to communicate emotions in robotic storytelling. Results show that adding colored light to the storytelling did not improve the recipients' transportation into the story. Their cognitive absorption was significantly decreased. The users' perception of the NAO robot was not influenced by colored lights except for animacy which was higher using only white LEDs. Problems might have been mismatches between colors and emotions, the lack of emotional gestures, and the too small light emission from NAO's eye LEDs. Since smart rooms lighting can affect the users' whole visual field, they could enhance emotional effects. Thus, future studies should investigate the integration of a robotic storyteller in smart rooms allowing for smart light control.

Social robots found their way into several economic fields of our daily life, e.g., as info points in shopping malls or hotels. However, research on robots in the tourism sector is still in its infancy. Conducting three online studies, we examined the design of robotic speech for a robotic concierge. Providing self-disclosure via speech or using local dialect did not affect robot acceptance and attitudes. In contrast, informal speech led to higher likeability and perceived warmth of the robot concierge. Overall, our results reveal positive evaluations of our robotic concierge regardless of its manipulation.

Social robots are applied in several fields such as education and entertainment. Using the method of storytelling, these two fields can be combined to allow for informal learning. As social robots can benefit students' motivation and learning process and storytelling is a consolidated method for vocabulary acquisition, integrating learning content into a robotic storyteller's narration might facilitate implicit learning. Focusing on language learning, we conducted an online study to compare a robotic storyteller for vocabulary learning to traditional media, e.g. text. Results show that a robotic storyteller is as effective as traditional media concerning long-term memory. While it is also perceived equally useful, participants were more satisfied when reading text. No differences were indicated for transportation into the story. This preliminary online study indicates the suitability of robotic storytelling for implicit language learning, however, future studies should examine live and co-located scenarios to investigate their full potential.

Abstract Reading is an essential ability and a cornerstone of education. However, learning to read can be challenging for children. To scaffold young learners, a number of reading interventions were developed, including a syllable-based approach in German, which has proven to be successful, but resource and time consuming through individual interaction by educators. To improve the reach of the reading intervention, we present the first step towards a digital intervention, following an iterative design approach. In this contribution, we present the implementation of a digital prototype, developed with the feedback of expert evaluations, as well as an interview study with second graders. The results of interviews with children showed that the app is suitable to be applied in the target age group, that children had fun using it and were motivated to further do so. In a next step towards a meaningful digitalization of the analogue intervention, we extended the application based on the evaluation results and conducted a usability evaluation of the extended app. The study as well as the usability evaluation provides design implications for iteratively transferring the analogue concept into a digital application.