As robots and virtual agents are increasingly envisioned as long-term companions, understanding how trust develops becomes crucial for ensuring safe and appropriate human-robot relationships. This research investigates how affective and cognitive trust evolve in social human-robot interactions. Participants (n=40) engaged in a 2 (social attitude: social, baseline) × 3 (time: t1, t2, t3) mixed-design user study with a social robot, using a novel Card Divination Task developed to elicit both cognitive and affective trust dimensions. Results show that cognitive trust develops early while affective trust emerges gradually. Moreover, social cues enhance both cognitive trust, affective trust, and participants’ certainty in trust judgment. These findings provide empirical support for the theoretical distinction between trust dimensions and highlight the role of social behavior in shaping trust over repeated interactions.

Music is widely used in human–computer interaction (HCI) to enhance engagement, sustain attention, and support cognitive stimulation. Yet its potential for deliberate mood regulation, particularly through personalized memory recall, remains largely unexplored. Music-evoked autobiographical memories (MEAMs) are often elicited by well-known, favorite songs, yielding stronger mood effects than music without personal memory associations. However, songs can also trigger distressing memories, and will never capture all positive personal memories. Since happy personal memories can enhance mood, broader methods for retrieval are needed. To address this, we introduce Constructed Music-Evoked Episodic Memories (CoMEEMs), a framework linking chosen episodic memories to music. By creating a personalized song-memory database, CoMEEMs enable autonomous mood regulation and communication in interactive systems, integrating memory cues—such as people and places—alongside mood congruence, to help choose songs with high mood regulatory impact. In an experiment with 71 Dutch and French adults, participants described 87 positive memories and received song recommendations based on associated people and places, with and without mood matching. Results showed that song familiarity and genre were the strongest predictors of perceived fit, while valence, arousal, tempo, and lyrics played smaller roles. Mood congruence, especially in valence, significantly influenced song relevance. Participants emphasized the need for user input on emotional states and memory context. Based on these findings, we propose design guidelines to improve future music recommendation systems targeting memories.

Social interactions form an essential aspect of people’s life, however, it is quite challenging for individuals to handle a wide range of social situations. Therefore, a variety of training systems have been developed to improve their skills. This literature review seeks to give an overview of the state of the art of technology-supported systems for social skills training. The studies eligible for inclusion described a technology-supported system with the purpose of training social skills and included an experimental or observational study to evaluate the efficacy of the system. 225 studies (224 publications) with 216 systems were identified, characterized, and analyzed in this literature review. Using the taxonomy as put forward in this study, the analysis shows that the majority of these systems were screen-based applications, with virtual reality technology being the most frequently observed. The systems most often targeted communication skills that focus on transferring information to produce greater understanding, i.e. mending general communication impairments in children with autism. In terms of functions, support for learning-by-doing was the most observed function, while focusing on job interviews provided the largest number of functions. Finally, the studies reported overwhelmingly positively regarding the systems’ impact, including 76 studies with a randomized controlled trial design. Still, most studies only used a quasi-experimental design based on self-report measures. We anticipate the proposed taxonomy to be a starting point for researchers to position their work and that the review will help them with gaining inspiration for the design and evaluation of social skills training systems.

This full-day workshop focuses on advancing research and development in socially-competent agents that care. Many agent platforms and kinds of embodiments of agents that have a primary focus on caring, one way or another, for the end users. In this workshop, we aim to look at value-based design, inclusiveness, empathetic design and long-term human-agent interaction for socially competent agents that care.

When a human makes a decision, an observer may want to understand the reasons and motivations behind the decision. This understanding is important when IVAs are involved in contextual decision-making or coaching practices. To address this challenge, we propose that an agent’s understanding of its user should include knowledge of the user’s underlying values. Humans prioritise different values – sometimes contradictory – in a manner that depends on the context. We present a method where the agent and user build the required context-sensitive value model together. We use Schwartz’s value theory, which places individuals’ values into ten categories. In a between-subject experiment, with three sessions on different days, we elicit user values by presenting them with moral dilemmas in different contexts on the first day, refine the model by asking users to argue about contradictions on the second day, and let them reflect on the model that they have built together with the system on the third day. We find that users exposed to a value-aware condition are more likely to agree with the robot’s representations of their values post-reflection than those in a baseline. Participants also prioritise different values depending on the context, agreeing with previous findings.

Whereas the reception task is a promising application domain for social robots, knowledge is lacking about how to design the appropriate re-usable communication styles for a reception robot. This paper presents the use and evaluation of an iterative interaction-design (ID) method with which task- and person-oriented multi-modal communication styles have been designed for such a robot. First, we report on an evaluation study of the ID-method with Industrial Design students (N =13) who designed these two communication styles for a Pepper robot. This provided a set of distinct designs of the two styles, for which the differences in design parameters were in line with social science theory. The task-oriented style showed a more formal, shorter and less chatty communication. Second, we present findings from a Mechanical Turk study conducted to evaluate the perception of these style designs. Participants (N =301) were presented with videos showing the robot acting as a receptionist and were asked to rate their perception of the robot, the service experience and the orientation of the designs. Overall, the interaction with the robot was appreciated well. The robot with a person-oriented style was perceived to be more animate and likeable. Analysis showed that chit-chat was the main contributor to the perceived difference between the person-oriented and task-oriented styles. This is an important finding as it gives interaction designers a validated best-practice approach to make interaction style more or less personal.

Inspired by the cognitive science theory of the explicit human memory systems, we have modeled an agent with short-term, episodic, and semantic memory systems, each of which is modeled with a knowledge graph. To evaluate this system and analyze the behavior of this agent, we designed and released our own reinforcement learning agent environment, “the Room”, where an agent has to learn how to encode, store, and retrieve memories to maximize its return by answering questions. We show that our deep Q-learning based agent successfully learns whether a short-term memory should be forgotten, or rather be stored in the episodic or semantic memory systems. Our experiments indicate that an agent with human-like memory systems can outperform an agent without this memory structure in the environment.

Abstract Virtual coaches have the potential to address the low adherence common to eHealth applications for behavior change by, for example, providing motivational support. However, given the multitude of factors affecting users’ attitudes toward virtual coaches, more insights are needed on how such virtual coaches can be designed to affect these attitudes in a specific use context positively. Especially valuable are insights that are based on users interacting with such a virtual coach for longer. We thus conducted a study in which more than 500 smokers interacted with the text-based virtual coach Sam in five sessions. In each session, Sam assigned smokers a new preparatory activity for quitting smoking and provided motivational support for doing the activity. Based on a mixed-methods analysis of users’ willingness to continue working and their relationship with Sam, we obtained eight themes for users’ attitudes toward Sam. These themes relate to whether Sam is seen as human or artificial, specific characteristics of Sam (e.g., caring character), the interaction with Sam, and the relationship with Sam. We used these themes to formulate literature-based recommendations to guide designers of virtual coaches for behavior change. For example, letting the virtual coach get to know users and disclose more information about itself may improve its relationship with users.

Social robots can support children in their socio-emotional development [38]. To improve the cooperation between a child and a social robot, a good relationship is vital. Self-disclosure is an essential element for building personal relationships. Yet, knowledge about the effects of self-disclosure in child-robot interactions is still lacking. To investigate effects of robot persona, child personality, and self-disclosure category on self-disclosure in child-robot interaction, we have conducted a field study at a science festival in which children had a conversation with a robot that either behaved human-like or robot-like. The results show a significant difference in the amount of self-disclosure (in conversation duration) between the two robot personas. Additionally, significant relationships were found between conscientiousness and extraversion and amount of self-disclosure (in word count). The participant disclosed significantly more about the category `Attitudes and Opinions’ than about ‘School’. Finally, a thematic analysis shows that the content of the conversations can be categorised in five plus one themes. Between robot personas, the content of the conversations did not differ in terms of conversation themes. However, in both conditions, we found that children generally feel comfortable sharing unpleasant experiences about present themes (such as COVID) in a first encounter with a robot.

In this article we introduceMementos: the first multimodal corpus for computational modeling of affect and memory processing in response to video content. It was collected online via crowdsourcing and captures 1995 individual responses collected from 297 unique viewers responding to 42 different segments of music videos. Apart from webcam recordings of their upper-body behavior (totaling 2012 minutes) and self-reports of their emotional experience, it contains detailed descriptions of the occurrence and content of 989 personal memories triggered by the video content. Finally, the dataset includes self-report measures related to individual differences in participants’ background and situation (Demographics,Personality, andMood), thereby facilitating the exploration of important contextual factors in research using the dataset. We describe 1) the construction and contents of the corpus itself, 2) analyse thevalidityof its content by investigating biases and consistency with existing research on affect and memory processing, 3) review previously published work that demonstrates theusefulnessof the multimodal data in the corpus for research on automated detection and prediction tasks, and 4) provide suggestions for how the dataset can be used in future research on modelingVideo-Induced Emotions,Memory-Associated Affect, andMemory Evocation.

While interacting with a social robot, children have a need to express themselves and have their expressions acknowledged by the robot—a need that is often unaddressed by the robot, due to its limitations in understanding the expressions of children. To keep the child-robot interaction manageable, the robot takes control, undermining children’s ability to co-regulate the interaction. Co-regulation is important for having a fulfilling social interaction. We developed a co-creation activity that aims to facilitate more co-regulation. Children are enabled to create sound effects, gestures, and light animations for the robot to use during their conversation. A crucial additional feature is that children are able to coordinate their involvement of the co-creation process. Results from a user study (n= 59 school children, 7–11 years old) showed that the co-creation activity successfully facilitated co-regulation by improving children’s agency. It also positively affected the acceptance of the robot. We furthermore identified five distinct profiles detailing the different needs and motivations children have for the level of involvement they chose during the co-creation process.

Discrimination and bias are pressing issues of many AI and robotics applications. These outcomes may derive from limited datasets that do not fully represent society as a whole or from the AI scientific community's western-male configuration bias. Although being a pressing issue, understanding how robotic systems can replicate and amplify inequalities and injustice among underrepresented communities is still in its infancy among social science and technical communities. This workshop contributes to filling this gap by exploring the research question: What do diversity and inclusion mean in the context of Human-Robot Interaction (HRI)? Here, attention is directed to three different levels of HRI: the technical, the community, and the target user level. Overall, this workshop will focus on the idea that AI systems can be created to be more attuned to inclusive societal needs, respect fundamental rights, and represent contemporary values in modern societies by integrating diversity and inclusion considerations.

After the novelty effect wears off children need a new motivator to keep interacting with a social robot. Enabling children to build a relationship with the robot is the key for facilitating a sustainable long-term interaction. We designed a memory-based personalization strategy that safeguards the continuity between sessions and tailors the interaction to the child's needs and interests to foster the child-robot relationship. A longitudinal (five sessions in two months) user study (N = 46, 8–10 y.o) showed that the strategy kept children interested longer in the robot, fosters more closeness, elicits more positive social cues, and adds continuity between sessions.

Self-disclosures can be valuable and sensitive parts of the human-robot interaction. This paper investigates how far human's tendency to self-disclose depends on the topic of interaction, individual's personality and perceived robot identity (i.e., human-, robot- or animal-like). Robot's (Pepper) identity was shown in its self-disclosure, interaction behaviors (gestures, sound and voice), and ’’clothing". In an"in-the- wild" study at a science festival, 80 visitors interacted with one of these robot identities. When questioned by the robot, they disclosed more about their attitudes and opinions than about other categories. Significant correlations appeared between personality characteristics and the degree of self-disclosure, as well as differences in self-disclosure categories. The different robot identities showed no effects on disclosures.

In ongoing and consecutive conversations with persons, a social robot has to determine which aspects to remember and how to address them in the conversation. In the health domain, important aspects concern the health-related goals, the experienced progress (expressed sentiment) and the ongoing motivation to pursue them. Despite the progress in speech technology and conversational agents, most social robots lack a memory for such experience sharing. This paper presents the design and evaluation of a conversational memory for personalized behavior change support conversations on healthy nutrition via memory-based motivational rephrasing. The main hypothesis is that referring to previous sessions improves motivation and goal attainment, particularly when references vary. In addition, the paper explores how far motivational rephrasing affects user’s perception of the conversational agent (the virtual Furhat). An experiment with 79 participants was conducted via Zoom, consisting of three conversation sessions. The results showed a significant increase in participants’ change in motivation when multiple references to previous sessions were provided.

The rapid advancement of technology empowered by artificial intelligence is believed to intensify the collaboration between humans and AI as team partners. Successful collaboration requires partners to learn about each other and about the task. This human-AI co-learning can be achieved by presenting situations that enable partners to share knowledge and experiences. In this paper we describe the development and implementation of a task context and procedures for studying co-learning. More specifically, we designed specific sequences of interactions that aim to initiate and facilitate the co-learning process. The effects of these interventions on learning were evaluated in an experiment, using a simplified virtual urban-search-and-rescue task for a human-robot team. The human participants performed a victim rescue- and evacuation mission in collaboration with a wizard-of-Oz (i.e., a confederate of the experimenter who executed the robot-behavior consistent with an ontology-based AI-model). The designed interaction sequences, formulated as Learning Design Patterns (LDPs), were intended to bring about co-learning. Results show that LDPs support the humans understanding and awareness of their robot partner and of the teamwork. No effects were found on collaboration fluency, nor on team performance. Results are used to discuss the importance of co-learning, the challenges of designing human-AI team tasks for research into this phenomenon, and the conditions under which co-learning is likely to be successful. The study contributes to our understanding of how humans learn with and from AI-partners, and our propositions for designing intentional learning (LDPs) provide directions for applications in future human-AI teams.

Making the transition to long-term interaction with social-robot systems has been identified as one of the main challenges in human-robot interaction. This article identifies four design principles to address this challenge and applies them in a real-world implementation: cloud-based robot control, a modular design, one common knowledge base for all applications, and hybrid artificial intelligence for decision making and reasoning. The control architecture for this robot includes a common Knowledge-base (ontologies), Data-base, “Hybrid Artificial Brain” (dialogue manager, action selection and explainable AI), Activities Centre (Timeline, Quiz, Break and Sort, Memory, Tip of the Day, \( \ldots \) ), Embodied Conversational Agent (ECA, i.e., robot and avatar), and Dashboards (for authoring and monitoring the interaction). Further, the ECA is integrated with an expandable set of (mobile) health applications. The resulting system is a Personal Assistant for a healthy Lifestyle (PAL), which supports diabetic children with self-management and educates them on health-related issues (48 children, aged 6–14, recruited via hospitals in the Netherlands and in Italy). It is capable of autonomous interaction “in the wild” for prolonged periods of time without the need for a “Wizard-of-Oz” (up until 6 months online). PAL is an exemplary system that provides personalised, stable and diverse, long-term human-robot interaction.

Current developments in Artificial Intelligence (AI) led to a resurgence of Explainable AI (XAI). New methods are being researched to obtain information from AI systems in order to generate explanations for their output. However, there is an overall lack of valid and reliable evaluations of the effects on users' experience of, and behavior in response to explanations. New XAI methods are often based on an intuitive notion what an effective explanation should be. Rule- and example-based contrastive explanations are two exemplary explanation styles. In this study we evaluate the effects of these two explanation styles on system understanding, persuasive power and task performance in the context of decision support in diabetes self-management. Furthermore, we provide three sets of recommendations based on our experience designing this evaluation to help improve future evaluations. Our results show that rule-based explanations have a small positive effect on system understanding, whereas both rule- and example-based explanations seem to persuade users in following the advice even when incorrect. Neither explanation improves task performance compared to no explanation. This can be explained by the fact that both explanation styles only provide details relevant for a single decision, not the underlying rational or causality. These results show the importance of user evaluations in assessing the current assumptions and intuitions on effective explanations.

Effective psychological interventions for anxiety disorders often include exposure to fearful situations. However, individuals with low self-efficacy may find such exposure too overwhelming. We created a vicarious experience in virtual reality, which enables observation of one’s experience from a first person perspective without actual performance and which might increase self-efficacy. With similarities to both traditional vicarious experiences and direct experiences, the level of self-identification with the experience was hypothesized to affect self-efficacy and its relationship with direct experiences. To test this, vicarious experiences with two distinct levels of self-identification were compared in a between-subjects experiment (n=60). After being exposed to a vicarious experience of giving lectures on elementary arithmetic in front of a virtual audience with either a high or low level of self-identification with the public speaker, participants from both conditions actively gave another lecture. The results revealed that self-identification affected people’s self-efficacy after vicarious experience. They further revealed that self-identification is a moderator of (1) the correlation between perceived performance and self-efficacy, (2) the correlation between self-efficacy measured after the vicarious and the follow-up direct experience; and (3) the correlation between the sense of presence reported in the vicarious and in the follow-up direct experience. We anticipate that the first-person-perspective experiences with high-level of self-identification have the potential to be beneficial for training where changing people’s self-efficacy is desirable.

Much of the research on eXplainable Artificial Intelligence (XAI) has centered on providing transparency of machine learning models. More recently, the focus on human-centered approaches to XAI has increased. Yet, there is a lack of practical methods and examples on the integration of human factors into the development processes of AI-generated explanations that humans prove to uptake for better performance. This paper presents a case study of an application of a human-centered design approach for AI-generated explanations. The approach consists of three components: Domain analysis to define the concept & context of explanations, Requirements elicitation & assessment to derive the use cases & explanation requirements, and the consequential Multi-modal interaction design & evaluation to create a library of design patterns for explanations. In a case study, we adopt the DoReMi-approach to design explanations for a Clinical Decision Support System (CDSS) for child health. In the requirements elicitation & assessment, a user study with experienced paediatricians uncovered what explanations the CDSS should provide. In the interaction design & evaluation, a second user study tested the consequential interaction design patterns. This case study provided a first set of user requirements and design patterns for an explainable decision support system in medical diagnosis, showing how to involve expert end users in the development process and how to develop, more or less, generic solutions for general design problems in XAI.

Abstract Virtual cognitions (VCs) are a stream of simulated thoughts people hear while emerged in a virtual environment, e.g. by hearing a simulated inner voice presented as a voice over. They can enhance people’s self-efficacy and knowledge about, for example, social interactions as previous studies have shown. Ownership and plausibility of these VCs are regarded as important for their effect, and enhancing both might, therefore, be beneficial. A potential strategy for achieving this is the synchronization of the VCs with people’s eye fixation using eye-tracking technology embedded in a head-mounted display. Hence, this paper tests this idea in the context of a pre-therapy for spider and snake phobia to examine the ability to guide people’s eye fixation. An experiment with 24 participants was conducted using a within-subjects design. Each participant was exposed to two conditions: one where the VCs were adapted to eye gaze of the participant and the other where they were not adapted, i.e. the control condition. The findings of a Bayesian analysis suggest that credibly more ownership was reported and more eye-gaze shift behaviour was observed in the eye-gaze-adapted condition than in the control condition. Compared to the alternative of no or negative mediation, the findings also give some more credibility to the hypothesis that ownership, at least partly, positively mediates the effect eye-gaze-adapted VCs have on eye-gaze shift behaviour. Only weak support was found for plausibility as a mediator. These findings help improve insight into how VCs affect people.

Abstract In this study, we attempt to specify the cognitive support behavior of a previously designed embodied conversational agent coach that provides learning support to low-literates. Three knowledge gaps are identified in the existing work: an incomplete specification of the behaviors that make up ‘support,’ an incomplete specification of how this support can be personalized, and unclear speech recognition rules. We use the socio-cognitive engineering method to update our foundation of knowledge with new online banking exercises, low-level scaffolding and user modeling theory, and speech recognition. We then refine the design of our coach agent by creating comprehensive cognitive support rules that adapt support based on learner needs (the ‘Generalized’ approach) and attune the coach’s support delay to user performance in previous exercises (the ‘Individualized’ approach). A prototype is evaluated in a 3-week within- and between-subjects experiment. Results show that the specified cognitive support is effective: Learners complete all exercises, interact meaningfully with the coach, and improve their online banking self-efficacy. Counter to hypotheses, the Individualized approach does not improve on the Generalized approach. Whether this indicates suboptimal operationalization or a deeper problem with the Individualized approach remains as future work.

In this paper we specify and validate three interaction design patterns for an interactive storytelling experience with an autonomous social robot. The patterns enable the child to make decisions about the story by talking with the robot, reenact parts of the story together with the robot, and recording self-made sound effects. The design patterns successfully support children's engagement and agency. A user study (N = 27, 8-10 y.o.) showed that children paid more attention to the robot, enjoyed the storytelling experience more, and could recall more about the story, when the design patterns were employed by the robot during storytelling. All three aspects are important features of engagement. Children felt more autonomous during storytelling with the design patterns and highly appreciated that the design patterns allowed them to express themselves more freely. Both aspects are important features of children's agency. Important lessons we have learned are that reducing points of confusion and giving the children more time to make themselves heard by the robot will improve the patterns efficiency to support engagement and agency. Allowing children to pick and choose from a diverse set of stories and interaction settings would make the storytelling experience more inclusive for a broader range of children.

Empirical evidence suggests that the emotional meaning of facial behavior in isolation is often ambiguous in real-world conditions. While humans complement interpretations of others' faces with additional reasoning about context, automated approaches rarely display such context-sensitivity. Empirical findings indicate that the personal memories triggered by videos are crucial for predicting viewers' emotional response to such videos ?- in some cases, even more so than the video's audiovisual content. In this article, we explore the benefits of personal memories as context for facial behavior analysis. We conduct a series of multimodal machine learning experiments combining the automatic analysis of video-viewers' faces with that of two types of context information for affective predictions: \beginenumerate* [label=(\arabic*)] \item self-reported free-text descriptions of triggered memories and \item a video's audiovisual content \endenumerate*. Our results demonstrate that both sources of context provide models with information about variation in viewers' affective responses that complement facial analysis and each other.

The design space of human-robot interaction is large and multi-dimensional. A sound design requires a systematic theory-driven exploration, specification and refinement of design variables. There is a need for a practical method and tool to iteratively specify the content of the dialogue (e.g., speech acts) with the accompanying expressive behavior (e.g., gesture openness) as prescribed by social science theory, e.g., task- and person-oriented communication. This paper presents an iterative interaction-design (ID) method for multi-modal robot communication. Following the ID-method, a designer first creates his/her "own" individual design and, subsequently, provides an iteration to the evolving iterative design. To support the design method, we developed an ID-tool (available for download). The tool support entails (a) selecting the theory-based communication style; (b) creating and linking the dialogue act components for the concerning use case; and (c) setting the associated expression parameters.We conducted a study with Industrial Design students (N = 13) who followed the ID-method and used our tool to design person- and task-oriented communications for a reception robot. Our method produced distinctive task- and person-oriented dialogue styles, i.e., provided the predicted theory-based multi-modal communicative behaviors. The task-oriented style showed a more formal, shorter and less chatty communication. Overall, there was a rather smooth design convergence process, in which the individual designs were harmonized into the iterative design. For the selected design problem, the ID-tool had a satisfactory usability. Next steps include validation of the communication styles in an empirical study and, subsequently, identification of reusable design patterns.

This paper contributes to the automatic estimation of the subjective emotional experience that audio-visual media content induces in individual viewers, e.g. to support affect-based recommendations. Making accurate predictions of these responses is a challenging task because of their highly person-dependent and situation-specific nature. Findings from psychology indicate that an important driver for the emotional impact of media is the triggering of personal memories in observers. However, existing research on automated predictions focuses on the isolated analysis of audiovisual content, ignoring such contextual influences. In a series of empirical investigations, we (1) quantify the impact of associated personal memories on viewers' emotional responses to music videos in-the-wild and (2) assess the potential value of information about triggered memories for personalizing automatic predictions in this setting. Our findings indicate that the occurrence of memories intensifies emotional responses to videos. Moreover, information about viewers' memory response explains more variation in video-induced emotions than either the identity of videos or relevant viewer-characteristics (e.g. personality or mood). We discuss the implications of these results for existing approaches to automated predictions and describe ways for progress towards developing memory-sensitive alternatives.

Decision support systems (DSS) have improved significantly but are more complex due to recent advances in Artificial Intelligence. Current XAI methods generate explanations on model behaviour to facilitate a user’s understanding, which incites trust in the DSS. However, little focus has been on the development of methods that establish and convey a system’s confidence in the advice that it provides. This paper presents a framework for Interpretable Confidence Measures (ICMs). We investigate what properties of a confidence measure are desirable and why, and how an ICM is interpreted by users. In several data sets and user experiments, we evaluate these ideas. The presented framework defines four properties: 1) accuracy or soundness, 2) transparency, 3) explainability and 4) predictability. These characteristics are realized by a case-based reasoning approach to confidence estimation. Example ICMs are proposed for -and evaluated on- multiple data sets. In addition, ICM was evaluated by performing two user experiments. The results show that ICM can be as accurate as other confidence measures, while behaving in a more predictable manner. Also, ICM’s underlying idea of case-based reasoning enables generating explanations about the computation of the confidence value, and facilitates user’s understandability of the algorithm.

The eight papers in this special issue focus on agent and system transparency in human-machine systems. The concept of transparency is investigated in a variety of contexts of human agent interaction—from a single robot to multiple heterogeneous agents and swarms. Two studies examine the effects of individual and cultural differences and, based on the compelling results, provide design recommendations related to transparent interfaces. One of the papers provides a thorough review on theoretical aspects of agent transparency and empirical findings on operator performance, situation awareness, trust and workload, among other outcomes. These measures are also among some of the most common metrics reported in studies in this special issue.

Recent advances in wearable sensor technology and smartphones enable simple and affordable collection of personal analytics. This paper reflects on the lessons learned in the SWELL project that addressed the design of user-centered ICT applications for self-management of vitality in the domain of knowledge workers. These workers often have a sedentary lifestyle and are susceptible to mental health effects due to a high workload. We present the sense–reason–act framework that is the basis of the SWELL approach and we provide an overview of the individual studies carried out in SWELL. In this paper, we revisit our work on reasoning: interpreting raw heterogeneous sensor data, and acting: providing personalized feedback to support behavioural change. We conclude that simple affordable sensors can be used to classify user behaviour and heath status in a physically non-intrusive way. The interpreted data can be used to inform personalized feedback strategies. Further longitudinal studies can now be initiated to assess the effectiveness of m-Health interventions using the SWELL methods.

An important aspect of human emotion perception is the use of contextual information to understand others' feelings even in situations where their behavior is not very expressive or has an emotionally ambiguous meaning. For technology to successfully detect affect, it must mimic this human ability when analyzing audiovisual input. Databases upon which machine learning algorithms are trained should capture the context of social interactions as well as the behavior expressed in them. However, there is a lack of consensus about what constitutes relevant context in such databases. In this article, we make two contributions towards overcoming this challenge: (a) we identify two principal sources of context for emotion perceptions based on psychological theory, and (b) we provide an overview of how each of these has been considered in published databases covering social interactions. Our results show that a similar set of contextual features are present across the reviewed databases. Between all the different databases researchers seem to have taken into account a set of contextual features reflecting the sources of context seen in psychological theory. However, within individual databases, these features are not yet systematically varied. This is problematic because it prevents them from being used directly as resources for the modeling of context-sensitive affect detection. Based on our findings, we suggest improvements for the future development of affective databases.

Explanation of actions is important for transparency of-, and trust in the decisions of smart systems. Literature suggests that emotions and emotion words - in addition to beliefs and goals - are used in human explanations of behaviour. Furthermore, research in e-health support systems and human-robot interaction stresses the need for studying long-term interaction with users. However, state of the art explainable artificial intelligence for intelligent agents focuses mainly on explaining an agent's behaviour based on the underlying beliefs and goals in short-term experiments. In this paper, we report on a long-term experiment in which we tested the effect of cognitive, affective and lack of explanations on children's motivation to use an e-health support system. Children (aged 6-14) suffering from type 1 diabetes mellitus interacted with a virtual robot as part of the e-health system over a period of 2.5 - 3 months. Children alternated between the three conditions. Agent behaviours that were explained to the children included why 1) the agent asks a certain quiz question; 2) the agent provides a specific tip (a short instruction) about diabetes; or, 3) the agent provides a task suggestion, e.g., play a quiz, or, watch a video about diabetes. Their motivation was measured by counting how often children would follow the agent's suggestion, how often they would continue to play the quiz or ask for an additional tip, and how often they would request an explanation from the system. Surprisingly, children proved to follow task suggestions more often when no explanation was given, while other explanation effects did not appear. This is to our knowledge the first longterm study to report empirical evidence for an agent explanation effect, challenging the next studies to uncover the underlying mechanism.

A mayor challenge in human-robot interaction and collaboration is the synthesis of non-verbal behaviour for the expression of social signals. Appropriate perception and expression of dominance (verticality) in non-verbal behaviour is essential for social interaction. In this paper, we present our work on algorithmic modulation of robot bodily movement to express varying degrees of dominance. We developed a parameter-based model for head tilt and body expansiveness. This model was applied to a variety of behaviours. These behaviours were evaluated by human observers in two different studies with respectively static pictures of key postures (N=772) and realtime gestures (N=31). Overall, specific behaviours proved to communicate different levels of dominance. Further, modulation of body expansiveness and head tilt robustly influenced perceived dominance independent of specific behaviours and observer viewing height and angle. The modulation did not influence perceived valence, but it did influence perceived arousal. Our study shows that dominance can be reliably expressed by both selection of specific behaviours and modulation of behaviours.

The current attention on quality monitoring instruments for hospitalized patients imposes a high data registration workload on nurses. The focus of our research was to investigate whether a social robot is able to take over some of this data collection by administering questionnaires autonomously. We performed an exploratory design experiment on the internal medicine ward of the Franciscus Gasthuis & Vlietland hospital. 35 patients (mean age 64.1±17.7, 20 female) participated in the study. We used the social robot Pepper to conduct five questionnaires on medical history, defecation, pain, memory and sleep. Patients and nurses found the robot reasonably acceptable in this role. Further research is needed to address concerns and optimize the nurse-robot task division.

Humans use several social cues, both verbal and nonverbal, to draw the attention of others. In this study we investigate whether similar behaviors can also be effectively used by a social robot for drawing attention. To this end, we setup a welcoming humanoid (Pepper) at the entrance of a university building. Its behaviors include one or a combination of behavioral modalities (i.e., a waving gesture, utterance and movement). These behaviors are triggered automatically based on people detection software which tracks passersby and monitors their head keypoints. Our findings imply that Pepper draws more attention when displaying a combination of modalities.

Drawing the attention of passersby is a basic task of a social robot to initiate an interaction in a public environment (e.g., shopping malls, museums or hospitals). Humans use several social cues, both verbal and nonverbal, to draw the attention of others. In this study, we investigate whether similar behaviors can also be effectively used by a social robot for drawing attention. To this end, we setup a humanoid robot (Pepper) to act as a welcoming robot at the entrance of a university building. The behaviors selected for Pepper include one or a combination of behavioral modalities (i.e., a waving gesture, utterance and movement). These behaviors are triggered automatically using the output of people detection software which tracks passersby and monitors their head keypoints (nose, eyes, and ears). The reactions of people toward Pepper are observed and recorded by means of an observation sheet. For several weeks, we deployed Pepper at the entrance with the aim of wearing off the novelty effect. In our final study, we collected data from several hundreds of passersby (N=364) and conducted post-interviews with randomly selected ones (N=28). Passersby noticed Pepper at the entrance and clearly recognized its role as a welcoming robot. In addition, Pepper was able to draw more attention when displaying a combination of behavioral modalities. However, passersby did not recall the robot utterance as they, for example, were unable to reproduce it or mistakenly claimed that the robot said something when it was only waving.

This paper presents the design and evaluation of human-like welcoming behaviors for a humanoid robot to draw the attention of passersby by following a three-step model: (1) selecting a target (person) to engage, (2) executing behaviors to draw the target's attention, and (3) monitoring the attentive response. A computer vision algorithm was developed to select the person, start the behaviors and monitor the response automatically. To vary the robot's enthusiasm when engaging passersby, a waving gesture was designed as basic welcoming behavioral element, which could be successively combined with an utterance and an approach movement. This way, three levels of enthusiasm were implemented: Mild (waving), moderate (waving and utterance) and high (waving, utterance and approach movement). The three levels of welcoming behaviors were tested with a Pepper robot at the entrance of a university building. We recorded data and observation sheets from several hundreds of passersby (N = 364) and conducted post-interviews with randomly selected passersby (N = 28). The level selection was done at random for each participant. The passersby indicated that they appreciated the robot at the entrance and clearly recognized its role as a welcoming robot. In addition, the robot proved to draw more attention when showing high enthusiasm (i.e., more welcoming behaviors), particularly for female passersby.

A mayor challenge in human-robot interaction is the synthesis of social signals through non-verbal behaviour expression. Appropriate perception and expression of dominance (verticality) is essential for social interaction. In this paper, we present our work on algorithmic modulation of robot bodily movement to control dominance expression. We developed a parameter-based model for body expansiveness. This model was applied to a variety of behaviours and evaluated by human observers in two different studies with respectively static postures (N=772) and gestures (N=31). Modulation of body expansiveness proved to robustly influence perceived dominance independent of behaviour and viewing angles.

Exploration games are games where agents (or robots) need to search resources and retrieve these resources. In principle, performance in such games can be improved either by adding more agents or by exchanging more messages. However, both measures are not free of cost and it is important to be able to assess the trade-off between these costs and the potential performance gain. The focus of this paper is on improving our understanding of the performance gain that can be achieved either by adding more agents or by increasing the communication load. Performance gain moreover is studied by taking several other important factors into account such as environment topology and size, resource-redundancy, and task size. Our results suggest that there does not exist a decision function that dominates all other decision functions, i.e. is optimal for all conditions. Instead we find that (i) for different team sizes and communication strategies different agent decision functions perform optimal, and that (ii) optimality of decision functions also depends on environment and task parameters. We also find that it pays off to optimize for environment topologies. Copyright © 2018 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved. Institute for Systems and Technologies of Information, Control and Communication (INSTICC)

Task allocation and management is crucial for human-robot collaboration in Urban Search And Rescue response efforts. The job of a mission team leader in managing tasks becomes complicated when adding multiple and different types of robots to the team. Therefore, to effectively accomplish mission objectives, shared situation awareness and task management support are essential. In this paper, we design and evaluate an ontology which provides a common vocabulary between team members, both humans and robots. The ontology is used for facilitating data sharing and mission execution, and providing the required automated task management support. Relevant domain entities, tasks, and their relationships are modeled in an ontology based on vocabulary commonly used by firemen, and a user interface is designed to provide task tracking and monitoring. The ontology design and interface are deployed in a search and rescue system and its use is evaluated by firemen in a task allocation and management scenario. Results provide support that the proposed ontology (1) facilitates information sharing during missions; (2) assists the team leader in task allocation and management; and (3) provides automated support for managing an Urban Search and Rescue mission. Copyright © 2018 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved Institute for Systems and Technologies of Information, Control and Communication (INSTICC)

End-users of machine learning-based systems benefit from measures that quantify the trustworthiness of the underlying models. Measures like accuracy provide for a general sense of model performance, but offer no detailed information on specific model outputs. Probabilistic outputs, on the other hand, express such details, but they are not available for all types of machine learning, and can be heavily influenced by bias and lack of representative training data. Further, they are often difficult to understand for non-experts. This study proposes an intuitive certainty measure (ICM) that produces an accurate estimate of how certain a machine learning model is for a specific output, based on errors it made in the past. It is designed to be easily explainable to non-experts and to act in a predictable, reproducible way. ICM was tested on four synthetic tasks solved by support vector machines, and a real-world task solved by a deep neural network. Our results show that ICM is both more accurate and intuitive than related approaches. Moreover, ICM is neutral with respect to the chosen machine learning model, making it widely applicable

Today, around 260.000 people in the Netherlands are diagnosed with dementia. This paper provides a first design and evaluation of a social robot that provides the music which supports positive self-disclosures of personal memories. Based on the situated cognitive engineering methodology, we developed software for the Pepper robot which can let people with dementia (PwD) listen to their favorite music and we tested this robot in a care center with seven persons with dementia. The test focused on robot's usability and music's effect on the person listening. Three general claims were tested: The robot (1) is easy to use by the PwD, relatives and caregivers, (2) brings PwD in a more positive emotional state, and (3) stimulates PwD to recall memories and talk about their past. Our results show that the “music robot” for PwD often elicits very strong positive responses and that our observations are in line with the three claims. However, using a very human-like robot like Pepper does pose certain challenges as the PwD expect it to truly understand every word they say, which is not yet the case.

Employees often report the experience of stress at work. In the SWELL project we investigate how new context aware pervasive systems can support knowledge workers to diminish stress. The focus of this paper is on developing automatic classifiers to infer working conditions and stress related mental states from a multimodal set of sensor data (computer logging, facial expressions, posture and physiology). We address two methodological and applied machine learning challenges: 1) Detecting work stress using several (physically) unobtrusive sensors, and 2) Taking into account individual differences. A comparison of several classification approaches showed that, for our SWELL-KW dataset, neutral and stressful working conditions can be distinguished with 90 percent accuracy by means of SVM. Posture yields most valuable information, followed by facial expressions. Furthermore, we found that the subjective variable`mental effort' can be better predicted from sensor data than, e.g., `perceived stress'. A comparison of several regression approaches showed that mental effort can be predicted best by a decision tree (correlation of 0.82). Facial expressions yield most valuable information, followed by posture. We find that especially for estimating mental states it makes sense to address individual differences. When we train models on particular subgroups of similar users, (in almost all cases) a specialized model performs equally well or better than a generic model.

Social commitments (SCs) provide a flexible, norm-based, governance structure for sharing and receiving data. However, users of data sharing applications can subscribe to multiple SCs, possibly producing opposing sharing and receiving requirements. We propose resolving such conflicts automatically through a conflict resolution model based on relevant user values such as privacy and safety. The model predicts a user’s preferred resolution by choosing the commitment that best supports the user’s values. We show through an empirical user study ( n = 396) that values, as well as recency and norm type, significantly improve a system’s ability to predict user preference in location sharing conflicts.

In the collaborative driving scenario of truck platooning, the first car is driven by its chauffeur and the next cars follow automatically via a so-called 'virtual tow-bar'. The chauffeurs of the following cars do not drive 'in the towbar mode', but need to be able to take back control in foreseen emph{and} unforeseen conditions. It is crucial that this transfer of control only takes place when the chauffeur is ready for it. This paper presents a Driver Readiness (DR) ontological model that specifies the core factors, with their relationships, of a chauffeur's current and near-future readiness for taking back the control of driving. A first model was derived from a literature study and an analysis of truck driving data, which was refined subsequently based on an expert review. This DR model distinguishes (a) current and required states for the physical (hand, feet, head, and seating position) and mental readiness (attention and situation awareness), (b) agents (human and machine actor), (c) policies for agent behaviors, and (d) states of the vehicle and its environment. It provides the knowledge base of a Control Transfer Support (CTS) agent that assesses the current and predicted chauffeur state and guides the transition of control in an adaptive and personalized manner. The DR model will be fed by information from the network and in-car sensors. The behaviors of the CTS agent will be generated and constrained by the instantiated policies, providing an important step towards a safe transfer of control from automation to human driver.

Educational technology needs a model of learning goals to support motivation, learning gain, tailoring of the learning process, and sharing of the personal goals between different types of users (i.e., learner and educator) and the system. This paper proposes a tree-based learning goal structuring to facilitate personal goal setting to shape and monitor the learning process. We developed a goal ontology and created a user interface representing this knowledge-base for the self-management education for children with Type 1 Diabetes Mellitus. Subsequently, a co-operative evaluation was conducted with healthcare professionals to refine and validate the ontology and its representation. Presentation of a concrete prototype proved to support professionals' contribution to the design process. The resulting tree-based goal structure enables three important tasks: ability assessment, goal setting and progress monitoring. Visualization should be clarified by icon placement and clustering of goals with the same difficulty and topic. Bloom's taxonomy for learning objectives should be applied to improve completeness and clarity of goal content.

A good explanation takes the user who is receiving the explanation into account. We aim to get a better understanding of user preferences and the differences between children and adults who receive explanations from a robot. We implemented a Nao-robot as a belief-desire-intention (BDI)-based agent and explained its actions using two different explanation styles. Both are based on how humans explain and justify their actions to each other. One explanation style communicates the beliefs that give context information on why the agent performed the action. The other explanation style communicates the goals that inform the user of the agent's desired state when performing the action. We conducted a user study (19 children, 19 adults) in which a Nao-robot performed actions to support type 1 diabetes mellitus management. We investigated the preference of children and adults for goalversus belief-based action explanations. From this, we learned that adults have a significantly higher tendency to prefer goal-based action explanations. This work is a necessary step in addressing the challenge of providing personalised explanations in human-robot and human-agent interaction.

Children are eager to anthropomorphize (ascribe human attributes to) social robots. As a consequence they expect a more unconstrained, substantive and useful interaction with the robot than is possible with the current state-of-the art. In this paper we reflect on several of our user studies and investigate the form and role of expectations in child-robot interaction. We have found that the effectiveness of the social assistance of the robot is negatively influenced by misaligned expectations. We propose three strategies that have to be worked out for the management of expectations in child-robot interaction: 1) be aware of and analyze children's expectations, 2) educate children, and 3) acknowledge robots are (perceived as) a new kind of `living' entity besides humans and animals that we need to make responsible for managing expectations.

Social robots are entering the private and public domain where they engage in social interactions with nontechnical users. This requires robots to be socially interactive and intelligent, including the ability to display appropriate social behaviour. Progress has been made in emotion modelling. However, research into behaviour style is less thorough; no comprehensive, validated model exists of non-verbal behaviours to express style in human-robot interactions. Based on a literature survey, we created a model of non-verbal behaviour to express high/low warmth and competence - two dimensions that contribute to teaching style. In a perception study, we evaluated this model applied to a NAO robot giving a lecture at primary schools and a diabetes camp in the Netherlands. For this, we developed, based on expert ratings, an instrument measuring perceived warmth, competence, dominance and affiliation. We show that even subtle manipulations of robot behaviour influence children's perceptions of the robot's level of warmth and competence.

A reliable wireless connection between the operator and the teleoperated unmanned ground vehicle (UGV) is critical in many urban search and rescue (USAR) missions. Unfortunately, as was seen in, for example, the Fukushima nuclear disaster, the networks available in areas where USAR missions take place are often severely limited in range and coverage. Therefore, during mission execution, the operator needs to keep track of not only the physical parts of the mission, such as navigating through an area or searching for victims, but also the variations in network connectivity across the environment.In this paper, we propose and evaluate a new teleoperation user interface (UI) that includes a way of estimating the direction of arrival (DoA) of the radio signal strength (RSS) and integrating the DoA information in the interface. The evaluation shows that using the interface results in more objects found, and less aborted missions due to connectivity problems, as compared to a standard interface.The proposed interface is an extension to an existing interface centered on the video stream captured by the UGV. But instead of just showing the network signal strength in terms of percent and a set of bars, the additional information of DoA is added in terms of a color bar surrounding the video feed. With this information, the operator knows what movement directions are safe, even when moving in regions close to the connectivity threshold.

In search and rescue missions robots are used to help rescue workers in exploring the disaster site. Our research focuses on how multiple robots and rescuers act as a team, and build up situation awareness. We propose a multi-agent system where each agent supports one member, either human or robot. For representing high-level information about the mission, we design an ontology that serves as a shared knowledge base for the agents. We investigate how to create agent-based ontological reasoning to provide team members with decision support, automate basic monitoring tasks, and realize the display logic that dictates how to display useful information for the rescuers, based on their different roles, tasks and situations.

This paper investigates the experiences of Italian and Dutch children while interacting with a social robot that is designed to support their diabetes self-management. Observations of children's behaviors and analyses of questionnaires at diabetes camps, showed positive experiences with variation (e.g., Italian children seemed to be more open and expressive, and more close to the robot compared to the Dutch children). A culture-aware robot should be sensitive to such differences.

Many researchers have started to explore natural interaction scenarios for children. No matter if these children are normally developing or have special needs, evaluating Child-Robot Interaction (CRI) is a challenge. To find methods that work well and provide reliable data is difficult, for example because commonly used methods such as questionnaires do not work well particularly with younger children. Previous research has shown that children need support in expressing how they feel about technology. Given this, researchers often choose time-consuming behavioral measures from observations to evaluate CRI. However, these are not necessarily comparable between studies and robots.

This paper describes ongoing work carried out in the European project PAL which will support childre in their diabetes self-management as well as assist health professionals and parents involved in the diabete regimen of the child. Here, we will focus on the construction of the PAL ontology which has been assemble from several independently developed sub-ontologies and which are brought together by a set of hand-writte interface axioms, expressed in OWL.We will describe in detail how the triple model of RDF has been extende towards transaction time in order to represent time-varying data. Examples of queries and rules involvin temporal information will be presented as well. The approach is currently been in use in diabetes camps. Copyright © 2016.Fundacao para a Ciencia e Tecnologia (FCT); Institute for Systems and Technologies of Information, Control and Communication (INSTICC)

Conversation Fillers (CFs), such as ‘um’, ‘hmm’, and ‘ah’, may help to improve the human-robot interaction by smoothening the robot's responses. This paper presents the design and test of such CFs - alongside iconic pensive or acknowledging gestures - for Wizard of Oz (WoZ) controlled open-ended dialogues in child-robot interactions. A controlled experiment with 26 children showed that these CFs can improve the perceived speediness, aliveness, humanness, and likability of the robot, without decreasing perceptions of intelligence, trustworthiness, or autonomy.

Personalisation can increase the learning efficacy of educational games by tailoring their content to the needs of the individual learner. This paper presents the Personalised Educational Game Architecture (PEGA). It uses a multi-agent organisation and an ontology to offer learners personalised training in a game environment. The multi-agent organisation's flexibility enables adaptive automation; the instructor can decide to control only parts of the training, while leaving the rest to the intelligent agents.

Experiencing stress during training is a way to prepare professionals for real-life crises. With the help of feedback tools, professionals can train to recognize and overcome negative effects of stress on task performances. This paper reports two studies that empirically examined the effect of such a feedback system. The system, based on the COgnitive Performance and Error (COPE) model, provides its users with physiological, predicted performance and predicted error-chance feedback. The first experiment focussed on creating stressful scenarios and establishing the parameters for the predictive models for the feedback system. Participants (n=9) performed fire-extinguishing tasks on a virtual ship. By altering time pressure, information uncertainty and consequences of performance, stress was induced. COPE variables were measured and models were established that predicted performance and the chances on specific errors. In the second experiment a new group of participants (n=29) carried out the same tasks while receiving eight different combinations of the three feedback types in a counterbalanced order. Performance scores improved when feedback was provided during the task. The number of errors made did not decrease. The usability score for the system with physiological feedback was significantly higher than a system without physiological feedback, unless combined with error feedback. This paper shows effects of feedback on performances and usability. To improve the effectiveness of the feedback system it is suggested to provide more in-depth tutorial sessions. Design changes are recommended that would make the feedback system more effective in improving performances.

Social robots have the potential to provide support in a number of practical domains, such as learning and behaviour change. This potential is particularly relevant for children, who have proven receptive to interactions with social robots. To reach learning and therapeutic goals, a number of issues need to be investigated, notably the design of an effective child-robot interaction (cHRI) to ensure the child remains engaged in the relationship and that educational goals are met. Typically, current cHRI research experiments focus on a single type of interaction activity (e.g. a game). However, these can suffer from a lack of adaptation to the child, or from an increasingly repetitive nature of the activity and interaction. In this paper, we motivate and propose a practicable solution to this issue: an adaptive robot able to switch between multiple activities within single interactions. We describe a system that embodies this idea, and present a case study in which diabetic children collaboratively learn with the robot about various aspects of managing their condition. We demonstrate the ability of our system to induce a varied interaction and show the potential of this approach both as an educational tool and as a research method for long-term cHRI.

A parameterized behavior model was developed for robots to show mood during task execution. In this study, we applied the model to the coverbal gestures of a robotic storyteller. This study investigated whether parameterized mood expression can 1) show mood that is changing over time; 2) reinforce affect communication when other modalities exist; 3) influence the mood induction process of the story; and 4) improve listeners' ratings of the storytelling experience and the robotic storyteller. We modulated the gestures to show either a congruent or an incongruent mood with the story mood. Results show that it is feasible to use parameterized coverbal gestures to express mood evolving over time and that participants can distinguish whether the mood expressed by the gestures is congruent or incongruent with the story mood. In terms of effects on participants we found that mood-modulated gestures (a) influence participants' mood, and (b) influence participants' ratings of the storytelling experience and the robotic storyteller.

To design good cooperative team members in robotics it is important to know what coordination mechanisms are required. Our approach to explore the need for a coordination mechanism is based on a systematic methodology to identify team coordination requirements. We show that a team combined of robots that each individually can solve a task not always is able to guarantee task completion as a team. In these cases some mechanism for coordination is required and we formally identify various problem classes that impose different requirements. We introduce a formal task model and distinguish between no, implicit and explicit coordination mechanisms. This model is used to study which mechanisms guarantee task completion. It allows us to prove some empirical findings reported in the literature such as that a simple foraging task does not require coordination.

The aim of our work is to design bodily mood expressions of humanoid robots for interactive settings that can be recognized by users and have (positive) effects on people who interact with the robots. To this end, we develop a parameterized behavior model for humanoid robots to express mood through body language. Different settings of the parameters, which control the spatial extent and motion dynamics of a behavior, result in different behavior appearances expressing different moods. In this study, we applied the behavior model to the gestures of the imitation game performed by the NAO robot to display either a positive or a negative mood. We address the question whether robot mood displayed simultaneously with the execution of functional behaviors in a task can (a) be recognized by participants and (b) produce contagion effects. Mood contagion is an automatic mechanism that induces a congruent mood state by means of the observation of another person’s emotional expression. In addition, we varied task difficulty to investigate how the task load mediates the effects. Our results show that participants are able to differentiate between positive and negative robot mood and they are able to recognize the behavioral cues (the parameters) we manipulated. Moreover, self-reported mood matches the mood expressed by the robot in the easy task condition. Additional evidence for mood contagion is provided by the fact that we were able to replicate an expected effect of negative mood on task performance: in the negative mood condition participants performed better on difficult tasks than in the positive mood condition, even though participants’ self-reported mood did not match that of the robot.

Children with diabetes can benefit from keeping a diary, but seldom keep one. Within the European ALIZ-E project a robot companion is being developed that, among other things, will be able to support and motivate diabetic children to keep a diary. This paper discusses the study of a robot supporting the use of an online diary. Diabetic children kept an online diary for two weeks, both with and without remote support from the robot via webcam. The effect of the robot was studied on children's use of the diary and their relationship with the robot. Results show that children shared significantly more personal experiences in their diaries when they were interacting with the robot. Furthermore, they greatly enjoyed working with the robot and came to see it as a helpful and supportive friend.

A field study was conducted in which CRI activities developed by the ALIZ-E project were tested with the project's primary user group: children with diabetes. This field study resulted in new insights in the modalities and roles a robot aimed at CRI in a healthcare setting might utilise, while in addition (re-)assessed some practises and technologies established within the project. Furthermore, it served as a means of strengthening the bonds with the project's principal stakeholders. The study illustrates on the one hand the feasibility of the activities that were developed within the project, while on the other hand highlights the importance of engaging with primary users in an ongoing, incremental fashion.

Expressive behaviour is a vital aspect of human interaction. A model for adaptive emotion expression was developed for the Nao robot. The robot has an internal arousal and valence value, which are influenced by the emotional state of its interaction partner and emotional occurrences such as winning a game. It expresses these emotions through its voice, posture, whole body poses, eye colour and gestures. An experiment with 18 children (mean age 9) and two Nao robots was conducted to study the influence of adaptive emotion expression on the interaction behaviour and opinions of children. In a within-subjects design the children played a quiz with both an affective robot using the model for adaptive emotion expression and a non-affective robot without this model. The affective robot reacted to the emotions of the child using the implementation of the model, the emotions of the child were interpreted by a Wizard of Oz. The dependent variables, namely the behaviour and opinions of the children, were measured through video analysis and questionnaires. The results show that children react more expressively and more positively to a robot which adaptively expresses itself than to a robot which does not. The feedback of the children in the questionnaires further suggests that showing emotion through movement is considered a very positive trait for a robot. From their positive reactions we can conclude that children enjoy interacting with a robot which adaptively expresses itself through emotion and gesture more than with a robot which does not do this.

This paper describes the new multimodal SWELL knowledge work (SWELL-KW) dataset for research on stress and user modeling. The dataset was collected in an experiment, in which 25 people performed typical knowledge work (writing reports, making presentations, reading e-mail, searching for information). We manipulated their working conditions with the stressors: email interruptions and time pressure. A varied set of data was recorded: computer logging, facial expression from camera recordings, body postures from a Kinect 3D sensor and heart rate (variability) and skin conductance from body sensors. The dataset made available not only contains raw data, but also preprocessed data and extracted features. The participants' subjective experience on task load, mental effort, emotion and perceived stress was assessed with validated questionnaires as a ground truth. The resulting dataset on working behavior and affect is a valuable contribution to several research fields, such as work psychology, user modeling and context aware systems.

This paper reports our investigation into the effects of bodily mood expression of a humanoid robot in a scenario close to real life. To this end, we used the NAO robot to perform as a lecturer in a university class. To display either a positive or a negative mood, we modulated 41 co-verbal gestures by adjusting behavior parameters that control spatial extent and motion dynamics, without modifying gesture function. Unique in this study is that (a) the robot gave an actual lecture to real students, (b) the interaction is one-to-many and relatively long (30 min), and (c) mood modulation was applied to a large set of behaviors. The robot presented the same lecture either in a positive or a negative mood to two audiences (between subjects). Although statistical analysis does not show that participants consciously recognized the robot mood, the results do show that participants in the positive mood condition rated their own arousal significantly higher than in the negative condition. Further, video annotation showed increased valence and arousal of the audience in the positive condition. Finally, participants' ratings of the lecturing quality and the gesture quality of the robot are higher in the positive condition, demonstrating the importance of robot mood expression in a one-to-many interaction setting.

Urban Search And Rescue (USAR) robots are used to find and save victims in the wake of disasters such as earthquakes or terrorist attacks. The operators of these robots are affected by high cognitive load; this hinders effective robot usage. This paper presents a cognitive task load model for real-time monitoring and, subsequently, balancing of workload on three factors that affect operator performance and mental effort: time occupied, level of information processing, and number of task switches. To test an implementation of the model, five participants drove a shape-shifting USAR robot, accumulating over 16 hours of driving time in the course of 485 USAR missions with varying objectives and difficulty. An accuracy of 69% was obtained for discrimination between low and high cognitive load; higher accuracy was measured for discrimination between extreme cognitive loads. This demonstrates that such a model can contribute, in a non-invasive manner, to estimating an operator's cognitive state. Several ways to further improve accuracy are discussed, based on additional experimental results.

Bodily expression of affect is crucial to human robot interaction. Our work aims at designing bodily expression of mood that does not interrupt ongoing functional behaviors. We propose a behavior model containing specific (pose and motion) parameters that characterize the behavior. Parameter modulation provides behavior variations through which affective behavioral cues can be integrated into behaviors. To investigate our model and parameter set, we applied our model to two concrete behaviors (waving and pointing) on a NAO robot, and conducted a user study in which participants (N=24) were asked to design such variations corresponding with positive, neutral, and negative moods. Preliminary results indicated that most parameters varied significantly with the mood variable. The results also suggest that the relative importance may be different between parameters, and parameters are probably interrelated. This paper presents the analysis of these aspects. The results show that the spatial extent parameters (hand-height and amplitude), the head vertical position, and the temporal parameter (motion-speed) are the most important parameters. Moreover, multiple parameters were found to be interrelated. These parameters should be modulated in combination to provide particular affective cues. These results suggest that a designer should focus on the design of the important behavior parameters and utilize the parameter combinations when designing mood expression.

Bodily expression of affect is crucial to human robot interaction. We distinguish between emotion and mood expression, and focus on mood expression. Bodily expression of an emotion is explicit behavior that typically interrupts ongoing functional behavior. Instead, bodily mood expression is integrated with functional behaviors without interrupting them. We propose a parameterized behavior model with specific behavior parameters for bodily mood expression. Robot mood controls pose and motion parameters, while those parameters modulate behavior appearance. We applied the model to two concrete behaviors - waving and pointing - of the NAO robot, and conducted a user study in which participants (N=24) were asked to design the expression of positive, neutral, and negative moods by modulating the parameters of the two behaviors. Results show that participants created different parameter settings corresponding with different moods, and the settings were generally consistent across participants. Various parameter settings were also found to be behavior-invariant. These findings suggest that our model and parameter set are promising for expressing moods in a variety of behaviors.

Currently, expressive virtual humans are used in psychological research, training, and psychotherapy. However, the behavior of these virtual humans is usually scripted and therefore cannot be modified freely at runtime. To address this, we created a virtual audience with parameterized behavioral styles. This paper presents a parameterized audience model based on probabilistic models abstracted from the observation of real human audiences (n = 16). The audience's behavioral style is controlled by model parameters that define virtual humans' moods, attitudes, and personalities. Employing these parameters as predictors, the audience model significantly predicts audience behavior. To investigate if people can recognize the designed behavioral styles generated by this model, 12 audience styles were evaluated by two groups of participants. One group (n = 22) was asked to describe the virtual audience freely, and the other group (n = 22) was asked to rate the audiences on eight dimensions. The results indicated that people could recognize different audience attitudes and even perceive the different degrees of certain audience attitudes. In conclusion, the audience model can generate expressive behavior to show different attitudes by modulating model parameters.

We describe and evaluate a new interface to induce positive emotions in users: a digital, interactive adaptive mirror. We study whether the induced affect is repeatable after a fixed interval (Study 1) and how copresence influences the emotion induction (Study 2). Results show that participants systematically feel more positive after an affective mirror session, that this effect is repeatable, and stronger when a friend is copresent.

This paper presents a Cognitive Task Load (CTL) model designed to keep track of an operator's mental workload, both quantitatively (amount of workload) and qualitatively (cognitive state). Every second, the CTL-model updates a diagnosis of the operator's cognitive state; by integrating this model in a (semi-)autonomous robot, the robot's level of automation and user interface can be attuned to the operator's state. The CTL-model's predictions were tested in an Urban Search And Rescue (USAR) setting. The test showed insufficient workload variations to validate the model. This indicates that participants should be subjected to more “high-pressure” conditions in future trials. These results also suggest that in a realistic environment, an operator's mental workload is affected by high-level coping strategies.

Children with a chronic disease like diabetes need to learn how to self manage their disease. Knowledge about their condition is indispensable to reach this goal. Within the European project ALIZ-E a robot companion is being developed that should, among others attributes, have the capability to educate children. In this paper, a virtual agent on a screen is compared with a physical robot on the aspects of performance (learning), attention and motivation. The experiment consisted of two sessions in which children played a quiz consisting of health related questions with both the robot and the virtual agent, there was a week between the two sessions. It was found that performance and motivation were not affected by the embodiment, but the robot did attract more attention and, when forced to choose, the children had a preference for the robot.

This paper presents a test battery method to investigate (relative) operational upsides and downsides of robot innovations. This method was applied to test the teleoperated Generaal robot and the NIFTi robot that could be operated in manual and semi-automatic mode for navigation and situation sensing. The unit tasks of our test battery proved to be discriminatory to assess differences between the robots. Most often the performance with the Generaal was faster than the semi-autonomous NIFTi robot, whereas the semi-autonomous NIFTi robot had sometimes less collisions. To lower workload for operators there is a clear need for a semi-autonomous robot, but the current version needs improvement on the planning and path-finding to better perform on speed.

Social robots can comfort and support children who have to cope with chronic diseases. In previous studies, a "facial robot", the iCat, proved to show well-recognized emotional expressions that are important in social interactions. The question is if a mobile robot without a face, the Nao, can express emotions with its body. First, dynamic body postures were created and validated that express fear, happiness, anger, sadness and surprise. Then, fourteen children had to recognize emotions, expressed by both robots. Recognition rates were relatively high (between 68% and 99% accuracy). Only for the emotion "sad", the recognition was better for the iCat (95%) compared to the Nao (68%). Providing context increased the number of correct recognitions. In a second session, the emotions were significantly better recognized than during the first session for both robots. In sum, we succeeded to design Nao emotions, which were well recognized and learned, and can be important ingredients of the social dialogs with children.

The ever-increasing involvement of computer technology in work and living environments—for training and actual task performances—sets continuously new challenges for cognitive ergonomics in diverse domains like transport, crisis management and healthcare (Brinkman, 2011). A major challenge is to harmonize the technology development to the dynamics and complexity of the social, cognitive and affective processes in these environments, taking into account of the diversity and multiplicity of human needs (cf. Klein et al., 2004). Such a harmonization comprises effective and efficient human-automation collaboration that proves (1) to be resilient for critical situations and (2) to facilitate creative problem solving in such situations. Current research focuses on collaborative artefacts that help to establish these two effects by enhancing work team’s conditions, knowledge and capabilities for acting in a safe and healthy way. For example, studies on flying, driving, and sailing provide requirements for pilot-automation collaboration that brings about adequate recover piloting behaviour after a “failure” (cf. Woods and Hollnagel, 2006; Lenior et al., 2006). As a second example, recent research on shared situation awareness of distributed teams in crisis management provide support that may improve team coordination and corresponding performance (van der Kleij et al., 2009). For the development of such human-automation collaboration, cognitive ergonomics methods are needed for deriving and testing of the “situational” requirements systematically (cf. Neerincx and Lindenberg, 2008). For example, game-based evaluations with Virtual-Reality tools can help to train for new situations or to test specific artefacts (Smets et al., 2011). Furthermore, a combination of scenario-based investigation and controlled lab experiments can, for example, help to study automated assistant functions to support therapists in high demand situations when treating multiple patients simultaneously over the internet (Paping et al., 2011). As a final example, user experience sampling methods that apply advanced interaction events analysis (Brinkman et al., 2005; Brinkman et al., 2007) or sensor technology can improve the insight into situated activities over time (such as heart-rate, eye tracking), but it might prove to be difficult to apply in high-demand environments (e.g. Grootjen et al., 2007).

A situation map that shows the overview of a disaster situation serves as a valuable tool for disaster response teams. It helps them to orientate their location and to make disaster response decisions. It is, however, a complicated task to rapidly generate a complete and comprehensive situation map of a disaster area, particularly due to the centralized organization of disaster management and the limited emergency services. In this study, we propose to let the affected population be utilized as an additional resource that can actively help to make such a situation map. The aim of this study was to investigate the possibility of constructing a shared situation map using a collaborative distributed mechanism. By examining earlier research, a detailed list of potential problems is identified in the collaborative map-making process. These problems were then addressed in an experiment which evaluated a number of proposed solutions. The results showed that more collaboration channels led to a situation map of better quality, and that including confidence information for objects and events in the map helped the discussion process during the map-making.

Space crews are in need for excellent cognitive support to perform nominal and off-nominal actions. This paper presents a coherent cognitive engineering methodology for the design of such support, which may be used to establish adequate usability, context-specific support that is integrated into astronaut’s task performance and/or electronic partners who enhance human–machine team’s resilience. It comprises (a) usability guidelines, measures and methods, (b) a general process guide that integrates task procedure design into user interface design and a software framework to implement such support and (c) theories, methods and tools to analyse, model and test future human–machine collaborations in space. In empirical studies, the knowledge base and tools for crew support are continuously being extended, refined and maintained.

Contains fulltext : 91351.pdf (author's version ) (Open Access)

Contains fulltext : 91352.pdf (Publisher’s version ) (Open Access)

Mobile information systems aid first responders in their tasks. Support is often based on mobile maps. People have different preferences for map orientations (heading-up or north-up), but map orientations also have different advantages and disadvantages. In general north-up maps are good for building up situation awareness and heading-up maps are better for navigational tasks. Because of heavily loaded visual modalities, we expect that tactile waypoint information can enhance navigation speed and situation awareness. In this paper we describe an experiment conducted in a synthetic task environment, in which we examined the effect of heading-up and north-up displays on search and rescue performance of first responders, and if adding the tactile display improves performance.

To increase police officer awareness of incident locations, the Dutch police developed and implemented a location-based notification system (LBNS). This mobile service notifies police officers proactively to warrants, agreements and police focal points in their current vicinity. To assess the accuracy, efficiency, effectiveness and user experience of this service, a longitudinal field evaluation was conducted with thirty police officers over four months. The results show that using the LBNS, police officers were better informed of relevant information in their environment and this led to positive operational results. Users considered the interface clear and easy to use. However, users indicated that the system presented too many or non-relevant notifications and that the system is overly complex. Recommendations for further development of the LBNS are to mitigate unwanted interruption by intelligent filtering of notifications and integration of system components.

The world population is getting older and more and more people suffer from a chronic disease such as diabetes. The need for medical (self-)care therefore increases, and we think a personal (robot) assistant could help. This paper gives guidelines for self-care that supports and shows how it could be incorporated in a (embodied) personal assistant. These guidelines were derived from motivational interviewing, persuasive technology, and from existing guidelines for personal assistants. Questions this paper addresses include: Is it possible to incorporate them in a personal assistant? Can a robot have the same kind of dialogs as a text interface? A first experiment, conducted with young participants, showed that the guidelines were best expressed in a socially intelligent iCat in comparison with a nonsocially intelligent iCat, a social and nonsocially intelligent Tiggie, and a text interface. Furthermore it showed that people indeed preferred the iCat over the text interface, possibly because the iCat's social intelligence

An analysis of context-aware user interfaces shows that adaptation mechanisms have a cost-benefit trade-off for usability. Unpredictable autonomous interface adaptations can easily reduce a system's usability. To reduce this negative effect of adaptive behaviour, we have attempted to help users building adequate mental models of such systems. A user support concept was developed and applied to a context-aware mobile device with an adaptive user interface. The approach was evaluated with users and as expected, the user support improved ease of use, but unexpectedly it reduced learnability. This shows that an increase of ease of use can be realised without actually improving the user's mental model of adaptive systems.

Current Network User Interfaces (NUIs) provide entrances to an enormous amount of Web-based services, bringing about new use problems such as laborious and unsuccessful navigation. Such problems are generally more severe for users with regression of cognitive functions (e.g. for some elderly). This paper identifies four fundamental cognitive determinants of navigation performance that may explain these problems: situation awareness, spatial ability, task-set switching and user control of support. Based on an analysis of these demands and current support functions for navigation, three “refined” support concepts were developed: categorising landmarks, history map and navigation assistant. Via the specification of humancomputer co-operative processes and scenarios, the concepts were implemented for two rather different web-based services. The present paper provides an example implementation of the navigation assistant. The results of the study will feed into a cognitive engineering method for the design of NUIs.

The idea of aiding as cognitive support is to offer the user the knowledge he or she is missing. Recently, we developed a design method for aiding that is based on explicit requirements of the human problem solver. This proved to be able to supplement a lack of human knowledge in a statistical analysis task. In this article we extend the aiding concept to time-pressured tasks and we investigate whether aiding can supplement lack of knowledge and capacity under tasks with high mental loading, such as dealing with irregularities in process control. We developed a simulator of the workplace of a railway traffic controller with an aiding function for dealing with irregularities (e.g., a switch getting out of order). Application of the design method proved to be possible for this task. We then conducted an experiment to study effects of the aiding on task performance, mental effort, and learning under low and high task load conditions. Users of the simulator dealt better and faster with irregularities when the computer provided aiding. The higher the task load was, the larger this beneficial effect was. For theory about human-computer interaction, this research points to possible positive effects of aiding on performance and learning as a consequence of reducing cognitive demands.