Validating claims and replicating findings on the impact of artificial social agents (ASA), such as virtual agents, conversational agents, and social robots, requires a standardised measurement instrument that researchers can employ in different settings and for various agents. Such an instrument would allow researchers to evaluate their agents and establish insights beyond their specific study context. Therefore, we present the long and short versions of the ASA questionnaire (ASAQ) for evaluating human-ASA interaction on 19 constructs, such as the agent’s believability, sociability, and coherence. It has been developed by an international workgroup with more than 100 ASA-researchers over multiple years who identified community-relevant constructs and associated questionnaire items and examined the questionnaire’s reliability, validity, and interpretability. The result is a questionnaire that can capture more than 80% of the constructs that studies in the intelligent virtual agent community investigate, with acceptable levels of reliability, content validity, construct validity, and cross-validity. We suggest that ASA-researchers use the ASAQ short version to report their agent’s psychographic information and the ASAQ long version to analyse any constructs in-depth that are specifically relevant to their agent or study. Finally, this paper gives instructions for practical use, such as sample size estimations, and how to interpret and present results. • The Artificial-Social-Agent Questionnaire (ASAQ) is a validated measure used to evaluate the human experience of interacting with an artificial social agent (ASA). • There are two versions of the ASAQ. The short version is used to report the ASAs psychographic information, while the long version allows for in-depth analysis of constructs relevant to a specific ASA. • More than 100 ASA researchers were involved in the development of ASAQ. • The 19 ASAQ constructs capture over 80% of the constructs investigated in the Intelligent Virtual Agent community between 2013-2018. • The ASAQ has demonstrated acceptable levels of reliability, content validity, construct validity, and cross-validity. • We present the ASAQ representative set 2024 of 29 agents. • Instructions for the practical use of ASAQ are given, including guidance on choosing the ASAQ version, estimating sample sizes, and interpreting and presenting the results (e.g., ASAQ chart).

We present the ASA Questionnaire, an instrument for evaluating human interaction with an artificial social agent (ASA), resulting from multi-year efforts involving more than 100 Intelligent Virtual Agent (IVA) researchers worldwide. It has 19 measurement constructs constituted by 90 items, which capture more than 80% of the constructs identified in empirical studies published in the IVA conference 2013--2018. This paper reports on construct validity analysis, specifically convergent and discriminant validity of initial 131 instrument items that involved 532 crowd-workers who were asked to rate human interaction with 14 different ASAs. The analysis included several factor analysis models and resulted in the selection of 90 items for inclusion in the long version of the ASA questionnaire. In addition, a representative item of each construct or dimension was selected to create a 24-item short version of the ASA questionnaire. Whereas the long version is suitable for a comprehensive evaluation of human-ASA interaction, the short version allows quick analysis and description of the interaction with the ASA. To support reporting ASA questionnaire results, we also put forward an ASA chart. The chart provides a quick overview of the agent profile.

In this paper, we report on the multi-year Intelligent Virtual Agents (IVA) community effort, involving more than 90 researchers worldwide, researching the IVA community interests and practice in evaluating human interaction with an artificial social agent (ASA). The joint efforts have previously generated a unified set of 19 constructs that capture more than 80% of constructs used in empirical studies published in the IVA conference between 2013 to 2018. In this paper, we present expert-content-validated 131 questionnaire items for the constructs and their dimensions, and investigate the level of reliability. We establish this in three phases. Firstly, eight experts generated 431 potential construct items. Secondly, 20 experts rated whether items measure (only) their intended construct, resulting in 207 content-validated items. Next, a reliability analysis was conducted, involving 192 crowd-workers who were asked to rate a human interaction with an ASA, which resulted in 131 items (about 5 items per measurement, with Cronbach's alpha ranged [.60 -- .87]). These are the starting points for the questionnaire instrument of human-ASA interaction.

In this paper, we report on the multi-year Intelligent Virtual Agents (IVA) community effort, involving more than 80 researchers worldwide, researching the IVA community interests and practises in evaluating human interaction with an artificial social agent (ASA). The effort is driven by previous IVA workshops and plenary IVA discussions related to the methodological crisis on the evaluation of ASAs. A previous literature review showed a continuous practise of creating new questionnaires instead of reusing validated questionnaires. We address this issue by examining questionnaire measurement constructs used in empirical studies between 2013 to 2018 published in the IVA conference. We identified 189 constructs used in 89 questionnaires that are reported across 81 studies. Although these constructs have different names, they often measure the same thing. In this paper, we, therefore, present a unifying set of 19 constructs that captures more than 80% of the 189 constructs initially identified. We established this set in two steps. First, 49 researchers classified the constructs in broad theoretically based categories. Next, 23 researchers grouped the constructs in each category on their similarity. The resulting 19 groups form a unifying set of constructs, which will be the basis for the future questionnaire instrument of human-ASA interaction.

Research into artificial social agents aims at constructing these agents and at establishing an empirically grounded understanding of them, their interaction with humans, and how they can ultimately deliver certain outcomes in areas such as health, entertainment, and education. Key for establishing such understanding is the community's ability to describe and replicate their observations on how users perceive and interact with their agents. In this paper, we address this ability by examining questionnaires and their constructs used in empirical studies reported in the intelligent virtual agent conference proceedings from 2013 to 2018. The literature survey shows the identification of 189 constructs used in 89 questionnaires that were reported across 81 papers. We found unexpectedly little repeated use of questionnaires as the vast majority of questionnaires (more than 76%) were only reported in a single paper. We expect that this finding will motivate joint effort by the IVA community towards creating a unified measurement instrument.

Within the affective computing and social signal processing communities, increasing efforts are being made in order to collect data with genuine (emotional) content. When it comes to negative emotions and even aggression, ethical and privacy related issues prevent the usage of many emotion elicitation methods, and most often actors are employed to act out different scenarios. Moreover, for most databases, emotional arousal is not explicitly checked, and the footage is annotated by external raters based on observable behavior. In the attempt to gather data a step closer to real-life, previous work proposed an elicitation method for collecting the database of negative affect and aggression that involved unscripted role-plays between aggression regulation training actors (actors) and naive participants (students), where only short role descriptions and goals are given to the participants. In this paper we present a validation study for the database of negative affect and aggression by investigating whether the actors' behavior (e.g. becoming more aggressive) had a real impact on the students' emotional arousal. We found significant changes in the students' heart rate variability (HRV) parameters corresponding to changes in aggression level and emotional states of the actors, and therefore conclude that this method can be considered as a good candidate for emotion elicitation.

In this paper, a user interface paradigm, called Talk-and-Tools, is presented for automated e-coaching. The paradigm is based on the idea that people interact in two ways with their environment: symbolically and physically. The main goal is to show how the paradigm can be applied in the design of interactive systems that offer an acceptable coaching process. As a proof of concept, an e-coaching system is implemented that supports an insomnia therapy on a smartphone. A human coach was replaced by a cooperative virtual coach that is able to interact with a human coachee. In the interface of the system, we distinguish between a set of personalized conversations (“Talk”) and specialized modules that form a coherent structure of input and output facilities (“Tools”). Conversations contained a minimum of variation to exclude unpredictable behavior but included the necessary mechanisms for variation to offer personalized consults and support. A variety of system and user tests was conducted to validate the use of the system. After a 6-week therapy, some users spontaneously reported the experience of building a relationship with the e-coach. It is concluded that the addition of a conversational component fills an important gap in the design of current mobile systems.

Knowing the situational information about the current world is the precondition for setting goals and domains of actions. Special events in the world can also be used to initiate new actions or interrupt ongoing processes. A prototype of a communication interface has been developed which enables users to create visual messages to represent concepts or ideas in mind. The messages are constructed using a spatial arrangement of visual symbols. The interface has been tested by observers in a simulated crisis situation. The system processes the incoming message to build a world model by the employment of ontology. A blackboard of a dynamic ad-hoc network shares the knowledge and updates the display of the mobile devices of the users within the same region of the crisis event.

To reduce the ambiguity and the different semantic interpretation of human observers' reports, we propose a new paradigm in collaborating information using icons to represent concepts or ideas. Two prototypes of icon-based communication interfaces were developed with which users can create iconic messages to report situations where a crisis event occurs. The interfaces interpret and convert the messages to human natural language. A context-aware collaborative information system filters irrelevant and infidelity reports and shares the results for further decision making

Augmented tabletops can be used to create multi-modal and collaborative environments in which natural interactions with tangible objects that represent virtual (digital) information can be performed. Such environments are considered potentially interesting for many different applications. In this paper, we address the question of whether or not it makes sense to use such environments to design learning experiences for young children. More specifically, we present the "Read-It" application that we have created to illustrate how augmented tabletops can support the development of reading skills. Children of five-to-seven-years old were actively involved in designing and testing this application. A pilot experiment was conducted with a prototype of the Read-It application, in order to confirm that it does indeed meet the a priori expectations. We hope that the Read-It application will inspire the development of more tabletop applications that are targeted at specific user groups and activities.