Abstract Handling emergencies requires efficient and effective collaboration of medical professionals. To analyze their performance, in an application study, we have developed VisCoMET, a visual analytics approach displaying interactions of healthcare personnel in a triage training of a mass casualty incident. The application scenario stems from social interaction research, where the collaboration of teams is studied from different perspectives. We integrate recorded annotations from multiple sources, such as recorded videos of the sessions, transcribed communication, and eye‐tracking information. For each session, an information‐rich timeline visualizes events across these different channels, specifically highlighting interactions between the team members. We provide algorithmic support to identify frequent event patterns and to search for user‐defined event sequences. Comparing different teams, an overview visualization aggregates each training session in a visual glyph as a node, connected to similar sessions through edges. An application example shows the usage of the approach in the comparative analysis of triage training sessions, where multiple teams encountered the same scene, and highlights discovered insights. The approach was evaluated through feedback from visualization and social interaction experts. The results show that the approach supports reflecting on teams' performance by exploratory analysis of collaboration behavior while particularly enabling the comparison of triage training sessions.

Based on a video corpus and logfiles of HRI in a museum guide scenario the paper investigates a model of a stepwise process for opening a focused encounter. Towards developing an autonomous system, a Wizard of Oz set-up is used to investigate strategies for dealing with a system’s time delay in the ‘pre-beginning’. Initial Analysis shows that the wizard orients to the visitor’s head orientation for proceeding to next steps.

This paper investigates how the pre-clinical ‘triage’infieldtrainings of mass casualty incidents is realized as a collaborative, situated practice. Whilst triage algorithms are mainly presented as an individualistic decision process, we show –using video data –how theprofessionals’ practicesare based on mutual monitoring and interactional coordinationand thus provide for reflecting on decisions and preparing for next actions.

This paper investigates how situations of medical emergency (CPR) are organized as collaborative practice. Whilst studies have assessed CPR mainly with regard to timing of medical actions, we suggest that interactional micro-coordination between the participants and their involvement in multiple activities may contribute to such timing.

Based on two contrasting case analyses we describe practices of handling long utterances, i.e. turn increments produced by users. The strategies to handle long utterances are executed by a human wizard in the context of a speech-based assistive system and reveal specific characteristics of human-human-interaction like precise timing that are not easily implemented into a technical system. The preliminary analysis shows two basic strategies of handling long utterances:1) to interrupt and 2) to wait-and-see. But the fine-grained analysis of the participants' perspective shows yet another dimension when evaluating forms of handling turn increments:depending on whether user input was taken up and so, the initiated action is continued, even interruptions (at uncommon points in interaction) are socially accepted by users. In contrast, the socially 'safe' strategy to wait-and-see might even cause trouble, when the jointly worked on task is not continued by the system.

Dialog modeling of making suggestions in human-agent interaction is a challenge due to the socially delicate nature of a suggestion and ensuing interactional negotiations. A basic first dialog model for making suggestions was tested in the context of schedule management assistance by an embodied conversational agent with elderly and mildly cognitively impaired persons. Analysis showed that users responded according to human social structures with most response types bearing potential challenges concerning the system's language understanding and the users' intention interpretation:next to explicit answers, users produced implicit versions for acceptance or resistance and further requests for information or modifications. Thus, an enhanced dialog model with a newly added clarification sequence and a new multi-conditional entry sequence was tested in a second study with the autonomous system. Initial observations show a promising performance of the dialog model.

The paper explores the interactional dynamics within groups of visitors (children & adults) who are interacting with an autonomous museum guide robot. Using the example of question-answer sequences, analysis reveals that 'answering' tends to be a process of interactional negotiation at the end of which a 'result' is addressed to the robot. This process, however, is often not transparent to the robot and considered solely as "silence" which leads to the disregard of the users' attempts to provide an answer.

On the basis of a video corpus of human-robot-interaction in a museum guide scenario, we address the challenge how a robot could determine the appropriate moment of opening an interaction with a visitor. Considering such "openings" as a stepwise coordinated process, we used a wizard-of-oz setup to investigate the wizard's timely decisions about when to look at the visitor resp. to verbally greet him/her (during early stages called "pre-beginnings"). Analysis is based on both robot internal and external data and combines qualitative sequential case analysis and quantification. Analysis reveals features for classification of two visitors types and corresponding opening strategies.

Based on a user study, we start from the observation that `long turns' uttered by users towards an assistive system constitute a challenge for the dialog management of a voice-operated system. Assuming an interactional perspective, we address the question as to how `long turns' emerge in interaction. We suggest to conceive of these utterances as being co-constructed by both, the user and the multimodal conduct of the technical system. In this paper, we examine how such `long turns' emerge step by step in terms of an initial utterance being expanded by so-called `increments' as well as their specific structure. Analysis shows that such utterance expansions (causing `long turns') indicate that the user faces problems with a lack of uptake resp. display of recipiency by the technical system. Combining qualitative micro-analysis with quantification, we discuss specific interactional contexts of turn increments, different actions performed by them and the role of uptake resources in the light of designing autonomous speech-based systems.

In speech based technical systems, a `reprompt' can be deployed as a verbally non-explicit and semantically unspeciflc practice of making a failure-to-understand transparent. Users' repeats or rephrasings of their previous answers might lead to further non-understandings, resulting in further reprompts by the system. On the basis of a Wizard-of-Oz video corpus in a schedule management setting with an embodied conversational agent and the special user groups of elderly and mildly cognitively impaired persons, we investigate in a conversation analytical approach the interactional impact of three-fold reprompts on subsequent user actions to an appointment suggestion. We focus especially on the type of user actions during the course of multiple reprompts in a confirmation/disconfirmation context. Analysis reveals more fine-grained user response types, testifying that all users ratify the first reprompt. After the second and third one, users tend to either add problem manifestations or initiations of the relevant next move. Or they substitute their previous answer by these types of actions. While additional or substituting problem manifestations call for more specific and linguistically restricting error handling practices, the user-initiated next moves are technically exploitable as implicit cues for confirmation in the presented special yes/no-context.

An autonomous robot system was equipped with basic means to monitor the users' success/failure in following a robot's verbal-gestural deictic reference to an object and - in case of problems - to provide additional help, i.e. to suggest a `repair' action. A real-world field trial with the robot acting as museum guide constitutes the basis for analysis of the users' reactions to the first reference and the subsequent `repair' action in two structurally different conditions. Results indicate that deictic `repair' actions are mostly successful helping users to correctly orient, but they may also lead to confusion for initially correctly oriented visitors in situations of small groups of users. A recurring user practice is revealed which consists of displaying to co-visitors their understanding of the reference by pointing to the location and thus providing additional orientational help.

On the basis of a videocorpus of human-robot-interactions in a museum guide scenario and with a combined approach of qualitative case analysis and quantitative evaluation we examine two research questions: (1) What type of group dynamics result from a robot's unexpected next move - especially focussing on gaze patterns? (2) What should a robot detect when searching for trouble events and how should it identify patterns of visitor behavior that are related to the robots actions? Systematically studying opening sequences, we extract structural gaze patterns out of two cases considered from an interactional linguistics perspective as well as keeping in mind a system's capabilities of perception.

Once a museum guide robot is deployed in the real world, it is faced with the difficulty to deal with user utterances within a noisy environment. We present explorative analysis of the ways in which visitors signal trouble on a multimodal level when they have difficulties in understanding the robot. Such moments of trouble are investigated in the context of robot-to-group situations. Analysis will give insights into the visitors conduct at moments of trouble in robot-to-single-user and robot-to-group interactions. In a limited data set, we found that members of a group show internal gaze coordination between each other when trouble occurs due to the robot's inadequate behavior. Implications will be drawn for further developing the interactional capabilities of an autonomous robot system to be attentive and responsive to the users' needs.

A basic task for robots interacting with humans consists in guiding their focus of attention. Existing guidelines for a robot's multimodal deixis are primarily focused on the speaker (talk-gesture-coordination, handshape). Conducting a field trial with a museum guide robot, we tested these individualistic referential strategies in the dynamic conditions of real-world HRI and found that their success ranges between 27% and 95%. Qualitative video-based micro-analysis revealed that the users experienced problems when they were not facing the robot at the moment of the deictic gesture. Also the importance of the robot's head orientation became evident. Implications are drawn as design guidelines for an interactional account of modeling referential strategies for HRI.

The topic of motivation is a crucial issue for various human-robot interaction (HRI) scenarios. Interactional aspects of motivation can be studied in human-human interaction (HHI) and build the basis for modeling a robot's interactional conduct. Using an ethnographic approach we explored the factors relevant in the formation of motivation-relevant processes in an indoor-cycling activity. We propose an interactive, action-based motivation model for HRI that has been implemented in an autonomous robot system and tested during a long-term HRI study. The model is based on micro-analyses of human indoor cycling courses and resulted in an adaption of specific dialog patterns for HRI. A qualitative evaluation - accompanied by a quantitative analysis - demonstrated that the transfer of interaction patterns from HHI to HRI was successful with participants benefitting from the interaction experience (e.g., performance, subjective feeling of being motivated).

Multimodal research in human interaction has to consider a variety of factors, ranging from local short-time phenomena to complex interaction patterns. As of today, no single discipline engaged in communication research offers the methods and tools to investigate the full complexity continuum in a time-efficient way. A synthesis of qualitative and quantitative analysis is required to merge insights about micro-sequential structures with big data patterns. Using the example of a co-present dyadic negotiation analysis to combine methods offered by Conversation Analysis and Data Mining, we show how such a partnership can benefit each discipline and lead to insights as well as new hypotheses evaluation opportunities.

In order to understand and model the non-verbal communicative conduct of humans, it seems fruitful to combine qualitative (Conversation Analysis [6] [10] [11]) and quantitative analytical (motion capturing) methods. Tools for data visualization and annotation are important as they constitute a central interface between different research approaches and methodologies. With this aim we have developed the pre-annotation tool "PAMOCAT - Pre Annotation Motion Capture Analysis Tool" that detects different phenomena. These phenomena are in the category single person and person overlapping phenomena. Included are functions for the analysis of head focused objects, hand activities, single DOF- degree of freedom activity, posture detection and intrusions into the co-participant's space. These detected phenomena related to the frames will be displayed in an overview. The phenomena can be chosen to search for a specific constellation between these different phenomena. A sophisticated user interface easily allows the annotating person to find correlations between different joints and phenomena, to analyze the corresponding 3D pose in a reconstructed virtual environment, and to export combined qualitative and quantitative annotations to standard annotation tools. Using this technique we are able to examine complex setups with three participants engaged in conversation. In this paper we propose how PAMOCAT can be integrated in the research cycle by showing a concrete PAMOCAT-based micro-analysis of a multimodal phenomenon, which deals with kinetic procedures to claim the floor.

The paper investigates the effects of a robot's “on-line” feedback during a tutoring situation with a human tutor. Analysis is based on a study conducted with an iCub robot that autonomously generates its feedback (gaze, pointing gesture) based on the system's perception of the tutor's actions using the idea of reciprocity of actions. Sequential micro-analysis of two opposite cases reveals how the robot's behavior (responsive vs. non-responsive) pro-actively shapes the tutor's conduct and thus co-produces the way in which it is being tutored. A dialogic and a monologic tutoring style are distinguished. The first 20 seconds of an encounter are found to shape the user's perception and expectations of the system's competences and lead to a relatively stable tutoring style even if the robot's reactivity and appropriateness of feedback changes.

Users draw on four sources to judge a robot's competence: (1) the robot's voice, (2) physical appearance of and (3) the interaction experience with the robot but also (4) the relationship between the robot's physical appearance and its conduct. Furthermore, most approaches in social robotics have an outcome-oriented focus and thus use questionnaires to measure a global evaluation of the robot after interaction took place. The present research takes a process-oriented approach to explore the factors relevant in the formation of users' attitudes toward the robot. To do so, an ethnographic approach (Conversation Analysis) was employed to analyze the micro-coordination between user and robot. We report initial findings from a study in which a robot took the role of a fitness instructor. Our results emphasize that the participant judges step-by-step the robot's capabilities and differentiates its competence on two levels regarding to the robot's role: a robot as a (1) social/interactional co-participant and as a (2) fitness instructor.

As a testbed for real-world experimentation on HRI and dynamic interaction models, this paper presents an autonomous robot system acting as guide in a German arts museum. The visitors' evaluation of this system is analyzed using a questionnaire and reveals issues for subsequent analysis of the real-time interaction.

Designing technologies that support the explanation of museum exhibits is a challenging domain. In this paper we develop an innovative approach - providing a robot guide with resources to engage visitors in an interaction about an art exhibit. We draw upon ethnographical fieldwork in an art museum, focusing on how tour guides interrelate talk and visual conduct, specifically how they ask questions of different kinds to engage and involve visitors in lengthy explanations of an exhibit. From this analysis we have developed a robot guide that can coordinate its utterances and body movement to monitor the responses of visitors to these. Detailed analysis of the interaction between the robot and visitors in an art museum suggests that such simple devices derived from the study of human interaction might be useful in engaging visitors in explanations of complex artifacts.

If robot systems are being deployed in real world settings with untrained users who happen to accidentally pass by or could leave at any moment in time, then this places specific demands on the robot system: it needs to secure and maintain the user's engagement. In this, a common and critical problem consists of entering into a `focused encounter'. It requires each interactional partner to closely react upon the other's actions on a very fine-grained level engaging in a stepwise and dynamic process of mutual adjustments. We report initial findings from a study in which we have developed a preliminary, simple solution to this problem inspired by work from Conversation Analysis. Using this as an instrument to explore the impact of a `contingent' (CE) vs. `non-contingent entry' (NCE), we find that users who enter into the interaction in a dynamic and contingent manner show a significantly different way of interacting with the robot than the NCE group.

In this paper we consider the development of a museum guide robot that has both autonomous and remotely controlled features. We focus on the capabilities such a robot could have to help focus the attention of a visitor on an object or artefact. Inspired by studies of social interaction, which investigate whether the robot could deploy "restarts" and "pauses" at certain moments in its talk to first elicit the visitor's attention/gaze towards the robot. We report an experiment where we deployed such a robot to interact with real visitors to a science museum. These experiments show that such a strategy does seem to have a significant impact on obtaining the visitor's gaze.