Methods for understanding the felt and situated experience of controlling autonomous systems are crucial for designing systems that are adjusted to the complexity of human interaction. Prior work has tended to overlook the bodily and experiential dimensions of monitoring systems at a distance, particularly in moments of losing control. We combined two approaches, ethnomethodology and conversation analysis, and soma design, to explore a case where a semi-autonomous system crashed when controlled by an inexperienced operator, captured in video ethnographic fieldwork. We report on our methodological approach, combining sequential video analysis of the unfolding sequence and interviews inspired by microphenomenology to unpack the operator’s experience of losing control. We contribute methodological considerations for interaction designers seeking to explore the felt experience of having and losing control of autonomous systems and discuss how insights gained through this combination of methods, rooted in phenomenology, support a designerly appreciation of safety operators’ work.

The proliferation of robots in public spaces necessitates a deeper understanding of how these robots can interact with those they share the space with. In this paper, we present findings from video analysis of publicly deployed cleaning robots in a transit space—a major commercial airport, using their navigational “troubles” as a tool to document what robots currently lack in interactional competence. We demonstrate that these robots, while technically proficient, can disrupt the social order of a space due to their inability to understand core aspects of human movement: mutual adjustment to others, the significance of understanding social groups, and the purpose of different locations. In discussion we argue for exploring a new design space of movement: socially-aware movement. By developing “strong concepts” that treat movement as an interactional and collaborative accomplishment, we can create systems that better integrate into the everyday rhythms of public life.

This paper analyzes honks directed at autonomous vehicles (AVs) by other drivers. As honks often mark problems, this focus allows us to better understand the challenges that AVs face in real traffic. Performing a sequential video analysis of 63 honk incidents uploaded by Tesla beta testers on YouTube, we identify how problematic situations emerge as honkable Traffic Gestalts. We identify four types of situated problems with AV driving performance marked by other drivers’ honks: they may wait too long, steer inconsistently, stop instead of going, and go too fast. We further show how a honk may be understandable as a warning, a nudge or a reprimand. Our work suggests designing honks for AVs to focus on relevant contexts, supported by developing bidirectional interfaces and audio analysis methods that consider the interplay of auditory and visual information in traffic.

Using publicly uploaded videos of the Waymo and Tesla FSD self-driving cars, this paper documents how self-driving vehicles still struggle with some basics of road interaction. To drive safely self-driving cars need to interact in traffic with other road users. Yet traffic is a complex, long established social domain. We focus on one core element of road interaction: when road users yield for each other. Yielding – such as by slowing down for others in traffic – involves communication between different road users to decide who will ‘go’ and who will ‘yield’. Videos of the Waymo and Tesla FSD self-driving cars show how these systems fail to both yield for others, as well as failing to go when yielded to. In discussion, we explore how these ‘problems’ illustrate both the complexity of designing for road interaction, but also how the space of physical machine/human social interactions more broadly can be designed for.

This paper explores how humans interpret displays of emotion produced by a social robot in real world situated interaction. Taking a multimodal conversation analytic approach, we analyze video data of families interacting with a Cozmo robot in their homes. Focusing on one happy and one sad robot animation, we study, on a turn-by-turn basis, how participants respond to audible and visible robot behavior designed to display emotion. We show how emotion animations are consequential for interactional progressivity: While displays of happiness typically move the interaction forward, displays of sadness regularly lead to a reconsideration of previous actions by humans. Furthermore, in making sense of the robot animations people may move beyond the designer's reported intentions, actually broadening the opportunities for their subsequent engagement. We discuss how sadness functions as an interactional "rewind button" and how the inherent vagueness of emotion displays can be deployed in design.

This paper explores how humans adapt to a conventional humanoid robot. Video data of participants playing a charade game with a Nao robot were analyzed from a multimodal conversation analysis perspective. Participants soon adjust aspects of turn-design such as word selection, turn length and prosody, thereby adapting to the robot's limited perceptive abilities as they become apparent in the interaction. However, coordination of turns-at-talk remains troublesome throughout the encounter, as evidenced by overlapping turns and lengthy silences around possible turn endings. The study discusses how the robot design can be improved to support the problematic taking of turns-at-talk with humans. Two programming strategies to address the identified problems are presented: 1. to program the robot so that it will be systematically receptive at the equivalence to transition relevance places in human-human interaction, and 2. to make the robot preferably produce verbal actions that require a response in a conditional way, rather than making a response only possible.

In this paper we explore the production of streaming media that involves live and recorded content. To examine this, we report on how the production practices and process are conducted through an empirical study of the production of live television, involving the use of live and non-live media under highly time critical conditions. In explaining how this process is managed both as an individual and collective activity, we develop the concept of temporal hybridity to explain the properties of these kinds of production system and show how temporally separated media are used, understood and coordinated. Our analysis is examined in the light of recent developments in computing technology and we present some design implications to support amateur video production.