Methods for understanding the felt and situated experience of controlling autonomous systems are crucial for designing systems that are adjusted to the complexity of human interaction. Prior work has tended to overlook the bodily and experiential dimensions of monitoring systems at a distance, particularly in moments of losing control. We combined two approaches, ethnomethodology and conversation analysis, and soma design, to explore a case where a semi-autonomous system crashed when controlled by an inexperienced operator, captured in video ethnographic fieldwork. We report on our methodological approach, combining sequential video analysis of the unfolding sequence and interviews inspired by microphenomenology to unpack the operator’s experience of losing control. We contribute methodological considerations for interaction designers seeking to explore the felt experience of having and losing control of autonomous systems and discuss how insights gained through this combination of methods, rooted in phenomenology, support a designerly appreciation of safety operators’ work.

Using non-expert language feedback in learning is crucial to making robots successful in human-centered environments. While language feedback has shown potential to teach robots complex tasks, it also brings challenges: humans leave important context, detail, and clarifications unspoken, making interactive approaches necessary to use the feedback effectively. In this work, we develop an interactive robot learning system that can ask clarifying questions to differentiate between hard and soft task constraints from user verbal feedback. The system uses feedback as either shields (hard constraints) or to shape the reward (soft constraints). We conducted a user study with 24 participants, comparing the use of both hard and soft constraints versus two baseline conditions. We show that participants significantly prefer a system using a combination of both hard and soft constraints, or only using soft constraints, compared to a system using only hard constraints. Qualitative analysis of the participants' interactions with the system revealed common feedback types: spatial, temporal and meta-level. To evaluate the learning performance of the system, we conducted simulated experiments showing that combining both hard and soft constraints performs best in terms of reaching high rewards and finding an efficient solution. Additionally, we provide demonstrations of our system on real robot hardware.

In this paper, we aim to explore how humans give unstructured free-form natural language feedback towards correcting robot task policies. We present a qualitative study based on crowd-sourced feedback from 66 participants. Participants give feedback on the execution of three robotic tasks in the form of mobile navigation, dexterous object manipulation and a robot arm opening a door. We find through reflexive thematic analysis which features participants reference most and what other patterns are present in the feedback, including that responders do not naturally give concrete and actionable feedback. We attribute the lack of feedback concreteness to a lack of engagement with robot behavior and false assumptions about who is receiving the feedback and how much knowledge participants have. The study presents a step towards better understanding unstructured language feedback for robotic learning.