We introduce PASTA (Perceptual Assessment System for explanaTion of Artificial Intelligence), a novel human-centric framework for evaluating eXplainable AI (XAI) techniques in computer vision. Our first contribution is the creation of the PASTA-dataset, the first large-scale benchmark that spans a diverse set of models and both saliency-based and concept-based explanation methods. This dataset enables robust, comparative analysis of XAI techniques based on human judgment. Our second contribution is an automated, data-driven benchmark that predicts human preferences using the PASTA-dataset. This scoring called PASTA-score method offers scalable, reliable, and consistent evaluation aligned with human perception. Additionally, our benchmark allows for comparisons between explanations across different modalities, an aspect previously unaddressed. We then propose to apply our scoring method to probe the interpretability of existing models and to build more human interpretable XAI methods.

Recent advances in Artificial Intelligence have enabled powerful generative models, yet few are tailored to dancers’ practices. We present a long-term collaboration with a Voguing and Dancehall collective to design movement generation models trained on their repertoire. Our initial study with the dancers revealed that, despite limited physical realism, the generated movements inspired them. Iterative development led to Korai, an interactive tool for monitoring training, visualizing motion data, and prompting generation, which improved output quality. A subsequent structured observation study compared three model variants with high, medium, and low fidelity to the original dataset’s style. Results show that dancers favored either highly faithful or highly unfaithful outputs, rejecting medium fidelity as neither authentic to their style nor creatively stimulating. Our findings highlight how direct collaboration with dancers not only informs model design but also deepens understanding of AI’s role in supporting creative movement practices.

Non-experts increasingly engage in user-driven algorithm auditing, interacting directly with AI systems to probe, document, and reflect on biased behavior. Yet, auditing remains challenging due to model opacity and limited support for navigating and interpreting outputs. This paper explores the design and evaluation of interfaces grounded in the sensemaking framework to support non-experts in auditing gender bias in image captioning. In a between-subjects study, 60 participants audited an image captioning model using one of three interface conditions: a Baseline interface, a Masking Tool for image manipulation, or a Filtering Tool for organizing captions. Our findings show that interface design shaped what participants noticed, how they interpreted model behavior, and supported their hypotheses. The Image Masking Tool enabled fine-grained testing of visual cues and context, while the Text Filtering Tool revealed broader asymmetries in gendered language. We argue that incorporating sensemaking into auditing practices can advance accountability and transparency in machine learning systems.

In the mid-2010s, media artists began developing practices using machine learning (ML) as an artistic medium. Since 2022, the rise of large generative models, the mainstreaming of AI as consumer products, and intensifying ethical disputes have reconfigured the conditions of their artistic practice. This paper aims to understand how artists working with ML over the past decade respond to these shifts, shedding light on how practices, tools, and culture co-evolve. We address this question through thematic analysis of semi-structured interviews with 30 artists active before 2020. Our findings show how artists experience narrowing aesthetics and reduced malleability of post-2020 ML systems, have diverging views on where to locate moral responsibility with large AI models, and face shifting cultural reception that challenges the legibility of their work. We map how artists envision their practice going forward and discuss those orientations with respect to HCI conversations on design and creativity.

Transformer-based ranking models, such as MonoBERT, are central to Information Retrieval; yet their inner workings remain largely opaque. This hinders not only our understanding of the systems implementing them, but also our ability to improve them. To alleviate this limitation, we introduce IR Lens, a new interpretability tool tailored to cross-encoders based on two key components: 1) Neuron Integrated Gradients to expose the contributions of model parts at multiple levels, and 2) targeted ablations to support hypothesis tracking. With its interactive graphical interface, IR Lens enables IR practitioners to explore, analyze, and manipulate neuron-level mechanisms in cross-encoders, facilitating a deeper understanding of neural ranking models. By extending the reach of existing interpretability methods, we believe IR Lens has the potential to support the improvement of cross-encoders.

Generative AI is increasingly used to create images from text, but its role in documentary photography remains under-explored. This paper investigates how generative AI can be integrated into documentary practice while maintaining ethical standards. Through interviews with six documentary photographers, we explored their views on AI’s potential to support community-driven storytelling. While AI presents opportunities for creative expression and community involvement, concerns about trust, authenticity, and decontextualization of images persist. Photographers expressed doubts about AI’s ability to accurately representlivedexperiences,fearingitcould compromise narrative integrity. Our findings suggest that AI tools should be designed to enhance collaboration and transparency in storytelling, complementing rather than replacing traditional documentary methods. This study contributes to the ongoing discourse on AI in photography, advocating for the development of tools that preserve the ethical foundations of documentary storytelling while empowering communities.

Trust between humans and AI in the context of decision-making has acquired an important role in public policy, research and industry. In this context, Human-AI Trust has often been tackled from the lens of cognitive science and psychology, but lacks insights from the stakeholders involved. In this paper, we conducted semi-structured interviews with 7 AI practitioners and 7 decision subjects from various decision domains. We found that 1) interviewees identified the prerequisites for the existence of trust and distinguish trust from trustworthiness, reliance, and compliance; 2) trust in AI-integrated systems is strongly influenced by other human actors, more than the system’s features; 3) the role of Human-AI trust factors is stakeholder-dependent. These results provide clues for the design of Human-AI interactions in which trust plays a major role, as well as outline new research directions in Human-AI Trust.

While human-centered approaches to machine learning explore various human roles within the interaction loop, the notion of Interactive Machine Teaching (IMT) emerged with a focus on leveraging the teaching skills of humans as a teacher to build machine learning systems. However, most systems and studies are devoted to single users. In this article, we study collaborative interactive machine teaching in the context of image classification to analyze how people can structure the teaching process collectively and to understand their experience. Our contributions are threefold. First, we developed a web application called TeachTOK that enables groups of users to curate data and train a model together incrementally. Second, we conducted a study in which ten participants were divided into three teams that competed to build an image classifier in nine days. Qualitative results of participants’ discussions in focus groups reveal the emergence of collaboration patterns in the machine teaching task, how collaboration helps revise teaching strategies and participants’ reflections on their interaction with the TeachTOK application. From these findings we provide implications for the design of more interactive, collaborative and participatory machine learning-based systems.

Pattern-recognition-based arm prostheses rely on recognizing muscle activation to trigger movements. The effectiveness of this approach depends not only on the performance of the machine learner but also on the user’s understanding of its recognition capabilities, allowing them to adapt and work around recognition failures. We investigate how different model training strategies to select gesture classes and record respective muscle contractions impact model accuracy and user comprehension. We report on a lab experiment where participants performed hand gestures to train a classifier under three conditions: (1) the system cues gesture classes randomly (control), (2) the user selects gesture classes (teacher-led), (3) the system queries gesture classes based on their separability (learner-led). After training, we compare the models’ accuracy and test participants’ predictive understanding of the prosthesis’ behavior. We found that teacher-led and learner-led strategies yield faster and greater performance increases, respectively. Combining two evaluation methods, we found that participants developed a more accurate mental model when the system queried the least separable gesture class (learner-led). Our results conclude that, in the context of machine learning-based myoelectric prosthesis control, guiding the user to focus on class separability during training can improve recognition performances and support users’ mental models about the system’s behavior. We discuss our results in light of several research fields : myoelectric prosthesis control, motor learning, human-robot interaction, and interactive machine teaching.

Seen both as a resource and an obstacle to clarity, uncertainty is a concept that permeates many areas of design. As the concept gains prominence in Human-Computer Interaction (HCI), this special issue specifically explores the interplay between uncertainty and prototyping in Research through Design (RtD). We first outline three histories of uncertainty in design, in relation to its philosophical significance, its role in statistical and algorithmic processes, and its importance in prototyping. The convergence of these aspects is crucial as design evolves toward more agentive and entangled systems, introducing challenges such as Design as a Probabilistic Outcome. We then investigate the design spaces for engaging with “being uncertain” that emerge from the papers: from nuancing the relationship between designers and quantitative data to blurring the line between humans, fungi, and algorithms. Finally, we illuminate some preliminary threads for how RtD can navigate and engage with these shifting technological and design landscapes thoughtfully.

User interfaces typically feature tools to act on objects and rely on the ability of users to discover or learn how to interact with them. Previous work in HCI has used the Theory of Affordances to explain how users understand the possibilities for action in digital environments. A complementary theory from cognitive neuroscience, Technical Reasoning, posits that users accumulate abstract knowledge of object properties and technical principles known as mechanical knowledge, essential in tool use. Drawing from this theory, we introduce interaction knowledge as the “mechanical” knowledge of digital environments. We provide evidence of its relevance by reporting on an experiment where participants performed tasks in a digital environment with ambiguous possibilities for interaction. We analyze how interaction knowledge was transferred across two digital domains, text editing and graphical editing, and conclude that interaction knowledge models an essential type of knowledge for interacting in the digital world.

This paper combines design, machine learning and social computing to explore generative deep learning as both tool and probe for respiratory care. We first present GANspire, a deep learning tool that generates fine-grained breathing waveforms, which we crafted in collaboration with one respiratory physician, attending to joint materialities of human breathing data and deep generative models. We then relate a probe, produced with breathing waveforms generated with GANspire, and led with a group of ten respiratory care experts, responding to its material attributes. Qualitative annotations showed that respiratory care experts interpreted both realistic and ambiguous attributes of breathing waveforms generated with GANspire, according to subjective aspects of physiology, activity and emotion. Semi-structured interviews also revealed experts' broader perceptions, expectations and ethical concerns on AI technology, based on their clinical practice of respiratory care, and reflexive analysis of GANspire. These findings suggest design implications for technological aids in respiratory care, and show how ambiguity of deep generative models can be leveraged as a resource for qualitative inquiry, enabling socio-material research with generative deep learning. Our paper contributes to the CSCW community by broadening how generative deep learning may be approached not only as a tool to design human-computer interactions, but also as a probe to provoke open conversations with communities of practice about their current and speculative uses of AI technology.

The Technical Reasoning hypothesis in cognitive neuroscience posits that humans engage in physical tool use by reasoning about mechanical interactions among objects. By modeling the use of objects as tools based on their abstract properties, this theory explains how tools can be re-purposed beyond their assigned function. This paper assesses the relevance of Technical Reasoning to digital tool use. We conducted an experiment with 16 participants that forced them to re-purpose commands to complete a text layout task. We analyzed self-reported scores of creative personality and experience with text editing, and found a significant association between re-purposing performance and creativity, but not with experience. Our results suggest that while most participants engaged in Technical Reasoning to re-purpose digital tools, some experienced “functional fixedness.” This work contributes Technical Reasoning as a theoretical model for the design of digital tools.

Machine Learning models can output confident but incorrect predictions. To address this problem, ML researchers use various techniques to reliably estimate ML uncertainty, usually performed on controlled benchmarks once the model has been trained. We explore how the two types of uncertainty—aleatoric and epistemic—can help non-expert users understand the strengths and weaknesses of a classifier in an interactive setting. We are interested in users’ perception of the difference between aleatoric and epistemic uncertainty and their use to teach and understand the classifier. We conducted an experiment where non-experts train a classifier to recognize card images, and are tested on their ability to predict classifier outcomes. Participants who used either larger or more varied training sets significantly improved their understanding of uncertainty, both epistemic or aleatoric. However, participants who relied on the uncertainty measure to guide their choice of training data did not significantly improve classifier training, nor were they better able to guess the classifier outcome. We identified three specific situations where participants successfully identified the difference between aleatoric and epistemic uncertainty: placing a card in the exact same position as a training card; placing different cards next to each other; and placing a non-card, such as their hand, next to or on top of a card. We discuss our methodology for estimating uncertainty for Interactive Machine Learning systems and question the need for two-level uncertainty in Machine Teaching.

Alongside recent advances in artificial intelligence (AI), a new art practice has emerged in recent years that borrows and transforms these advances in the production of artworks. The actors of this emergent practice are coming from contemporary art, media and digital arts. These artists have developed an original practice of AI within their creative field. In this article, we propose a qualitative study to explore the nature of this practice. We interviewed five internationally renowned artists about how AI is integrated into their work. Through a thematic analysis of the interviews, we first find that their practice relies on crafting algorithms and data as materials. We uncover how they explicitly use this material unpredictability rather than avoid it. Secondly, we highlight the politics of their practice that consist of resisting the culture of AI research, as well as its inherent power dynamics. We also highlight how their relationship with the technology is imbued with ethics and how they rethink their role with respect to the technology. In this paper, we aim to provide the CSCW community with a way to expand the framework in which AI can be understood not only as a tool but also as cultural and political design material.

In this paper, we outline a diffractive practice of machine learning (ML) in the frame of material-centered interaction design. To this aim, we review related work in ML, HCI, design, new interfaces for musical expression, and computational art, and introduce two practice-based studies of music performance and robotic art based on interactive machine learning tools, with the hope of revealing the computational materiality of ML, and the potential of embodiment to craft prototypes of ML that reconfigure conceptual or technical approaches to ML. We derive five interference conditions for such art-based ML prototypes—situational whole, small data, shallow model, learnable algorithm, and somaesthetic behaviour—and describe their widening of design and engineering practices of ML prototyping. Finally, we sketch how a process of intra-active machine learning could complement that of interactive machine learning to take materiality as an entry point for ML design within HCI.

Human-centered approaches to machine learning have established theoretical foundations, design principles and interaction techniques to facilitate end-user interaction with machine learning systems. Yet, general-purpose toolkits supporting the design of interactive machine learning systems are still missing, despite their potential to foster reuse, appropriation and collaboration between different stakeholders including developers, machine learning experts, designers and end users. In this paper, we present an architectural model for toolkits dedicated to the design of human interactions with machine learning. The architecture is built upon a modular collection of interactive components that can be composed to build interactive machine learning workflows, using reactive pipelines and composable user interfaces. We introduce Marcelle, a toolkit for the design of human interactions with machine learning that implements this model. We illustrate Marcelle with two implemented case studies: (1) a HCI researcher conducts user studies to understand novice interaction with machine learning, and (2) a machine learning expert and a clinician collaborate to develop a skin cancer diagnosis system. Finally, we discuss our experience with the toolkit, along with its limitation and perspectives.

Preparing a new dance performance involves more than learning individual steps. We are interested in understanding how dancers collaborate as they rehearse a new dance piece, with a particular emphasis on how they use physical and digital artifacts to support this process. We conducted a 12-month longitudinal observational study with a dance company that re-staged a dance piece, taken from the contemporary repertoire and unknown to the dancers. The study focused on the role that artifacts play in shaping the learning of a dance piece. We showed how dancers produced an ecology of artifacts with the aim of analyzing the choreographic ideas behind the dance and sharing them with other learners. We showed that sharing these artifacts was challenging because they are idiosyncratic and embody their creator perspective and vocabulary. We then illustrated how dancers overcome this challenge by compiling artifacts and distributing the learning task among the group in order to create a common knowledge of the piece which improves the learning process. We conclude with design opportunities for technologies supporting long-term dance learning processes.

Machine learning systems became pervasive in modern interactive technology but provide users with little, if any, agency with respect to how their models are trained from data. In this paper, we are interested in the way novices handle learning algorithms, what they understand from their behavior and what strategy they may use to "make it work". We developed a web-based sketch recognition algorithm based on Deep Neural Network (DNN), called Marcelle-Sketch, that end-users can train incrementally. We present an experimental study that investigate people's strategies and (mis)understandings in a realistic algorithm-teaching task. Our study involved 12 participants who performed individual teaching sessions using a think-aloud protocol. Our results show that participants adopted heterogeneous strategies in which variability affected the model performances. We highlighted the importance of sketch sequencing, particularly at the early stage of the teaching task. We also found that users' understanding is facilitated by simple operations on drawings, while confusions are caused by certain inherent properties of DNN. From these findings, we propose implications for design of IML systems dedicated to novices and discuss the socio-cultural aspect of this research.

The spread of AI-embedded systems involved in human decision making makes studying human trust in these systems critical. However, empirically investigating trust is challenging. One reason is the lack of standard protocols to design trust experiments. In this paper, we present a survey of existing methods to empirically investigate trust in AI-assisted decision making and analyse the corpus along the constitutive elements of an experimental protocol. We find that the definition of trust is not commonly integrated in experimental protocols, which can lead to findings that are overclaimed or are hard to interpret and compare across studies. Drawing from empirical practices in social and cognitive studies on human-human trust, we provide practical guidelines to improve the methodology of studying Human-AI trust in decision-making contexts. In addition, we bring forward research opportunities of two types: one focusing on further investigation regarding trust methodologies and the other on factors that impact Human-AI trust.

Software tools for generating digital sound often present users with high-dimensional, parametric interfaces, that may not facilitate exploration of diverse sound designs. In this article, we propose to investigate artificial agents using deep reinforcement learning to explore parameter spaces in partnership with users for sound design. We describe a series of user-centred studies to probe the creative benefits of these agents and adapting their design to exploration. Preliminary studies observing users’ exploration strategies with parametric interfaces and testing different agent exploration behaviours led to the design of a fully-functioning prototype, called Co-Explorer, that we evaluated in a workshop with professional sound designers. We found that the Co-Explorer enables a novel creative workflow centred on human–machine partnership, which has been positively received by practitioners. We also highlight varied user exploration behaviours throughout partnering with our system. Finally, we frame design guidelines for enabling such co-exploration workflow in creative digital applications.

Our goal is to understand how dancers learn complex dance phrases. We ran three workshops where dancers learned dance fragments from videos. In workshop 1, we analyzed how dancers structure their learning strategies by decomposing movements. In workshop 2, we introduced MoveOn, a technology probe that lets dancers decompose video into short, repeatable clips to support their learning. This served as an effective analysis tool for identifying the changes in focus and understanding their decomposition and recomposition processes. In workshop 3, we compared the teacher's and dancers' decomposition strategies, and how dancers learn on their own compared to teacher-created decompositions. We found that they all ungroup and regroup dance fragments, but with different foci of attention, which suggests that teacher-imposed decomposition is more effective for introductory dance students, whereas personal decomposition is more suitable for expert dancers. We discuss the implications for designing technology to support analysis, learning and teaching of dance through movement decomposition.

This note presents a system that learns expressive and idiosyncratic gesture variations for gesture-based interaction. The system is used as an interaction technique in a music conducting scenario where gesture variations drive music articulation. A simple model based on Gaussian Mixture Modeling is used to allow the user to configure the system by providing variation examples. The system performance and the influence of user musical expertise is evaluated in a user study, which shows that the model is able to learn idiosyncratic variations that allow users to control articulation, with better performance for users with musical expertise.

Sonic interaction is the continuous relationship between user actions and sound, mediated by some technology. Because interaction with sound may be task oriented or experience-based it is important to understand the nature of action-sound relationships in order to design rich sonic interactions. We propose a participatory approach to sonic interaction design that first considers the affordances of sounds in order to imagine embodied interaction, and based on this, generates interaction models for interaction designers wishing to work with sound. We describe a series of workshops, called Form Follows Sound, where participants ideate imagined sonic interactions, and then realize working interactive sound prototypes. We introduce the Sonic Incident technique, as a way to recall memorable sound experiences. We identified three interaction models for sonic interaction design: conducting; manipulating; substituting. These three interaction models offer interaction designers and developers a framework on which they can build richer sonic interactions.

This article presents a gesture recognition/adaptation system for human--computer interaction applications that goes beyond activity classification and that, as a complement to gesture labeling, characterizes the movement execution. We describe a template-based recognition method that simultaneously aligns the input gesture to the templates using a Sequential Monte Carlo inference technique. Contrary to standard template-based methods based on dynamic programming, such as Dynamic Time Warping, the algorithm has an adaptation process that tracks gesture variation in real time. The method continuously updates, during execution of the gesture, the estimated parameters and recognition results, which offers key advantages for continuous human--machine interaction. The technique is evaluated in several different ways: Recognition and early recognition are evaluated on 2D onscreen pen gestures; adaptation is assessed on synthetic data; and both early recognition and adaptation are evaluated in a user study involving 3D free-space gestures. The method is robust to noise, and successfully adapts to parameter variation. Moreover, it performs recognition as well as or better than nonadapting offline template-based methods.

Expressivity is a visceral capacity of the human body. To understand what makes a gesture expressive, we need to consider not only its spatial placement and orientation but also its dynamics and the mechanisms enacting them. We start by defining gesture and gesture expressivity, and then we present fundamental aspects of muscle activity and ways to capture information through electromyography and mechanomyography. We present pilot studies that inspect the ability of users to control spatial and temporal variations of 2D shapes and that use muscle sensing to assess expressive information in gesture execution beyond space and time. This leads us to the design of a study that explores the notion of gesture power in terms of control and sensing. Results give insights to interaction designers to go beyond simplistic gestural interaction, towards the design of interactions that draw on nuances of expressive gesture.

We investigated gesture description of sound stimuli performed during a listening task. Our hypothesis is that the strategies in gestural responses depend on the level of identification of the sound source and specifically on the identification of the action causing the sound. To validate our hypothesis, we conducted two experiments. In the first experiment, we built two corpora of sounds. The first corpus contains sounds with identifiable causal actions. The second contains sounds for which no causal actions could be identified. These corpora properties were validated through a listening test. In the second experiment, participants performed arm and hand gestures synchronously while listening to sounds taken from these corpora. Afterward, we conducted interviews asking participants to verbalize their experience while watching their own video recordings. They were questioned on their perception of the listened sounds and on their gestural strategies. We showed that for the sounds where causal action can be identified, participants mainly mimic the action that has produced the sound. In the other case, when no action can be associated with the sound, participants trace contours related to sound acoustic features. We also found that the interparticipants’ gesture variability is higher for causal sounds compared to noncausal sounds. Variability demonstrates that, in the first case, participants have several ways of producing the same action, whereas in the second case, the sound features tend to make the gesture responses consistent.

In this paper, we explore the use of movement qualities as interaction modality. The notion of movement qualities is widely used in dance practice and can be understood as how the movement is performed, independently of its specific trajectory in space. We implemented our approach in the context of an artistic installation called A light touch. This installation invites the participant to interact with a moving light spot reacting to the hand movement qualities. We conducted a user experiment that showed that such an interaction based on movement qualities tends to enhance the user experience favouring explorative and expressive usage.