We present a Research-through-Design exploration of Machine Learning (ML) as design material in music-making. We designed Picnic, an interactive musical installation that augments everyday objects in a picnic basket into a loop-based sampler which allows users to build rhythms with a variety of percussive, harmonic and more-than-human sounds. Embracing ML’s inherent uncertainty, we intentionally used an underfitted real-time classification model to create a playful and ambiguous music-making experience with the system. Through an evaluation with 23 participants of varying musical expertise and AI interest, we found that the system’s misclassifications made participants engage in a creative dialogue, constantly adapting to its unpredictability. Furthermore, when errors occurred, participants tended to criticise themselves rather than the system, indicating a tendency to overtrust the system. Our findings contribute with insights into the potential for using ML as design material for music-making and other creative domains.

Image-generation AI models have triggered a paradigm shift in how we can express ourselves in visual art. Despite their widespread use in a short amount of time, embedding these models into interactive artifacts is still largely unexplored. In this pictorial, we unpack the design and development process of GenFrame, an image generating picture frame that utilizes generative AI capabilities to mimic traditional paintings. Our work details the necessary steps to integrate generative AI into interactive artifacts and highlights important design considerations for controlling image-generation models in order to achieve specific design intents. Our insights provide interaction designers with a more comprehensive understanding and approach towards utilizing image-generation AI models for interactive artifacts. A demo can be viewed at https://youtu.be/1rhW4fazaBY

Deep Generative Models (DGM) have had a great impact both on visual art and broader visual culture. In this research-through-design project we investigate the use of a DGM for helping museum visitors explore the aesthetics of Edvard Munch’s art. We designed and built an interactive drawing table that allows a user to explore a StyleGAN model trained on sketches by Edvard Munch. The paper makes two novel contributions: 1. It presents a system that allows users to interact with a DGM by drawing on paper (rather than the typical text prompts used by most current systems). 2. We demonstrate how this mode and quality of interaction establish a unique perspective on Munch’s drawings as a practice. Through qualitative evaluation, we discuss how this setup led users towards a specific hermeneutic drawing strategy that enables building competency with the model and by proxy the data it is trained on. We suggest that the resulting interaction may contribute to an "education of attention" helping museum visitors to become attentive to certain visual qualities in Munch’s drawing practice. Finally, we discuss how the concepts of technological mediation and relationality are useful for designing how the output of a DGM is understood by its users.

This Research through Design paper explores how object detection may be applied to a large digital art museum collection to facilitate new ways of encountering and experiencing art. We present the design and evaluation of an interactive application called SMKExplore, which allows users to explore a museum’s digital collection of paintings by browsing through objects detected in the images, as a novel form of open-ended exploration. We provide three contributions. First, we show how an object detection pipeline can be integrated into a design process for visual exploration. Second, we present the design and development of an app that enables exploration of an art museum’s collection. Third, we offer reflections on future possibilities for museums and HCI researchers to incorporate object detection techniques into the digitalization of museums.

Ongoing efforts to turn Machine Learning (ML) into a design material have encountered limited success. This paper examines the burgeoning area of AI art to understand how artists incorporate ML in their creative work. Drawing upon related HCI theories, we investigate how artists create ambiguity by analyzing nine AI artworks that use computer vision and image synthesis. Our analysis shows that, in addition to the established types of ambiguity, artists worked closely with the ML process (dataset curation, model training, and application) and developed various techniques to evoke the ambiguity of processes. Our finding indicates that the current conceptualization of ML as a design material needs to reframe the ML process as design elements, instead of technical details. Finally, this paper offers reflections on commonly held assumptions in HCI about ML uncertainty, dependability, and explainability, and advocates to supplement the artifact-centered design perspective of ML with a process-centered one.

Museums have increasingly focused on digital technologies and play as means to provide personalized, engaging experiences for their audience. Balancing educational and playful values is often conflicting. To address that conflict, museums often employ participatory design strategies. However, those strategies usually end after the deployment of those experiences, thus they do not accommodate for what occurs during actual use. In this article, we follow Light House, a research-through-design experiment of an installation that was developed using an iterative design approach which expands on actual use by deploying undetermined artefacts to support the discovery of novel interactions by visitors. Through our findings, we explore the “failure” of Light House to support the discovery of such interactions in relation to its educational character, but rather it inspired people to incorporate it in the activities supported by the surrounding space. Finally, we discuss the implications those discovered interactions in terms of potential re-design directions.

Museums are interested in designing emotional visitor experiences to complement traditional interpretations. HCI is interested in the relationship between Affective Computing and Affective Interaction. We describe Sensitive Pictures, an emotional visitor experience co-created with the Munch art museum. Visitors choose emotions, locate associated paintings in the museum, experience an emotional story while viewing them, and self-report their response. A subsequent interview with a portrayal of the artist employs computer vision to estimate emotional responses from facial expressions. Visitors are given a souvenir postcard visualizing their emotional data. A study of 132 members of the public (39 interviewed) illuminates key themes: designing emotional provocations; capturing emotional responses; engaging visitors with their data; a tendency for them to align their views with the system's interpretation; and integrating these elements into emotional trajectories. We consider how Affective Computing can hold up a mirror to our emotions during Affective Interaction

Despite recent advances in object detection using deep learning neural networks, these neural networks still struggle to identify objects in art images such as paintings and drawings. This challenge is known as the cross depiction problem and it stems in part from the tendency of neural networks to prioritize identification of an object's texture over its shape. In this paper we propose and evaluate a process for training neural networks to localize objects — specifically people — in art images. We generate a large dataset for training and validation by modifying the images in the COCO dataset using AdaIn style transfer. This dataset is used to fine-tune a Faster R-CNN object detection network, which is then tested on the existing People-Art testing dataset. The result is a significant improvement on the state of the art and a new way forward for creating datasets to train neural networks to process art images.

We reflect on two museum visiting experiences that adopted the strategy of interpersonalization in which one visitor creates an experience for another. In the Gift app, visitors create personal mini-tours for specific others. In Never let me go, one visitor controls the experience of another by sending them remote instructions as they follow them around the museum. By reflecting on the design of these experiences and their deployment in museums we show how interpersonalization can deliver engaging social visits in which visitors make their own interpretations. We contrast the approach to previous research in customization and algorithmic personalization. We reveal how these experiences relied on intimacy between pairs of visitors but also between visitors and the museum. We propose that interpersonalization requires museums to step-back to make space for interpretation, but that this then raises the challenge of how to reintroduce the museum’s own perspective. Finally, we articulate strategies and challenges for applying this approach.

Concepts and theories that emerge within the social sciences tend to be nuanced, dealing with complex social phenomena. While their relevance to design could be high, it is difficult to make sense of them in design projects, especially when participants have a variety of backgrounds. We report on our experiences using role-play scenarios as a way to sensitize heterogeneous designer teams to complex theoretical concepts related to museology as social and cultural phenomena. We discuss design requirements on such scenarios, and the importance of connecting their execution closely to the context of the design and the current stage of the design process.

This paper presents the GIFT smartphone app, an artist-led Research through Design project benefitting from a three-day in-the-wild deployment. The app takes as its premise the generative potential of combining the contexts of gifting and museum visits. Visitors explore the museum, searching for objects that would most appeal to the gift-receiver they have in mind, then photographing those objects and adding audio messages for their receivers describing the motivation for their choices. This paper charts the designers' key aim of creating a new frame of mind using voice, and the most striking findings discovered during in-the-wild deployment in a museum -- 'seeing with new eyes' and fostering personal connections. We discuss empathy, motivation, and bottom-up personalisation in the productive space revealed by this combination of contexts. We suggest that this work reveals opportunities for designers of gifting services as well as those working in cultural heritage.