Although directive prompting is the predominant way to interact with Large Language Models (LLMs), many creative practices rely on language that is open-ended, associative, phonaesthetic, symbolic, and that unfolds across multiple temporalities. In this work, we explore how creative practitioners might work with AI systems when language is treated not merely as instruction but as material. We conducted an ecological two-week study with four creative practitioners using a design probe: the Memetic Mixer, a tangible interactive device that constrains interaction with an LLM. Analysis of post-study interviews and device logs identified distinct modes of material language use and temporalities that shaped each participant’s engagement with AI and their creative practice. We reflect on these findings and contribute design considerations that support open-ended interaction with AI in creative practice.

This paper presents a practice-based case study that explores personalisation as a design methodology for embodied gestural instruments. Through a multi-year collaboration with a professional performer with physical disability, we employed research-through-design and co-design methods to iteratively develop a personalised instrument responsive to the performer’s unique movement style and creative vision. The system enabled real-time control of sound, stage lighting, and visualisations, and was ecologically validated in rehearsals and in an award-winning disability-led theatre production. Extending beyond the stage, we refined the instrument through workshops with young people with motor, sensory, and communication disabilities, demonstrating its adaptability to diverse bodies and creative practices. Rather than focusing on generalised solutions, this work advances methods for designing technologies that embrace difference, tailoring interaction to individual capabilities. We contribute to HCI research by articulating personalisation as a methodological approach to inclusive interaction design, expanding opportunities for creative expression among people with physical disability.

Recent developments in prompt-based generative AI has given rise to discourse surrounding the perceived ethical concerns, economic implications, and consequences for the future of cultural production. As generative imagery becomes pervasive in mainstream society, dominated primarily by emerging industry leaders, we encourage that the role of the CHI community be one of inquiry; to investigate the numerous ways in which generative AI has the potential to, and already is, augmenting human creativity. In this paper, we conducted a diffractive analysis exploring the potential role of prompt-based interfaces in artists’ creative practice. Over a two week period, seven visual artists were given access to a personalised instance of Stable Diffusion, fine-tuned on a dataset of their work. In the following diffractive analysis, we identified two dominant modes adopted by participants, AI for ideation, and AI for production. We furthermore present a number of ethical design considerations for the future development of generative AI interfaces.

As we become increasingly entangled with digital technologies, the boundary between human and machine is progressively blurring. Adopting a performative, posthumanist perspective resolves this ambiguity by proposing that such boundaries are not predetermined, rather they are enacted within a certain material configuration. Using this approach, dubbed ‘Entanglement HCI’, this paper presents Message Ritual – a novel, integrated AI system that encourages the re-framing of memory through machine generated poetics. Embodied within a domestic table lamp, the system listens in on conversations occurring within the home, drawing out key topics and phrases of the day and reconstituting them through machine generated poetry, delivered to household members via SMS upon waking each morning. Participants across four households were asked to live with the lamp over a two week period. We present a diffractive analysis exploring how the lamp becomes with participants and discuss the implications of this method for future HCI research.

Design—especially of physical objects—can be understood as creative acts solving practical problems. In this article, we describe a biologically inspired developmental model as the basis of a generative form-finding system. Using local interactions between cells in a 2-D environment, then capturing the state of the system at every time step, complex three-dimensional (3-D) forms can be generated by the system. Unlike previous systems, our method is capable of directly producing 3-D printable objects, eliminating intermediate transformations and manual manipulation often necessary to ensure the 3-D form is printable. We devise fitness measures for optimizing 3-D printability and aesthetic complexity and use a covariance matrix adaptation evolutionary strategies algorithm (CMA-ES) to find 3-D forms that are both aesthetically interesting and physically printable using fused deposition modeling printing techniques. We investigate the system’s capabilities by evolving and 3-D printing objects at different levels of structural consistency, and assess the quality of the fitness measures presented to explore the design space of our generative system. We find that by evolving first for aesthetic complexity, then evolving for structural consistency until the form is “just printable,” gives the best results.

This research examines the reflexive dimensions of cinematic virtual reality (CVR) storytelling. We created Anonymous, an interactive CVR piece that employs a reflexive storytelling method. This method is based on distancing effects and is used to elicit audience awareness and self-reflection about loneliness and death. To understand the audience's experiences, we conducted in-depth interviews to study which design factors and elements prompted reflexive thoughts and feelings. Our findings highlight how the audience experience was impacted by four reflexive dimensions: abstract and minimal aesthetics, everyday materials and textures, the restriction of control, and multiple, disembodied points of view. We use our findings to discuss how these dimensions can inform the design of VR storytelling experiences that provoke self and social reflection.

This paper introduces a new form of real-time affective interface that engages the user in a process of conceptualisation of their emotional state. Inspired by Barrett's Theory of Constructed Emotion, 'Mirror Ritual' aims to expand upon the user's accessible emotion concepts, and to ultimately provoke emotional reflection and regulation. The interface uses classified emotions – obtained through facial expression recognition -- as a basis for dynamically generating poetry. The perceived emotion is used to seed a poetry generation system based on OpenAI's GPT-2 model, fine-tuned on a specially curated corpus. We evaluate the device's ability to foster a personalised, meaningful experience for individual users over a sustained period. A qualitative analysis revealed that participants were able to affectively engage with the mirror, with each participant developing a unique interpretation of its poetry in the context of their own emotional landscape.

The aim of this art-based research is to explore the duality of embodiment and presence in virtual reality (VR) through a narrative about the South Korean narcoleptic artist, Sungeun Lee. Narcolepsy is a neurological disorder where a person falls asleep unexpectedly. A narcoleptic struggles with a blurred and unsure sense of self, as well as confusion between reality and dreaming. We used VR to simulate his traumatized dream world and symptoms of narcolepsy. However, we also investigated the fundamental duality of embodiment/disembodiment in the virtual space and its parallels in the dream world. A non-linear narrative and generative poetic aesthetics driven by biometric feedback were applied to represent a surreal and uncertain sense of self. Gesture-based navigable interaction was used to connect people with their physical bodies, the virtual environment, and Sungeun's experiences.

Mirror Ritual is an interactive installation that challenges the existing paradigms in our understanding of human emotion and machine perception. In contrast to prescriptive interfaces, the work's real-time affective interface engages the audience in the iterative conceptualisation of their emotional state through the use of affectively-charged machine generated poetry. The audience are encouraged to make sense of the mirror's poetry by framing it with respect to their recent life experiences, effectively 'putting into words' their felt emotion. This process of affect labelling and contextualisation works to not only regulate emotion, but helps to construct the rich personal narratives that constitute human identity.

The generation of music that adapts dynamically to content and actions has an important role in building more immersive, memorable, and emotive game experiences. To date, the development of adaptive music systems (AMSs) for video games is limited both by the nature of algorithms used for real-time music generation and the limited modeling of player action, game-world context, and emotion in current games. We propose that these issues must be addressed in tandem for the quality and flexibility of adaptive game music to significantly improve. Cognitive models of knowledge organization and emotional effect are integrated with multimodal, multiagent composition techniques to produce a novel AMS. The system is integrated into two stylistically distinct games. Gamers reported an overall higher immersion and correlation of music with game-world concepts with the AMS than that with the original game soundtracks in both the games.

Collaboration is built on trust, and establishing trust with a creative Artificial Intelligence is difficult when the decision process or internal state driving its behaviour isn't exposed. When human musicians improvise together, a number of extra-musical cues are used to augment musical communication and expose mental or emotional states which affect musical decisions and the effectiveness of the collaboration. We developed a collaborative improvising AI drummer that communicates its confidence through an emoticon-based visualisation. The AI was trained on musical performance data, as well as real-time skin conductance, of musicians improvising with professional drummers, exposing both musical and extra-musical cues to inform its generative process. Uni- and bi-directional extra-musical communication with real and false values were tested by experienced improvising musicians. Each condition was evaluated using the FSS-2 questionnaire, as a proxy for musical engagement. The results show a positive correlation between extra-musical communication of machine internal state and human musical engagement.

Libraries are increasingly offering on-line digital access to their collections. However, traditional search-based interfaces are restrictive and do not encourage the user to explore the collection in the same way that a physical collection does. We present the Mitchell WordCloud, a novel on-line interface to the David Scott Mitchell collection of the State Library of New South Wales. Based on interface design principles for explorative search, it presents the user with a word cloud derived from the collection and a list of titles. As the user drags words from the word cloud to tell the system what they like or dislike the title list is reordered. The surrounding interface elements -- image bar, time line and Dewey bar -- provide complementary insights into the collection. The traditional vector space model for measuring text similarity was extended to take account of user dislikes and to order words in the word cloud. User studies confirmed that the Mitchell WordCloud is easy to use and encourages exploration.

We use a new measure of creativity as a guide in an interactive evolutionary art task and tie the results to natural language usage of the term “creative.” Following previous work, we explore a tractable definition of creativity, one emphasizing the novelty of systems, and its addition to an interactive application. We next introduce a generative ecosystemic art system, EvoEco, an agent-based pixel-level means of generating images. EvoEco is used as a component of an online survey which asks users to evolve a pleasing image and then rank the success of the process and its output. Evolutionary search is augmented with the creativity measure, and compared with control groups augmented with either random search or a measure of phenotypic distance. We show that users consistently rate the creativity measure-enhanced version as more “creative” and “novel” than other search techniques. We further derive additional insights into appropriate forms of genetic representation and pattern space-traversal in an interactive evolutionary algorithm.

This paper examines how models from ecology and evolutionary multi-agent systems can be used for creative applications. We outline a minimal, general specification for an ecosystem to be applied to the production of creative artefacts, based on spaces, materials, features and actions. Using this specification, we consider how basic evolutionary principles based on niche construction can be used to establish the emergence of heterogeneous structures in the simplest imaginable cases to which our framework applies. We investigate the minimal conditions for the formation and maintenance of boundaries, and then apply the results of our initial models to the design of a creative artwork, contrasting differences between niche-constructing and neutral evolutionary models.

In this paper, we explore a generative art system designed to promote the creation of a diverse range of aesthetically pleasing images. We introduce our system, EvoEco, an agent-based pixel-level means of generating images based on artificial ecosystems. This art system is driven by interactive evolutionary computation, and further augmented using special measures to promote the diversity of the individuals. Following previous work, we explore a tractable definition of creativity and its addition to this interactive search. EvoEco was released online, and used by forty-one anonymous users to generate artwork. Here we present some of the discovered results.

In this paper, we introduce a new image database, consisting of examples of artists' work. Successful classification of this database suggests the capacity to automatically recognize an artist's aesthetic style. We utilize the notion of Transform-based Evolvable Features as a means of evolving features on the space, these features are then evaluated through a standard classifier. We obtain recognition rates for our six artistic styles - relative to images by the other artists and images randomly downloaded from a search engine - of a mean true positive rate of 0.946 and a mean false positive rate of 0.017. Distance metrics designed to indicate the similarity between an arbitrary greyscale image and one of the artistic styles are created from the evolved features. These metrics are capable of ranking control images so that artist-drawn instances appear at the front of the list. We provide evidence that other images ranked as similar by the metric correspond to naïve human notions of similarity as well, suggesting the distance metric could serve as a content-based aesthetic recommender.

This paper discusses the concept of an artificial ecosystem for use in machine-assisted creative discovery. Properties and processes from natural ecosystems are abstracted and applied to the design of creative systems, in a similar way that evolutionary computing methods use the metaphor of Darwinian evolution to solve problems in search and optimisation. The paper examines some appropriate mechanisms and metaphors when applying artificial ecosystems to problems in creative design. General properties and processes of evolutionary artificial ecosystems are presented as a basis for developing individual systems that automate the discovery of novelty without explicit teleological goals. The adaptation of species to fit their environment drives the creative solutions, so the role of the designer shifts to the design of environments. This allows a variety of creative solutions to emerge in simulation without the need for explicit or human-evaluated fitness measures, such as those used in interactive evolution. Two example creative ecosystems are described to highlight the effectiveness of the method presented.

Implicit surfaces obtained by convolution of multi‐dimensional primitives with some potential function, are a generalisation of popular implicit surface models: blobs, metaballs and soft objects. These models differ in their choice of potential function but agree upon the use of underlying modelling primitives, namely, points. In this paper a method is described for modelling and rendering implicit surfaces built upon an expanded set of skeletal primitives: points, line segments, polygons, arcs and planes. An analytical solution to the convolution is described. This solution offers a more accurate and robust representation of the resultant implicit surface than previous methods. An algorithm for ray‐tracing the surfaces formed through convolution of any combination of these primitives is also outlined.