Text-to-image and image-to-text models allow automated (but imperfect) semantic translation across modalities. This paper presents results and preliminary analysis of an empirical study of recursive information processing in popular open-weight generative artificial intelligence (genAI) models such as FluxSchnell and BLIP-2. Through clustering and topological data analysis we show some of the ways that different genAI models and initial prompts give rise to different semantic embedding trajectories, and suggest some ways forward for understanding how semantic information is transmitted through these types of complex information-processing systems.

Indoor asset tracking is an essential task in many areas of industry such as shipping and warehousing. Widely-used asset tracking technologies typically require supporting infrastructure (signal transponders) to communicate with active tags on the assets. Continuous asset tracking in large indoor spaces like warehouses can be costly, and reliable reference points are needed to calculate asset positions accurately. In this paper, we describe an indoor asset tracking technique for augmented reality (AR) that combines the use of (passive) fiducial markers together with flexible spatial anchors. Our approach, called SABIAT, continuously tracks the approximate location of an asset using spatial anchors and, when needed, the precise location of that asset using fiducial markers. We have applied our SABIAT technique to build a demonstrator system, AR-IPS, to show how assets can be tracked and located inside of a large, multi-level building.

In this paper we introduce Camera Adversaria; a mobile app designed to disrupt the automatic surveillance of personal photographs by technology companies. The app leverages the brittleness of deep neural networks with respect to high-frequency signals, adding generative adversarial perturbations to users' photographs. These perturbations confound image classification systems but are virtually imperceptible to human viewers. Camera Adversaria builds on methods developed by machine learning researchers as well as a growing body of work, primarily from art and design, which transgresses contemporary surveillance systems. We map the design space of responses to surveillance and identify an under-explored region where our project is situated. Finally we show that the language typically used in the adversarial perturbation literature serves to affirm corporate surveillance practices and malign resistance. This raises significant questions about the function of the research community in countenancing systems of surveillance.

Sign language translation (SLT) aims to interpret sign video sequences into text-based natural language sentences. Sign videos consist of continuous sequences of sign gestures with no clear boundaries in between. Existing SLT models usually represent sign visual features in a frame-wise manner so as to avoid needing to explicitly segmenting the videos into isolated signs. However, these methods neglect the temporal information of signs and lead to substantial ambiguity in translation. In this paper, we explore the temporal semantic structures of sign videos to learn more discriminative features. To this end, we first present a novel sign video segment representation which takes into account multiple temporal granularities, thus alleviating the need for accurate video segmentation. Taking advantage of the proposed segment representation, we develop a novel hierarchical sign video feature learning method via a temporal semantic pyramid network, called TSPNet. Specifically, TSPNet introduces an inter-scale attention to evaluate and enhance local semantic consistency of sign segments and an intra-scale attention to resolve semantic ambiguity by using non-local video context. Experiments show that our TSPNet outperforms the state-of-the-art with significant improvements on the BLEU score (from 9.58 to 13.41) and ROUGE score (from 31.80 to 34.96) on the largest commonly used SLT dataset. Our implementation is available at https://github.com/verashira/TSPNet.

Live coding is an artistic performance practice where computer music and graphics are created, displayed and manipulated in front of a live audience together with projections of the (predominantly text based) computer code. Although the pressures of such performance practices are considerable, until now only familiar programming aids (for example syntax highlighting and textual overlays) have been used to assist live coders. In this paper we integrate an intelligent agent into a live coding system and examine how that agent performed in two contrasting modes of operation-as a recommender, where on request the the agent would generate (but not execute) domain-appropriate code transformations in the performer's text editor, and as a disruptor, where the agent generated and immediately executed changes to the code and the streaming music being performed. A within-subjects study of six live coders suggested that participants of the study preferred the agent's disruptive mode as a mechanism for enhancing the creativity of their live coding performances.

We present the results of two controlled studies of free-improvised ensemble music-making on touch-screens. In our system, updates to an interface of harmonically-selected pitches are broadcast to every touch-screen in response to either a performer pressing a GUI button, or to interventions from an intelligent agent. In our first study, analysis of survey results and performance data indicated significant effects of the button on performer preference, but of the agent on performance length. In the second follow-up study, a mixed-initiative interface, where the presence of the button was interlaced with agent interventions, was developed to leverage both approaches. Comparison of this mixed-initiative interface with the always-on button-plus-agent condition of the first study demonstrated significant preferences for the former. The different approaches were found to shape the creative interactions that take place. Overall, this research offers evidence that an intelligent agent and a networked GUI both improve aspects of improvised ensemble music-making.

Percussionists are unique among western classical instrumentalists in that their artistic practice is defined by an approach to interaction rather than their instruments. While percussionists are accustomed to exploring non-traditional objects to create music, these objects have yet to encompass touch-screen computing devices to any great extent. The proliferation and popularity of these devices now presents an opportunity to explore their use in combining computer-generated sound together with percussive interaction in a musical ensemble.

Livecoding is an artistic programming practice in which an artist's low-level interaction can be observed with sufficiently high fidelity to allow for transcription and analysis. This paper presents the first reported "coding" of livecoding videos. From an identified corpus of videos available on the web, we coded performances of two different livecoding artists, recording both the (textual) programming edit events and the musical effect of these edits. Our analysis includes a novel, transition-matrix visualisation of the textual and musical dimensions of this data to create a "performer fingerprint". We show how detailed transcriptions of livecoding videos can be made which, we hope, will provide a foundation for further research into describing and understanding livecoding.

Network connectivity offers the potential for a group of musicians to play together over the network. This paper describes a trans-Atlantic networked musical livecoding performance between Andrew Sorensen in Germany (at the Schloss Daghstuhl conference on Collaboration and Learning through Live Coding) and Ben Swift in San Jose (at YL/HCC) in September 2013. In this paper we describe the infrastructure developed to enable this performance.

The concepts of affect and assemblage proposed by thinkers such as Gilles Deleuze and Brian Massumi can help us to understand the interaction between users and artefacts in interactive systems, particularly in the context of computer-supported improvisation and creativity. In this paper I provide an introduction to affect and assemblage theory for HCI practitioners. I then use a case study of Viscotheque, an iOS-based interface for group musical collaboration, to demonstrate the application of affective analysis in making sense of improvisational group music making.