Creativity support tools (CSTs) increasingly include image-generation features. The underlying diffusion models enact a particular image diffusing process that AI CSTs tend to obscure within a black-box. Artists’ creative control is limited to indirect manipulation (prompting), chaining these “black-boxes” together, or using ML-engineering skills to build custom black-boxes. Seeking to maintain the low-threshold offered by prompting, while raising the ceiling of expressive interactions, we built Noise Pilot: a multi-layered approach to supporting diffusion-based creative processes at three levels of depth. We used Noise Pilot as a probe to study the artistic processes of 9 artists over a 2-week period. Artists engaged with diffusion at different levels of manipulative depth and crafted reusable artifacts to enact bespoke diffusion processes; some produced results impossible to achieve with prompting alone. We discuss how black-box AIs in CSTs limit creative power, and propose subverting this by favoring visibility over obscurity, and materiality over personification.

Procedural content generation (PCG), the process of algorithmically creating game components instead of manually, has been a common tool of game development for decades. Recent advances in large language models (LLMs) enable the generation of game behaviors based on player input at runtime. Such code generation brings with it the possibility of entirely new gameplay interactions that may be difficult to integrate with typical game development workflows. We explore these implications through GROMIT, a novel LLM-based runtime behavior generation system for Unity. When triggered by a player action, GROMIT generates a relevant behavior which is compiled without developer intervention and incorporated into the game. We create three demonstration scenarios with GROMIT to investigate how such a technology might be used in game development. In a system evaluation we find that our implementation is able to produce behaviors that result in significant downstream impacts to gameplay. We then conduct an interview study with n=13 game developers using GROMIT as a probe to elicit their current opinion on runtime behavior generation tools, and enumerate the specific themes curtailing the wider use of such tools. We find that the main themes of concern are quality considerations, community expectations, and fit with developer workflows, and that several of the subthemes are unique to runtime behavior generation specifically. We outline a future work agenda to address these concerns, including the need for additional guardrail systems for behavior generation.

A common way to enable immersion in VR is to render a virtual body that mirrors the user’s physical movements. VR allows us to design interaction schemes that go beyond direct avatar embodiments. In particular, there is a growing body of literature investigating the simultaneous control of multiple bodies in VR. We contribute to this literature by investigating the important case where multiple bodies perform a coordinated interaction with each other. Such actions directly question what kind of embodiment users experience. Concretely, we investigate people’s abilities to perform coordinated bimanual selection and handoff tasks between a first-person and third-person body through a user study with 19 participants. Results provide quantitative & qualitative evidence for people’s ability to perform complex coordinated tasks through two bodies. Furthermore we characterize participant performance in different task and interaction configurations, summarize the strategies they employed, and discuss qualities of user’s proprioception.

Programming tutorials are a pervasive, versatile medium for teaching programming. In this paper, we report on the content and structure of programming tutorials, the pain points authors experience in writing them, and a design for a tool to help improve this process. An interview study with 12 experienced tutorial authors found that they construct documents by interleaving code snippets with text and illustrative outputs. It also revealed that authors must often keep related artifacts of source programs, snippets, and outputs consistent as a program evolves. A content analysis of 200 frequently-referenced tutorials on the web also found that most tutorials contain related artifacts—duplicate code and outputs generated from snippets—that an author would need to keep consistent with each other. To address these needs, we designed a tool called Torii with novel authoring capabilities. An in-lab study showed that tutorial authors can successfully use the tool for the unique affordances identified, and provides guidance for designing future tools for tutorial authoring.

Computer vision is applied in an ever expanding range of applications, many of which require custom training data to perform well. We present a novel interface for rapid collection of labeled training images to improve CV-based object detectors. LabelAR leverages the spatial tracking capabilities of an AR-enabled camera, allowing users to place persistent bounding volumes that stay centered on real-world objects. The interface then guides the user to move the camera to cover a wide variety of viewpoints. We eliminate the need for post hoc labeling of images by automatically projecting 2D bounding boxes around objects in the images as they are captured from AR-marked viewpoints. In a user study with 12 participants, LabelAR significantly outperforms existing approaches in terms of the trade-off between detection performance and collection time.

Our perception of our surrounding environment is limited by the constraints of human biology. The field of augmented perception asks how our sensory capabilities can be usefully extended through computational means. We argue that spatial awareness can be enhanced by exploiting recent advances in computer vision which make high-accuracy, real-time object detection feasible in everyday settings. We introduce HindSight, a wearable system that increases spatial awareness by detecting relevant objects in live 360-degree video and sonifying their position and class through bone conduction headphones. HindSight uses a deep neural network to locate and attribute semantic information to objects surrounding a user through a head-worn panoramic camera. It then uses bone conduction headphones, which preserve natural auditory acuity, to transmit audio notifications for detected objects of interest. We develop an application using HindSight to warn cyclists of approaching vehicles outside their field of view and evaluate it in an exploratory study with 15 users. Participants reported increases in perceived safety and awareness of approaching vehicles when using HindSight.