As agentic AI systems grow increasingly capable of operating for hours or days at a time, users’ prompts are transforming into highly elaborate specifications for the AI to autonomously work on. While prompting for bounded, single-turn tasks has been extensively studied, less is known about how people communicate specifications for long-horizon tasks. In this work, we conducted a qualitative study in which 16 professionals drafted specifications for both a human colleague and an AI, revealing a core divergence: participants treated human delegation as a “compass,” offering high-level intent to encourage flexible exploration. In contrast, communication with AI resembled painstakingly laying down “railway tracks”: rigid, exhaustive instructions to minimize ambiguity and deviation. This reflected a perception that current AI struggles to infer intent, prioritize, and make judgments on its own. When envisioning an ideal AI collaborator, users expressed a desire for a hybrid : a collaborator blending AI’s efficiency and large context window with the critical thinking and agency of a human colleague. We discuss design implications for future AI systems, proposing that they align on outcomes through generated rough drafts, verify feasibility via end-to-end “test runs,” and monitor execution through intelligent check-ins—ultimately transforming AI from a passive instruction-follower into a reliable collaborator for ambiguous, long-horizon tasks.

Multimodal large language models (MLLMs), with their expansive world knowledge and reasoning capabilities, present a unique opportunity for end-users to create personalized AI sensors capable of reasoning about complex situations. A user could describe a desired sensing task in natural language (e.g., "alert if my toddler is getting into mischief"), with the MLLM analyzing the camera feed and responding within seconds. In a formative study, we found that users saw substantial value in defining their own sensors, yet struggled to articulate their unique personal requirements and debug the sensors through prompting alone. To address these challenges, we developed Gensors, a system that empowers users to define customized sensors supported by the reasoning capabilities of MLLMs. Gensors 1) assists users in eliciting requirements through both automatically-generated and manually created sensor criteria, 2) facilitates debugging by allowing users to isolate and test individual criteria in parallel, 3) suggests additional criteria based on user-provided images, and 4) proposes test cases to help users "stress test" sensors on potentially unforeseen scenarios. In a user study, participants reported significantly greater sense of control, understanding, and ease of communication when defining sensors using Gensors. Beyond addressing model limitations, Gensors supported users in debugging, eliciting requirements, and expressing unique personal requirements to the sensor through criteria-based reasoning; it also helped uncover users' "blind spots" by exposing overlooked criteria and revealing unanticipated failure modes. Finally, we discuss how unique characteristics of MLLMs--such as hallucinations and inconsistent responses--can impact the sensor-creation process. These findings contribute to the design of future intelligent sensing systems that are intuitive and customizable by everyday users.

AI-generated images are proliferating as a new visual medium. However, state-of-the-art image generation models do not output alternative (alt) text with their images, rendering them largely inaccessible to screen reader users (SRUs). Moreover, less is known about what information would be most desirable to SRUs in this new medium. To address this, we invited AI image creators and SRUs to evaluate alt text prepared from various sources and write their own alt text for AI images. Our mixed-methods analysis makes three contributions. First, we highlight creators’ perspectives on alt text, as creators are well-positioned to write descriptions of their images. Second, we illustrate SRUs’ alt text needs particular to the emerging medium of AI images. Finally, we discuss the promises and pitfalls of utilizing text prompts written as input for AI models in alt text generation, and areas where broader digital accessibility guidelines could expand to account for AI images.

Recent advances in multimodal large language models (LLMs) have made it easier to rapidly prototype AI-powered features, especially for mobile use cases. However, gathering early, mobile-situated user feedback on these AI prototypes remains challenging. The broad scope and flexibility of LLMs means that, for a given use-case-specific prototype, there is a crucial need to understand the wide range of in-the-wild input users are likely to provide and their in-context expectations for the AI’s behavior. To explore the concept of in situ AI prototyping and testing, we created MobileMaker: a platform that enables designers to rapidly create and test mobile AI prototypes directly on devices. This tool also enables testers to make on-device, in-the-field revisions of prototypes using natural language. In an exploratory study with 16 participants, we explored how user feedback on prototypes created with MobileMaker compares to that of existing prototyping tools (e.g., Figma, prompt editors). Our findings suggest that MobileMaker prototypes enabled more serendipitous discovery of: model input edge cases, discrepancies between AI’s and user’s in-context interpretation of the task, and contextual signals missed by the AI. Furthermore, we learned that while the ability to make in-the-wild revisions led users to feel more fulfilled as active participants in the design process, it might also constrain their feedback to the subset of changes perceived as more actionable or implementable by the prototyping tool.

This paper examines the art practices, artwork, and motivations of prolific users of the latest generation of text-to-image models. Through interviews, observations, and a user survey, we present a sampling of the artistic styles and describe the developed community of practice around generative AI. We find that: 1) artists hold the text prompt and the resulting image can be considered collectively as a form of artistic expression (prompts as art), and 2) prompt templates (prompts with “slots” for others to fill in with their own words) are developed to create generative art styles. We discover that the value placed by this community on unique outputs leads to artists seeking specialized vocabulary to produce distinctive art pieces (e.g., by reading architectural blogs to find phrases to describe images). We also find that some artists use “glitches” in the model that can be turned into artistic styles of their own right. From these findings, we outline specific implications for design regarding future prompting and image editing options.

Despite the large body of work in accessibility concerning the design of novel navigation technologies, little is known about commonly available technologies that people with visual impairments currently use for navigation. We address this gap with a qualitative study consisting of interviews with 23 people with visual impairments, ten of whom also participated in a follow-up diary study. We develop the idea of complementarity first introduced by Williams et al. [53] and find that in addition to using apps to complement mobility aids, technologies and apps complemented each other and filled in for the gaps inherent in one another. Furthermore, the complementarity between apps and other apps/aids was primarily the result of the differences in information and modalities in which this information is communicated by apps, technology and mobility aids. We propose design recommendations to enhance this complementarity and guide the development of improved navigation experiences for people with visual impairments.

Despite the major role of Global Positioning Systems (GPS) as a navigation tool for people with visual impairments (VI), a crucial missing aspect of point-to-point navigation with these systems is the last-few-meters wayfinding problem. Due to GPS inaccuracy and inadequate map data, systems often bring a user to the vicinity of a destination but not to the exact location, causing challenges such as difficulty locating building entrances or a specific storefront from a series of stores. In this paper, we study this problem space in two parts: (1) A formative study (N=22) to understand challenges, current resolution techniques, and user needs; and (2) A design probe study (N=13) using a novel, vision-based system called Landmark AI to understand how technology can better address aspects of this problem. Based on these investigations, we articulate a design space for systems addressing this challenge, along with implications for future systems to support precise navigation for people with VI.

Alternative and augmentative communication (AAC) systems used by people with speech disabilities rely on text-to-speech (TTS) engines for synthesizing speech. Advances in TTS systems allowing for the rendering of speech with a range of emotions have yet to be incorporated into AAC systems, leaving AAC users with speech that is mostly devoid of emotion and expressivity. In this work, we describe voicesetting as the process of authoring the speech properties of text. We present the design and evaluation of two voicesetting user interfaces: the Expressive Keyboard, designed for rapid addition of expressivity to speech, and the Voicesetting Editor, designed for more careful crafting of the way text should be spoken. We evaluated the perceived output quality, requisite effort, and usability of both interfaces; the concept of voicesetting and our interfaces were highly valued by end-users as an enhancement to communication quality. We close by discussing design insights from our evaluations.

Augmentative and alternative communication (AAC) devices are a critical technology for people with disabilities that affect their speech. One challenge with AAC systems is their inability to portray aspects of nonverbal communication that typically accent, complement, regulate, or substitute for verbal speech. In this paper, we explore the design space of awareness displays that can supplement AAC devices, considering their output features and their effects on the perceptions of interlocutors. Through designing prototypes and getting feedback on our designs from people with ALS, their primary caregivers, and other communication partners, we consider (1) the consistent tensions that arose between abstractness and clarity in meaning for these designs and (2) the ways in which these designs can further mark users as "other." Overall, we contribute a generative understanding of designing AAC awareness displays to augment and contextualize communication.

Gaze-based alternative and augmentative communication (AAC) devices provide users with neuromuscular diseases the ability to communicate with other people through only the movement of their eyes. These devices suffer from slow input, causing a host of communication breakdowns to occur during face-to-face conversations. These breakdowns lead to decreased user autonomy, conversation quality, and communication partner engagement. Attempts to improve communication through these devices has mainly focused on throughput and rate enhancement, though this has only attained meager results to date. In this work, we address this issue from the top down by considering AAC devices as a form of groupware and designing interactions around this groupware that facilitate better conversations for all involved communicators. We first present qualitative findings on issues with gaze-based AAC and end-user communication preferences; we identify several design guidelines for improving these systems and then present AACrobat, a system that embodies these guidelines and introduces novel interactions by extending gaze-based AAC devices with a mobile companion app. Finally, we present early feedback on AACrobat through three case studies of users with ALS.

Simulation of a visual impairment may lead to a better understanding of how individuals with visual impairments perceive the world around them and could be a useful design tool for interface designers to identify accessibility barriers. Current simulation tools, however, suffer from a number of limitations, pertaining cost, accuracy and immersion. We present a simulation tool (SIMVIZ) that mounts a wide angle camera on a head-mounted display to create a see-through stereoscopic display that simulates various types and levels of visual impairments. A qualitative user study evaluates the immersiveness, usability and effectiveness of SIMVIZ versus using a smartphone based simulator. SIMVIZ enables quick accessibility inspections during iterative software development.

Simulation of a visual impairment may lead to a better understanding of how individuals with visual impairments perceive the world around them and could be a useful design tool for interface designers to identify accessibility barriers. Current simulation tools, however, suffer from a number of limitations, pertaining cost, accuracy and immersion. We present a simulation tool (SimViz) that mounts a wide angle camera on a head-mounted display to create a see-through stereoscopic display that simulates various types and levels of visual impairments. SimViz enables quick accessibility inspections during iterative software development.

Traversing large open spaces is a challenging task for blind cane users, as such spaces are often devoid of tactile features that can be followed. Consequently, in such spaces cane users may veer from their intended paths. Wearable devices have great potential for assistive applications for users who are blind as they typically feature a camera and support hands and eye free interaction. We present HEADLOCK; a navigation aid for an optical head-mounted display that helps blind users traverse large open spaces by letting them lock onto a salient landmark across the space, such as a door, and then providing audio feedback to guide the user towards the landmark. A user study with 8 blind users evaluated the usability and effectiveness of two types of audio feedback (sonification and text-to-speech) for guiding a user across an open space to a doorway. Qualitative results are reported, which may inform the design of assistive wearable technology for users who are blind.

Traversing large open spaces is a challenging task for blind cane users, as such spaces are often devoid of tactile features that can be followed. Consequently, in such spaces cane users may veer from their intended paths. Wearable devices have great potential for assistive applications for users who are blind as they typically feature a camera and support hands and eye free interaction. We present HEADLOCK; a navigation aid for an optical head-mounted display that helps blind users traverse large open spaces by letting them lock onto a salient landmark across the space, such as a door, and then providing audio feedback to guide the user towards the landmark. HEADLOCK consists of interface modes for discovering landmarks, guiding a user towards a landmark, and recovering from an error state if a landmark is lost. HEADLOCK is designed with two forms of audio feedback: sonification and text-to-speech.