Knowledge workers increasingly rely on conversational AI for sense-making tasks (e.g., conducting market analysis), yet must repeatedly reconstruct context and intent to meet their goals. A formative study (N=10) showed that workflow reuse with AI often failed. Current tools either only remember preferences or enforce rigid, predefined workflows—neither adapts to evolving goals. We present ThinkFlow, a system that maintains a dynamic common ground through a cognitive workflow schema, enabling users to express intent and AI to adapt and reuse workflows across contexts. An expert-rating study shows that the schema can accurately capture the collocutor’s reasoning process, and when reused for a similar task, improves the AI’s responses compared to when the schema isn’t present. A user study with eight knowledge workers demonstrates that ThinkFlow supports awareness of evolving workflows, intent expression, and flexible application across contexts.

Large language models (LLMs) have shown strong performance on standardized social science instruments, but their value for product discovery remains unclear. We investigate whether interview-informed generative agents can simulate user responses in concept testing scenarios. Using in-depth workflow interviews with knowledge workers, we created personalized agents and compared their evaluations of novel AI concepts against the same participants’ responses. Our results show that agents are distribution-calibrated but identity-imprecise: they fail to replicate the specific individual they are grounded in, yet approximate population-level response distributions. These findings highlight both the potential and the limits of LLM simulation in design research. While unsuitable as a substitute for individual-level insights, simulation may provide value for early-stage concept screening and iteration, where distributional accuracy suffices. We discuss implications for integrating simulation responsibly into product development workflows.

CHI ’25, Yokohama, Japan

Nishant Balepur, Alexa Siu, Nedim Lipka, Franck Dernoncourt, Tong Sun, Jordan Lee Boyd-Graber, Puneet Mathur. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

PDF documents are usually not born-accessible, and so document authors need to put in additional work (remediation) to make them accessible for people with disabilities. Unfortunately, this step is often overlooked and hard to execute, resulting in a large number of inaccessible PDF documents on the internet. Previously, there have been research efforts to investigate potential solutions for remediating PDF documents for accessibility. However, most of the existing research focuses on accessibility of long or scientific PDF documents meant for passive reading. PDF documents come in different types, and this research project focuses on a distinct type of PDF document-forms-where the user is required to interact with the PDF document and enter data. Through our research work we identified that the PDF form remediation process is non-intuitive, repetitive, and overwhelming due to the high-information density of PDF forms, and existing research and tools do not yet address the challenges. Our research work culminated in the creation of a tool - FormA11y - that addresses these challenges by making the repetitive and painstaking process of form remediation easier. To evaluate the effectiveness and efficiency of FormA11y against the industry standard tool - Adobe Acrobat - for PDF form remediation, we performed a within-subject user study with 20 participants. With FormA11y, users remediated forms 2.8 times faster while creating more accurately accessible PDF forms.

The `pre-train, prompt, predict' paradigm of large language models (LLMs) has achieved remarkable success in open-domain question answering (OD-QA). However, few works explore this paradigm in multi-document question answering (MD-QA), a task demanding a thorough understanding of the logical associations among the contents and structures of documents. To fill this crucial gap, we propose a Knowledge Graph Prompting (KGP) method to formulate the right context in prompting LLMs for MD-QA, which consists of a graph construction module and a graph traversal module. For graph construction, we create a knowledge graph (KG) over multiple documents with nodes symbolizing passages or document structures (e.g., pages/tables), and edges denoting the semantic/lexical similarity between passages or document structural relations. For graph traversal, we design an LLM-based graph traversal agent that navigates across nodes and gathers supporting passages assisting LLMs in MD-QA. The constructed graph serves as the global ruler that regulates the transitional space among passages and reduces retrieval latency. Concurrently, the graph traversal agent acts as a local navigator that gathers pertinent context to progressively approach the question and guarantee retrieval quality. Extensive experiments underscore the efficacy of KGP for MD-QA, signifying the potential of leveraging graphs in enhancing the prompt design and retrieval augmented generation for LLMs. Our code: https://github.com/YuWVandy/KG-LLM-MDQA.

Knowledge workers often need to extract and analyze information from a collection of documents to solve complex information tasks in the workplace, e.g., hiring managers reviewing resumes or analysts assessing risk in contracts. However, foraging for relevant information can become tedious and repetitive over many documents and criteria of interest. We introduce Marco, a mixed-initiative workspace supporting sensemaking over diverse business document collections. Through collection-centric assistance, Marco reduces the cognitive costs of extracting and structuring information, allowing users to prioritize comparative synthesis and decision making processes. Users interactively communicate their information needs to an AI assistant using natural language and compose schemas that provide an overview of a document collection. Findings from a usability study (n=16) demonstrate that when using Marco, users complete sensemaking tasks 16% more quickly, with less effort, and without diminishing accuracy. A design probe with seven domain experts identifies how Marco can benefit various real-world workflows.

We introduce MoiréWidgets, a novel approach for tangible interaction that harnesses the Moiré effect—a prevalent optical phenomenon—to enable high-precision event detection on physical widgets. Unlike other electronics-free tangible user interfaces which require close coupling with external hardware, MoiréWidgets can be used at greater distances while maintaining high-resolution sensing of interactions. We define a set of interaction primitives, e.g., buttons, sliders, and dials, which can be used as standalone objects or combined to build complex physical controls. These consist of 3D printed structural mechanisms with patterns printed on two layers—one on paper and the other on a plastic transparency sheet—which create a visual signal that amplifies subtle movements, enabling the detection of user inputs. Our technical evaluation shows that our method outperforms standard fiducial markers and maintains sub-millimeter accuracy at 100 cm distance and wide viewing angles. We demonstrate our approach by creating an audio console and indicate how our approach could extend to other domains.

Many academic subjects are inaccessible for students who are blind or have low vision (BLV) due to the prevalence of visual aids to represent concepts. Interactive devices offer promise as creative tools for teachers of the visually impaired (TVIs) as they can support real-time iteration of adapted learning materials, display changing information, and provide embodied learning experiences for BLV students. We conducted semi-structured interviews with 5 educators (all have TVI experience) and identified their considerations when creating adaptations, attitudes towards technology, and perspectives on existing barriers to access. Our findings reveal and reaffirm unresolved challenges in the adaptation process as well as offer insights into key factors that must be considered when selecting the type of adaptation. From these findings, we formulate design recommendations for interactive tools that support TVIs in creating effective adaptations for BLV students.

The emergence of contract specific programming languages has struggled to translate into widespread adoption of computable contracts due largely to high conversion costs. In this work, we present the first system for converting natural language contracts into code through the extraction of key entities, relationships, and formulas into a graph representation called the Obligation Logic Graph (OLG). This approach allows the semantic meaning of contract obligations, including dependencies between obligations, to be captured through the OLG and mapped to code downstream. We also introduce OLG extraction as a new joint entity and relation prediction task for legal contracts, and present the Contract-OLG dataset, consisting of 1,876 contract provisions, 18,597 entities and 18,170 relationships. We perform detailed experiments to understand the capabilities of state-of-the-art Transformer and graph-based models at completing these tasks, and identify where there is currently a significant gap between human expert and machine performance, particularly for relation extraction.

For sighted users, computer graphics and virtual reality allow them to model and perceive imaginary objects and worlds. However, these approaches are inaccessible to blind and visually impaired (BVI) users, since they primarily rely on visual feedback. To this end, we introduce ShapeSonic, a system designed to convey vivid 3D shape perception using purely audio feedback or sonification. ShapeSonic tracks users’ fingertips in 3D and provides real-time sound feedback (sonification). The shape’s geometry and sharp features (edges and corners) are expressed as sounds whose volumes modulate according to fingertip distance. ShapeSonic is based on a mass-produced, commodity hardware platform (Oculus Quest). In a study with 15 sighted and 6 BVI users, we demonstrate the value of ShapeSonic in shape landmark localization and recognition. ShapeSonic users were able to quickly and relatively accurately “touch” points on virtual 3D shapes in the air.

Story ideation is a critical part of the story-writing process. It is challenging to support computationally due to its exploratory and subjective nature. Tropes, which are recurring narrative elements across stories, are essential in stories as they shape the structure of narratives and our understanding of them. In this paper, we propose to use tropes as an intermediate representation of stories to approach story ideation. We present TaleStream, a canvas system that uses tropes as building blocks of stories while providing steerable suggestions of story ideas in the form of tropes. Our trope suggestion methods leverage data from the tvtropes.org wiki. We find that 97% of the time, trope suggestions generated by our methods provide better story ideation materials than random tropes. Our system evaluation suggests that TaleStream can support writers’ creative flow and greatly facilitates story development. Tropes, as a rich lexicon of narratives with available examples, play a key role in TaleStream and hold promise for story-creation support systems.

We increasingly rely on up-to-date, data-driven graphs to understand our environments and make informed decisions. However, many of the methods blind and visually impaired users (BVI) rely on to access data-driven information do not convey important shape-characteristics of graphs, are not refreshable, or are prohibitively expensive. To address these limitations, we introduce two refreshable, 1-DOF audio-haptic interfaces based on haptic cues fundamental to object shape perception. Slide-tone uses finger position with sonification, and Tilt-tone uses fingerpad contact inclination with sonification to provide shape feedback to users. Through formative design workshops (n = 3) and controlled evaluations (n = 8), we found that BVI participants appreciated the additional shape information, versatility, and reinforced understanding these interfaces provide; and that task accuracy was comparable to using interactive tactile graphics or sonification alone. Our research offers insight into the benefits, limitations, and considerations for adopting these haptic cues into a data visualization context.

Online data visualizations play an important role in informing public opinion but are often inaccessible to screen reader users. To address the need for accessible data representations on the web that provide direct, multimodal, and up-to-date access to the data, we investigate audio data narratives –which combine textual descriptions and sonification (the mapping of data to non-speech sounds). We conduct two co-design workshops with screen reader users to define design principles that guide the structure, content, and duration of a data narrative. Based on these principles and relevant auditory processing characteristics, we propose a dynamic programming approach to automatically generate an audio data narrative from a given dataset. We evaluate our approach with 16 screen reader users. Findings show with audio narratives, users gain significantly more insights from the data. Users describe data narratives help them better extract and comprehend the information in both the sonification and description.

The ability to effectively read and interpret tactile graphics and charts is an essential part of a tactile learner’s path to literacy, but is a skill that requires instruction and training. Many teachers of the visual impaired (TVIs) report that blind and visually impaired students have trouble interpreting graphics independently without individual instruction. We present PantoGuide, a low-cost system that provides audio and haptic guidance, via skin-stretch feedback to the dorsum of a user’s hand while the user explores a tactile graphic overlaid on a touchscreen. This system allows programming of haptic guidance patterns and cues for tactile graphics that can be experienced by students learning remotely or that can be reviewed by a student independently. We propose two teaching scenarios (synchronous and asynchronous) and two guidance interactions (point-to-point and continuous) that the device can support – and demonstrate their use in a set of applications we co-design with one co-author who is blind and a tactile graphics user.

As data visualization has become increasingly important in our society, many challenges prevent people who are blind and visually impaired (BVI) from fully engaging with data and data graphics. For example, tactile data representations are commonly used by BVI people to explore spatial graphics, but it is difficult for BVI people to construct and understand tactile representations without prior training or expert assistance. In this work, we adopt a constructive visualization framework of using simple and versatile tokens to engage non-experts in the construction of tactile data representations. We present preliminary results of how participants chose to interpret and create tactile data representations and the preferred haptic exploratory procedures used for retrieving information. All participants used similar construction strategies and converged upon 3D compact spatial forms to retrieve and display analytical information. These insights can inform future data visualization authoring and consumption tools that users of more diverse skill backgrounds can effectively navigate.

Social media platforms facilitate communication through sharing photos and videos. The abundance of visual content creates accessibility issues, particularly for people who are blind or have low vision. While assistive technologies like screen readers can help when alt-text for images is provided, synthesized voices lack the human element that is important for social interaction. Here, we investigate when it makes the most sense to use human narration as opposed to a screen reader to describe photos in a social media context. We explore the effects of voice familiarity (i.e., whether you hear the voice of someone you know) and the perspective of the description (i.e., first vs. third person point-of-view (POV)). Preliminary study suggests that users prefer hearing from a person they know when the content is described in first person POV, whereas synthesized voice is preferred for content described in third person POV.

Current Virtual Reality (VR) technologies focus on rendering visuospatial effects, and thus are inaccessible for blind or low vision users. We examine the use of a novel white cane controller that enables navigation without vision of large virtual environments with complex architecture, such as winding paths and occluding walls and doors. The cane controller employs a lightweight three-axis brake mechanism to provide large-scale shape of virtual objects. The multiple degrees-of-freedom enables users to adapt the controller to their preferred techniques and grip. In addition, surface textures are rendered with a voice coil actuator based on contact vibrations; and spatialized audio is determined based on the progression of sound through the geometry around the user. We design a scavenger hunt game that demonstrates how our device enables blind users to navigate a complex virtual environment. Seven out of eight users were able to successfully navigate the virtual room (6x6m) to locate targets while avoiding collisions. We conclude with design consideration on creating immersive non-visual VR experiences based on user preferences for cane techniques, and cane material properties.

We present PIVOT, a wrist-worn haptic device that renders virtual objects into the user's hand on demand. Its simple design comprises a single actuated joint that pivots a haptic handle into and out of the user's hand, rendering the haptic sensations of grasping, catching, or throwing an object anywhere in space. Unlike existing hand-held haptic devices and haptic gloves, PIVOT leaves the user's palm free when not in use, allowing users to make unencumbered use of their hand. PIVOT also enables rendering forces acting on the held virtual objects, such as gravity, inertia, or air-drag, by actively driving its motor while the user is firmly holding the handle. When wearing a PIVOT device on both hands, they can add haptic feedback to bimanual interaction, such as lifting larger objects. In our user study, participants (n=12) evaluated the realism of grabbing and releasing objects of different shape and size with mean score 5.19 on a scale from 1 to 7, rated the ability to catch and throw balls in different directions with different velocities (mean=5.5), and verified the ability to render the comparative weight of held objects with 87% accuracy for ~100g increments.

The Maker movement promises access to activities from crafting to digital fabrication for anyone to invent and customize technology. But people with disabilities, who could benefit from Making, still encounter significant barriers to do so. In response, we share our personal experiences Making nonvisually and supporting its instruction. Specifically, we draw on examples from a series of workshops where we introduced Arduino to blind hobbyists and guided assembly of an accessible voltmeter prototype [24]. In so doing, we offer future directions for accessible Making research and application.

Skimming new code with a screen reader can be a time-consuming task for blind and visually impaired programmers. Screen readers aid with code navigation, but dictate code line-by-line and read spaces and tabs individually. This often provides more information than is needed. In this work, we present the Tactile Code Skimmer (TCS), a tool to aid blind and visually impaired programmers with skimming code. The device physically reflects the indentation levels of code with actuated slide potentiometers, thus helping reduce the "hearing load" that often accompanies screen readers. We describe the TCS design and implementation. Based on feedback from participants obtained through demos and an in-depth session, we discuss some considerations for tactile tools that aid with code skimming.

Affordable rapid 3D printing technologies have become key enablers of the Maker Movement by giving individuals the ability to create physical finished products. However, existing computer-aided design (CAD) tools that allow authoring and editing of 3D models are mostly visually reliant and limit access to people with blindness and visual impairment (BVI). Through a series of co-design sessions with three blind users of mixed programming ability, we identify accessibility challenges in existing 3D modelling scripting tools and design interactions to support dynamic feedback of scripts using a 2.5D tactile shape display. With these insights, we implement shapeCAD. Interacting with shapeCAD, BVI users are able to leverage the low resolution output from a 2.5D shape display to complement programming of 3D models. shapeCAD allows users to haptically explore and modify existing models, and to author new models. We further validate usability and user experience through an evaluation with five BVI programmers. In a short period of time, novices were able to design a range of new objects. BVI users can bring a valuable perspective to design and it is imperative to increase accessibility in tools that enable this community to also participate as designers.

Affordable rapid 3D printing technologies have become a key enabler in the maker movement by giving individuals the ability to create physical finished products. However, existing computer-aided design (CAD) tools that allow authoring and editing of 3D models are mostly visually reliant and limit access for people who are blind or visually impaired. We propose an accessible CAD workflow where 3D models are generated through OpenSCAD, a script-based 3D modeling tool, and rendered at interactive speeds in an actuated 2.5D shape display. We report preliminary findings on a case study with one blind user. Based on our observations, we frame design imperatives on interactions that might be important in future accessible CAD systems with tactile output.

We explore interactions enabled by 2D spatial manipulation and self-actuation of a tabletop shape display. To explore these interactions, we developed shapeShift, a compact, high-resolution (7 mm pitch), mobile tabletop shape display. shapeShift can be mounted on passive rollers allowing for bimanual interaction where the user can freely manipulate the system while it renders spatially relevant content. shapeShift can also be mounted on an omnidirectional-robot to provide both vertical and lateral kinesthetic feedback, display moving objects, or act as an encountered-type haptic device for VR. We present a study on haptic search tasks comparing spatial manipulation of a shape display for egocentric exploration of a map versus exploration using a fixed display and a touch pad. Results show a 30% decrease in navigation path lengths, 24% decrease in task time, 15% decrease in mental demand and 29% decrease in frustration in favor of egocentric navigation.