It is challenging to generate the code for a complete user interface using a Large Language Model (LLM). User interfaces are complex and their implementations often consist of multiple, inter-related files that together specify the contents of each screen, the navigation flows between the screens, and the data model used throughout the application. It is challenging to craft a single prompt for an LLM that contains enough detail to generate a complete user interface, and even then the result is frequently a single large and intricate file that contains all of the generated screens. In this paper, we introduce Athena, a prototype application generation environment that demonstrates how the use of shared intermediate representations, including an app storyboard, data model, and GUI skeletons, can help a developer work with an LLM in an iterative fashion to craft a complete user interface. These intermediate representations also scaffold the LLM’s code generation process, producing organized and structured code in multiple files while limiting errors. We evaluated Athena with a user study with 12 developers. Participants appreciated Athena’s support for prototyping multi-screen iOS apps, acknowledged that the intermediate representations improved their control and understanding of generated code, and discussed the limitations of the system and potential directions for improvement.

Large language model (LLM)-based computer use agents execute user commands by interacting with available UI elements, but little is known about how users want to interact with these agents or what design factors matter for their user experience (UX). We conducted a two-phase study to map the UX design space for computer use agents. In Phase 1, we reviewed existing systems to develop a taxonomy of UX considerations, then refined it through interviews with eight UX and AI practitioners. The resulting taxonomy included categories such as user prompts, explainability, user control, and users’ mental models, with corresponding subcategories and example design features. In Phase 2, we ran a Wizard-of-Oz study with 20 participants, where a researcher acted as a web-based computer use agent and probed user reactions during normal, error-prone and risky execution. We used the findings to validate the taxonomy from Phase 1 and deepen our understand of the design space by identifying the connections between design areas and divergence in user needs and scenarios. Our taxonomy and empirical insights provide a map for developers to consider different aspects of user experience in computer use agent design and to situate their designs within users’ diverse needs and scenarios.

Creating 2D animations is challenging because it requires iterative refinement of movement and transitions across multiple elements within a scene. We explored the potential of LLMs to support animation design by first identifying current challenges in formative interviews with animation creators, and then developing a design probe and LLM-based animation design tool called Keyframer. From user-provided graphics and natural language prompts, Keyframer generates animation code, enables users to preview rendered animations inline, and supports direct edits for iterative design refinement. We utilized this design probe to uncover user prompting styles for describing animation in natural language and observe user strategies for iterating on animations in an exploratory user study with 13 novices and experts in animation design and programming. Through this study, we contribute a categorization of prompting styles users employed for specifying animation goals, along with design insights on supporting iterative refinement of animations through the combination of direct editing and natural language interfaces.

Developers and quality assurance testers often rely on manual testing to test accessibility features throughout the product lifecycle. Unfortunately, manual testing can be tedious, often has an overwhelming scope, and can be difficult to schedule amongst other development milestones. Recently, Large Language Models (LLMs) have been used for a variety of tasks including automation of UIs. However, to our knowledge, no one has yet explored the use of LLMs in controlling assistive technologies for the purposes of supporting accessibility testing. In this paper, we explore the requirements of a natural language based accessibility testing workflow, starting with a formative study. From this we build a system that takes a manual accessibility test instruction in natural language (e.g., “Search for a show in VoiceOver”) as input and uses an LLM combined with pixel-based UI Understanding models to execute the test and produce a chaptered, navigable video. In each video, to help QA testers, we apply heuristics to detect and flag accessibility issues (e.g., Text size not increasing with Large Text enabled, VoiceOver navigation loops). We evaluate this system through a 10-participant user study with accessibility QA professionals who indicated that the tool would be very useful in their current work and performed tests similarly to how they would manually test the features. The study also reveals insights for future work on using LLMs for accessibility testing.

Programmers frequently engage with machine learning tutorials in computational notebooks and have been adopting code generation technologies based on large language models (LLMs). However, they encounter difficulties in understanding and working with code produced by LLMs. To mitigate these challenges, we introduce a novel workflow into computational notebooks that augments LLM-based code generation with an additional ephemeral UI step, offering users UI scaffolds as an intermediate stage between user prompts and code generation. We present this workflow in Biscuit, an extension for JupyterLab that provides users with ephemeral UIs generated by LLMs based on the context of their code and intentions, scaffolding users to understand, guide, and explore with LLMgenerated code. Through a user study where 10 novices used Biscuit for machine learning tutorials, we found that Biscuit offers users representations of code to aid their understanding, reduces the complexity of prompt engineering, and creates a playground for users to explore different variables and iterate on their ideas.

While revolutionary AI-powered code generation tools have been rising rapidly, we know little about how and how to help software developers form appropriate trust in those AI tools. Through a two-phase formative study, we investigate how online communities shape developers’ trust in AI tools and how we can leverage community features to facilitate appropriate user trust. Through interviewing 17 developers, we find that developers collectively make sense of AI tools using the experiences shared by community members and leverage community signals to evaluate AI suggestions. We then surface design opportunities and conduct 11 design probe sessions to explore the design space of using community features to support user trust in AI code generation systems. We synthesize our findings and extend an existing model of user trust in AI technologies with sociotechnical factors. We map out the design considerations for integrating user community into the AI code generation experience.

In this paper, we present a new visual block-based programming system designed for children to process, analyze, and visualize data. We introduce the system and describe how it was used during a series of 7 workshops with 27 children. During the workshops, children played the role of investigators and followed a storyline as part of the system to conduct data analyses to help the story’s protagonist locate a missing family member. We present our findings as a framework of computational data literacy that builds on the dimensions of Computational Thinking proposed by Brennan and Resnick [8], with a focus on aspects that are specific to using programming for data processing, analysis, and visualization. We conclude with a series of recommendations for future designers of systems to support the development of computational data literacy.

Through a mixed-method analysis of data from Scratch, we examine how novices learn to program with simple data structures by using community-produced learning resources. First, we present a qualitative study that describes how community-produced learning resources create archetypes that shape exploration and may disadvantage some with less common interests. In a second quantitative study, we find broad support for this dynamic in several hypothesis tests. Our findings identify a social feedback loop that we argue could limit sources of inspiration, pose barriers to broadening participation, and confine learners’ understanding of general concepts. We conclude by suggesting several approaches that may mitigate these dynamics.

Ruijia Cheng, Alison Smith-Renner, Ke Zhang, Joel Tetreault, Alejandro Jaimes-Larrarte. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Vivian Lai, Alison Smith-Renner, Ke Zhang, Ruijia Cheng, Wenjuan Zhang, Joel Tetreault, Alejandro Jaimes-Larrarte. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Feedback is a critical piece of the creative process. Prior CSCW research has invented peer-based and crowd-based systems that exchange feedback between online strangers at scale. However, creators run into socio-psychological challenges when engaging in online critique exchange with people they have never met. In this study, we step back and take a different approach to investigate online feedback by casting our attention to creators in online affinity spaces, where feedback exchange happens naturally as part of the interest-driven participatory culture. We present an interview study with 29 fanfiction writers that investigated how they seek feedback online, and how they identified and built connections with feedback providers. We identify four distinct feedback practices and unpack the social needs in feedback exchange. Our findings surface the importance of affinity and trust in online feedback exchange and illustrate how writers built relationships with feedback providers in public and private online spaces. Inspired by the powerful stories we heard about connection and feedback, we conclude with a series of design considerations for future feedback systems-namely addressing a range of social needs in feedback, helping feedback seekers signal interests and identity, supporting authentic relationships in feedback exchange, and building inclusive, safe community spaces for feedback.

Although informal online learning communities have proliferated over the last two decades, a fundamental question remains: What are the users of these communities expected to learn? Guided by the work of Etienne Wenger on communities of practice, we identify three distinct types of learning goals common to online informal learning communities: the development of domain skills, the development of identity as a community member, and the development of community-specific values and practices. Given these goals, what is the best way to support learning? Drawing from previous research in social computing, we ask how different types of legitimate peripheral participation by newcomers-contribution to core tasks, engagement with practice proxies, social bonding, and feedback exchange-may be associated with these three learning goals. Using data from the Scratch online community, we conduct a quantitative analysis to explore these questions. Our study contributes both theoretical insights and empirical evidence on how different types of learning occur in informal online environments.

Knowledge building is a prevalent feature in open online systems, but it is challenging to motivate participants to contribute and to maintain quality in the participants' contributions. Open online competitions, where participants compete for prizes with knowledge artifacts, offer a potential design model for online systems to incentivize community knowledge building activities. However, while there is evidence that participants contribute to public knowledge and share it during competitions, it remains unclear how and why they do so. In this study, through interviews with 14 participants in Kaggle Competitions, we investigate participants' motivation, practices, and challenges when contributing to community knowledge under a competitive structure. We find that competitive mechanisms impact expert and beginner participants very differently in their public knowledge building behaviors. Experts contribute to shared knowledge in order to compete for reputation, while they tend to form their own niches and only share knowledge artifacts that are abstract and not usable by less experienced participants. Beginners are often driven away from contributing to shared knowledge because of their vulnerable social image. We leverage Scardamalia's framework for Knowledge Building Communities to discuss the different challenges and opportunities that competitive design brings to expert and beginner participants. We offer design implications for effectively implementing competitive mechanisms that could benefit both expert and beginner participants in future knowledge building systems.

Creative workers frequently turn to online critique communities for feedback on their work. While past research has focused primarily on how to yield better feedback from providers, less is known about the strategies feedback seekers use to engage providers and request feedback. We present two studies to explore the feedback exchange dynamics between feedback requesters and providers in the subreddit community, r/design\_critiques. In Study 1, we interviewed 12 community members and found that while creators have strategies to request feedback, they expressed uncertainty about whether and how to include details about the design context, personal background, and specific feedback needs. In Study 2, through a mixed-method analysis, we identified how specific request strategies impact the quantity and quality of community feedback, and found several key, but undervalued strategies: signaling as a novice, critiquing one's own design, and providing design variants. These strategies led to better community response, but were rarely used. We offer design implications around how to leverage these insights to improve online feedback exchange