Generative AI (GenAI) tools are increasingly pervasive, pushing instructors to redesign how students use GenAI tools in coursework. We conceptualize this work as emergency pedagogical design: reactive, indirect efforts by instructors to shape student-AI interactions without control over commercial interfaces. To understand practices of lead users conducting emergency pedagogical design, we conducted interviews (n=13) and a survey (n=169) of computing instructors. These instructors repeatedly encountered five barriers: fragmented buy-in for revising courses; policy crosswinds from non-prescriptive institutional guidance; implementation challenges as instructors attempt interventions; assessment misfit as student-AI interactions are only partially visible to instructors; and lack of resources, including time, staffing, and paid tool access. We use these findings to present emergency pedagogical design as a distinct design setting for HCI and outline recommendations for HCI researchers, academic institutions, and organizations to effectively support instructors in adapting courses to GenAI.

The rise of Large Language Models has intensified the need for reliable assessments of foundational programming skills in introductory computer science courses. While frequent, low-stakes testing is an effective pedagogical strategy, its adoption is often slowed by the need for institutional infrastructure like a Computer-Based Testing Facility, which instructors may lack or find too inflexible. This experience report presents a practical, instructor-driven model for Skill Tests: weekly, 10-minute, proctored, computer-based assessments run by course staff in any campus room using student laptops. We provide a logistical blueprint refined over five offerings of a large CS1 course, detailing our strategies for question design, student scheduling, and staff management. Our findings show this model is effective: replacing a midterm with Skill Tests reduced student anxiety, with 91% of students preferring the new format. Student performance on the final exam remained consistent, indicating that knowledge synthesis was not compromised. This paper offers educators a framework for deploying frequent, low-stakes assessments without dedicated institutional resources.

One key challenge for instructors is creating high-quality educational content, such as programming practice questions for introductory programming courses. While Large Language Models (LLMs) show promise for this task, their output quality can be inconsistent, and it is often unclear how to systematically improve their performance. In this experience report, we present the development process for ContentGen, an open-source tool that generates programming questions within the context of data science instructional materials. We describe our process of designing the tool and iteratively improving the tool through prompt engineering. To evaluate our changes, we designed and open-sourced a dataset of 91 test cases based on our course materials and developed three metrics to assess the generated questions: Correctness, Contextual Fit, and Coherence. We compare three prompting strategies and find that providing detailed instructions and an automatically generated summary of recently covered instructional materials to the LLM substantially improves the quality of the generated questions across our metrics. A usability study with six data science instructors further suggests that our final prototype is perceived as usable and effective. Our work contributes a case study of evidence-based prompt engineering for an educational tool and offers a practical approach for instructors and tool designers to evaluate and enhance LLM-based content generation.

Data scientists often need to read and understand messy and undocumented code that relies on large software libraries. What makes data science experts more effective than novices at this task? To understand expert practices, we conducted a think-aloud study where 4 novice and 5 expert data scientists reasoned about an unfamiliar data analysis script with realistic complexity that used the Python pandas library. Surprisingly, familiarity of the pandas package had relatively minor importance for experts. Instead, experts consistently performed three practices that novices did not: experts examined the data in detail rather than fixating on surface-level code features; experts consistently verified their assumptions about how the data was transformed; and experts navigated lengthy program outputs in a goal-directed way. Using these findings, we provide a practical set of guidelines for data science pedagogy and for future tools to support data science learners.

As data science and artificial intelligence continue to impact society, more and more people are learning how to manipulate data with code. To support these learners, program visualization tools automatically generate diagrams to show how code transforms data, in contrast to tools based on large language models (LLMs) that primarily focus on textual explanations. Although program visualization tools are popular among instructors, do novices find these tools usable and useful for data science programs that often manipulate datasets with many rows? To address this, we evaluate a popular, publicly available tool that generates diagrams for Python pandas code through a randomized, in-lab usability study with 17 data science novices. Despite minimal instruction on how to use the tool, novices found that program visualizations increased their confidence in comprehending and debugging code. In addition, even though the tool sometimes produced diagrams with many visual elements, participant performance on the study tasks was not negatively impacted. These findings suggest design guidelines for program visualization tools to help manage cognitive load for data science novices. To our knowledge, this is the first empirical study that investigates how novices use program visualization tools to understand code that manipulates data tables, and suggests a future where novices can use automatically generated diagrams as a complement to LLM tools for effectively understanding unfamiliar programs in data science.

Over the past few years, millions of people have been using LLM-based AI tools to aid in programming, data analysis, and software engineering tasks. These AI coding assistants range from specialized tools like GitHub Copilot to general-purpose chatbots like Claude. In parallel, academics have published dozens of papers on forward-looking prototypes to expand our collective thinking beyond present-day industry trends. However, despite rapid advances in both sectors in recent years, we still lack an understanding of how their designs relate to one another and what tradeoffs are commonly made. At this key moment in 2025 when design patterns are starting to emerge, it is important to zoom out to see the forest instead of the trees. To do so, we performed the first comprehensive design analysis of 90 LLM-based AI coding assistants. We categorized the feature sets of 58 industry products and 32 academic projects, then formulated a design space that captures key variations in their user experiences. Our design space covers $\mathbf{1 0}$ dimensions related to UI modalities, system inputs, capabilities, and outputs. We use this design space to reveal trends in both industry and academic projects across three eras ranging from autocomplete to chat to agent-based interfaces. Lastly, to address the question of who the target users of these tools are, we present six user personas whose preferences lie in different regions of our design space: professional software engineers, HCI researchers and hobbyist programmers, UX designers, conversational programmers (e.g., product managers and marketers), data scientists, and students.

Generative AI (GenAI) has seen great advancements in the past two years and the conversation around adoption is increasing. Widely available GenAI tools are disrupting classroom practices as they can write and explain code with minimal student prompting. While most acknowledge that there is no way to stop students from using such tools, a consensus has yet to form on how students should use them if they choose to do so. At the same time, researchers have begun to introduce new pedagogical tools that integrate GenAI into computing curricula. These new tools offer students personalized help or attempt to teach prompting skills without undercutting code comprehension. This working group aims to detail the current landscape of education-focused GenAI tools and teaching approaches, present gaps where new tools or approaches could appear, identify good practice-examples, and provide a guide for instructors to utilize GenAI as they continue to adapt to this new era.

Over the past year (2022–2023), recently-released AI tools such as ChatGPT and GitHub Copilot have gained significant attention from computing educators. Both researchers and practitioners have discovered that these tools can generate correct solutions to a variety of introductory programming assignments and accurately explain the contents of code. Given their current capabilities and likely advances in the coming years, how do university instructors plan to adapt their courses to ensure that students still learn well? To gather a diverse sample of perspectives, we interviewed 20 introductory programming instructors (9 women + 11 men) across 9 countries (Australia, Botswana, Canada, Chile, China, Rwanda, Spain, Switzerland, United States) spanning all 6 populated continents. To our knowledge, this is the first empirical study to gather instructor perspectives about how they plan to adapt to these AI coding tools that more students will likely have access to in the future. We found that, in the short-term, many planned to take immediate measures to discourage AI-assisted cheating. Then opinions diverged about how to work with AI coding tools longer-term, with one side wanting to ban them and continue teaching programming fundamentals, and the other side wanting to integrate them into courses to prepare students for future jobs. Our study findings capture a rare snapshot in time in early 2023 as computing instructors are just starting to form opinions about this fast-growing phenomenon but have not yet converged to any consensus about best practices. Using these findings as inspiration, we synthesized a diverse set of open research questions regarding how to develop, deploy, and evaluate AI coding tools for computing education.

Instructors who teach large-scale technical courses, especially on data science and programming, must do a large amount of logistical work when updating their courses. All of this behind-the-scenes labor takes time away from the pedagogically-meaningful work of teaching students. Over the past five years, the authors of this paper have created and updated eight courses for an undergraduate data science program that serves over 2,000 students per year. We present four case studies from our teaching experiences that highlight major challenges in maintaining and updating technical courses: 1) There were intricate dependencies between course materials, so making updates to one part of the course would require updating many other parts. 2) We needed to maintain several variants of course materials such as assignments. 3) We wrote large amounts of ad-hoc custom software infrastructure to manage logistics. 4) We could not easily reuse software written by others. Our case studies point to design ideas for instructor-oriented tools that can reduce the logistical complexities of teaching at scale, thus letting instructors focus on the substance of teaching rather than on mundane logistics.

Over the past decade, data science courses have been growing more popular across university campuses. These courses often involve a mix of programming and statistics and are taught by instructors from diverse backgrounds. In our experiences launching a data science program at a large public U.S. university over the past four years, we noticed one central tension within many such courses: instructors must finely balance how much computing versus statistics to teach in the limited available time. In this experience report, we provide a detailed firsthand reflection on how we have personally balanced these two major topic areas within several offerings of a large introductory data science course that we taught and wrote an accompanying textbook for; our course has served several thousand students over the past four years. We present three case studies from our experiences to illustrate how computer science and statistics instructors approach data science differently on topics ranging from algorithmic depth to modeling to data acquisition. We then draw connections to deeper tradeoffs in data science to help guide instructors who design interdisciplinary courses. We conclude by suggesting ways that instructors can incorporate both computer science and statistics perspectives to improve data science teaching.

End-user programmers opportunistically copy-and-paste code snippets from colleagues or the web to accomplish their tasks. Unfortunately, these snippets often don’t work verbatim, so these people—who are non-specialists in the programming language—make guesses and tweak the code to understand and apply it successfully. To support their desired workflow and facilitate tweaking and understanding, we built a prototype tool, TweakIt, that provides users with a familiar live interaction to help them understand, introspect, and reify how different code snippets would transform their data. Through a usability study with 14 data analysts, participants found the tool to be useful to understand the function of otherwise unfamiliar code, to increase their confidence about what the code does, to identify relevant parts of code specific to their task, and to proactively explore and evaluate code. Overall, our participants were enthusiastic about incorporating TweakIt in their own day-to-day work.

Computational notebooks such as Jupyter are now used by millions of data scientists, machine learning engineers, and computational researchers to do exploratory and end-user programming. In recent years, dozens of different notebook systems have been developed across academia and industry. However, we still lack an understanding of how their individual designs relate to one another and what their tradeoffs are. To provide a holistic view of this rapidly-emerging landscape, we performed, to our knowledge, the first comprehensive design analysis of dozens of notebook systems. We analyzed 60 notebooks (16 academic papers, 29 industry products, and 15 experimental/R&D projects) and formulated a design space that succinctly captures variations in system features. Our design space covers 10 dimensions that include diverse ways of importing data, editing code and prose, running code, and publishing notebook outputs. We conclude by suggesting ways for researchers to push future projects beyond the current bounds of this space.

Misunderstandings of science affect many lives. Novices commonly misunderstand explanations by overly relying on surface details instead of evaluating underlying logic. Prior work has found adding a patina of neuroscience leads readers towards positively assessing explanations. How might we help people better understand science explanations? A between-subjects experiment tested whether asking readers to reconstruct experiments leads them to focus more on underlying logic. Participants relied less on irrelevant surface details when reconstructing experiments. However, this did not impact their subsequent assessment of explanations. Our results suggest that reconstruction is a useful strategy for understanding explanations but is not readily transferred towards evaluating explanations.