Knowledge workers increasingly rely on conversational AI for sense-making tasks (e.g., conducting market analysis), yet must repeatedly reconstruct context and intent to meet their goals. A formative study (N=10) showed that workflow reuse with AI often failed. Current tools either only remember preferences or enforce rigid, predefined workflows—neither adapts to evolving goals. We present ThinkFlow, a system that maintains a dynamic common ground through a cognitive workflow schema, enabling users to express intent and AI to adapt and reuse workflows across contexts. An expert-rating study shows that the schema can accurately capture the collocutor’s reasoning process, and when reused for a similar task, improves the AI’s responses compared to when the schema isn’t present. A user study with eight knowledge workers demonstrates that ThinkFlow supports awareness of evolving workflows, intent expression, and flexible application across contexts.

Despite growing interest in using LLMs to generate feedback on students’ writing, little is known about how students respond to AI-mediated versus human-provided feedback. We address this gap through a randomized controlled trial in a large introductory economics course (N=354), where we introduce and deploy FeedbackWriter—a system that generates AI suggestions to teaching assistants (TAs) while they provide feedback on students’ knowledge-intensive essays. TAs have the full capacity to adopt, edit, or dismiss the suggestions. Students were randomly assigned to receive either handwritten feedback from TAs (baseline) or AI-mediated feedback where TAs received suggestions from FeedbackWriter. Students revise their drafts based on the feedback, which is further graded. In total, 1,366 essays were graded using the system. We found that students receiving AI-mediated feedback produced significantly higher-quality revisions, with gains increasing as TAs adopted more AI suggestions. TAs found the AI suggestions useful for spotting gaps and clarifying rubrics.

Background and Context. There are growing efforts to increase the percentage of secondary students who take computing courses. However, U.S. women identifying as Black/African American, Hispanic/Latina, and/or Native American remain underrepresented in computing and face persistent challenges in learning to program. Moreover, it remains underexplored what types of programming learning support minoritized high school women desire.

Figure 1: Rubikon is an intelligent tutoring system for Rubik's Cube learning.(a) The foundational design of Rubikon is an AR setup, where learners manipulate a physical cube with ArUco markers attached to each square, and pose a camera towards the cube to enable tracking and rendering.With this setup, learners see a rendered Rubik's Cube on a display while manipulating the physical cube in their hands.(b) Through AR rendering, Rubikon automatically generates new configurations of the Rubik's Cube for the user to practice unmastered skills.Rubikon detects the status of the cube to infer user behavior and provide immediate feedback and hints.(c) Rubikon supports the learning of a 3D physical task by integrating key design principles of cognitive tutors which have seen success in tutoring math and programming.

Peer Reviewed

Peer Reviewed

Peer Reviewed

As generative AI products could generate code and assist students with programming learning seamlessly, integrating AI into programming education contexts has driven much attention. However, one emerging concern is that students might get answers without learning from the LLM-generated content. In this work, we deployed the LLM-powered personalized Parsons puzzles as scaffolding to write-code practice in a Python learning classroom (PC condition) and conducted an 80-minute randomized between-subjects study. Both conditions received the same practice problems. The only difference was that when requesting help, the control condition showed students a complete solution (CC condition), simulating the most traditional LLM output. Results indicated that students who received personalized Parsons puzzles as scaffolding engaged in practicing significantly longer than those who received complete solutions when struggling.

Video meeting platforms display conversations linearly through transcripts or summaries. However, ideas during a meeting do not emerge linearly. We leverage LLMs to create dialogue maps in real time to help people visually structure and connect ideas. Balancing the need to reduce the cognitive load on users during the conversation while giving them sufficient control when using AI, we explore two system variants that encompass different levels of AI assistance. In Human-Map, AI generates summaries of conversations as nodes, and users create dialogue maps with the nodes. In AI-Map, AI produces dialogue maps where users can make edits. We ran a within-subject experiment with ten pairs of users, comparing the two MeetMap variants and a baseline. Users preferred MeetMap over traditional methods for taking notes, which aligned better with their mental models of conversations. Users liked the ease of use for AI-Map due to the low effort demands and appreciated the hands-on opportunity in Human-Map for sense-making.

Shared gaze visualizations have been found to enhance collaboration and communication outcomes in diverse HCI scenarios including computer supported collaborative work and learning contexts. Given the importance of gaze in surgery operations, especially when a surgeon trainer and trainee need to coordinate their actions, research on the use of gaze to facilitate intra-operative coordination and instruction has been limited and shows mixed implications. We performed a field observation of 8 surgeries and an interview study with 14 surgeons to understand their visual needs during operations, informing ways to leverage and augment gaze to enhance intra-operative coordination and instruction. We found that trainees have varying needs in receiving visual guidance which are often unfulfilled by the trainers’ instructions. It is critical for surgeons to control the timing of the gaze-based visualizations and effectively interpret gaze data. We suggest overlay technologies, e.g., gaze-based summaries and depth sensing, to augment raw gaze in support of surgical coordination and instruction.

Videos are prominent learning materials to prepare surgical trainees before they enter the operating room (OR). In this work, we explore techniques to enrich the video-based surgery learning experience. We propose Surgment, a system that helps expert surgeons create exercises with feedback based on surgery recordings. Surgment is powered by a few-shot-learning-based pipeline (SegGPT+SAM) to segment surgery scenes, achieving an accuracy of 92%. The segmentation pipeline enables functionalities to create visual questions and feedback desired by surgeons from a formative study. Surgment enables surgeons to 1) retrieve frames of interest through sketches, and 2) design exercises that target specific anatomical components and offer visual feedback. In an evaluation study with 11 surgeons, participants applauded the search-by-sketch approach for identifying frames of interest and found the resulting image-based questions and feedback to be of high educational value.

Providing feedback is widely recognized as crucial for refining students' writing skills.Recent advances in language models (LMs) have made it possible to automatically generate feedback that is actionable and well-aligned with humanspecified attributes.However, it remains unclear whether the feedback generated by these models is truly effective in enhancing the quality of student revisions.Moreover, prompting LMs with a precise set of instructions to generate feedback is nontrivial due to the lack of consensus regarding the specific attributes that can lead to improved revising performance.To address these challenges, we propose PROF that PROduces Feedback via learning from LM simulated student revisions.PROF aims to iteratively optimize the feedback generator by directly maximizing the effectiveness of students' overall revising performance as simulated by LMs.Focusing on an economic essay assignment, we empirically test the efficacy of PROF and observe that our approach not only surpasses a variety of baseline methods in effectiveness of improving students' writing but also demonstrates enhanced pedagogical values, even though it was not explicitly trained for this aspect.

This paper explores the space of optimizing feedback mechanisms in complex domains such as data science, by combining two prevailing approaches: Artificial Intelligence (AI) and learnersourcing. Towards addressing the challenges posed by each approach, this work compares traditional learnersourcing with an AI-supported approach. We report on the results of a randomized controlled experiment conducted with 72 Master’s level students in a data visualization course, comparing two conditions: students writing hints independently versus revising hints generated by GPT-4. The study aimed to evaluate the quality of learnersourced hints, examine the impact of student performance on hint quality, gauge learner preference for writing hints with versus without AI support, and explore the potential of the student-AI collaborative exercise in fostering critical thinking about LLMs. Based on our findings, we provide insights for designing learnersourcing activities leveraging AI support and optimizing students’ learning as they interact with LLMs.

Learning to program can be challenging, and providing high-quality and timely support at scale is hard. Generative AI and its products, like ChatGPT, can create a solution for most intro-level programming problems. However, students might use these tools to just generate code for them, resulting in reduced engagement and limited learning. In this paper, we present CodeTailor, a system that leverages a large language model (LLM) to provide personalized help to students while still encouraging cognitive engagement. CodeTailor provides a personalized Parsons puzzle to support struggling students. In a Parsons puzzle, students place mixed-up code blocks in the correct order to solve a problem. A technical evaluation with previous incorrect student code snippets demonstrated that CodeTailor could deliver high-quality (correct, personalized, and concise) Parsons puzzles based on their incorrect code. We conducted a within-subjects study with 18 novice programmers. Participants perceived CodeTailor as more engaging than just receiving an LLM-generated solution (the baseline condition). In addition, participants applied more supported elements from the scaffolded practice to the posttest when using CodeTailor than baseline. Overall, most participants preferred using CodeTailor versus just receiving the LLM-generated code for learning. Qualitative observations and interviews also provided evidence for the benefits of CodeTailor, including thinking more about solution construction, fostering continuity in learning, promoting reflection, and boosting confidence. We suggest future design ideas to facilitate active learning opportunities with generative AI techniques.

Figure 1: The design of the prompt architecture of Generative Students is based on the KLI framework, which uses knowledge components (KCs) to define the elements students are expected to learn.With the KCs identified for a given task (a), the generative student's profile is a function of the list of KCs the student has mastered, has confusion about, or has no evidence of knowledge of (b).Users can define master prompt, confusion prompt, and unknown prompt for a given task (c).This architecture thus supports automatic creation of diverse student profiles (d).

aendees to leave the workshop with a practical understanding of how to engage with learnersourcing.Participants will get hands-on experience with current tools,

Novice programmers need to write basic code as part of the learning process, but they often face difficulties. To assist struggling students, we recently implemented personalized Parsons problems, which are code puzzles where students arrange blocks of code to solve them, as pop-up scaffolding. Students found them to be more engaging and preferred them for learning, instead of simply receiving the correct answer, such as the response they might get from generative AI tools like ChatGPT. However, a drawback of using Parsons problems as scaffolding is that students may be able to put the code blocks in the correct order without fully understanding the rationale of the correct solution. As a result, the learning benefits of scaffolding are compromised. Can we improve the understanding of personalized Parsons scaffolding by providing textual code explanations? In this poster, we propose a design that incorporates multiple levels of textual explanations for the Parsons problems. This design will be used for future technical evaluations and classroom experiments. These experiments will explore the effectiveness of adding textual explanations to Parsons problems to improve instructional benefits.

The widespread consumer-grade 3D printers and learning resources online enable novices to self-train in remote settings. While troubleshooting plays an essential part of 3D printing, the process remains challenging for many remote novices even with the help of well-developed online sources, such as online troubleshooting archives and online community help. We conducted a formative study with 76 active 3D printing users to learn how remote novices leverage online resources in troubleshooting and their challenges. We found that remote novices cannot fully utilize online resources. For example, the online archives statically provide general information, making it hard to search and relate their unique cases with existing descriptions. Online communities can potentially ease their struggles by providing more targeted suggestions, but a helper who can provide custom help is rather scarce, making it hard to obtain timely assistance. We propose 3DPFIX, an interactive 3D troubleshooting system powered by the pipeline to facilitate Human-AI Collaboration, designed to improve novices' 3D printing experiences and thus help them easily accumulate their domain knowledge. We built 3DPFIX that supports automated diagnosis and solution-seeking. 3DPFIX was built upon shared dialogues about failure cases from Q&A discourses accumulated in online communities. We leverage social annotations (i.e., comments) to build an annotated failure image dataset for AI classifiers and extract a solution pool. Our summative study revealed that using 3DPFIX helped participants spend significantly less effort in diagnosing failures and finding a more accurate solution than relying on their common practice. We also found that 3DPFIX users learn about 3D printing domain-specific knowledge. We discuss the implications of leveraging community-driven data in developing future Human-AI Collaboration designs.

Despite that reading assignments are prevalent, methods to encourage students to actively read are limited. We propose a system ReadingQuizMaker that supports instructors to conveniently design high-quality questions to help students comprehend readings. ReadingQuizMaker adapts to instructors’ natural workflows of creating questions, while providing NLP-based process-oriented support. ReadingQuizMaker enables instructors to decide when and which NLP models to use, select the input to the models, and edit the outcomes. In an evaluation study, instructors found the resulting questions to be comparable to their previously designed quizzes. Instructors praised ReadingQuizMaker for its ease of use, and considered the NLP suggestions to be satisfying and helpful. We compared ReadingQuizMaker with a control condition where instructors were given automatically generated questions to edit. Instructors showed a strong preference for the human-AI teaming approach provided by ReadingQuizMaker. Our findings suggest the importance of giving users control and showing an immediate preview of AI outcomes when providing AI support.

Novice programmers struggle with writing code from scratch. One possible way to help them is by using an equivalent Parsons problem on demand, where learners place mixed-up code blocks in the correct order. In a classroom study with 89 undergraduate students, we examined how using a Parsons problem as scaffolding impacts performance and problem-solving efficiency. Results showed that students in the Parsons as Help group achieved significantly higher practice performance and problem-solving efficiency than students who wrote code without help, while achieving the same level of posttest scores. These results improve the understanding of Parsons problems and contribute to the design of future coding practices.

As enrollment in college classes rises, it is increasingly difficult to grade and provide feedback to open-ended assignments. It is a timeconsuming and labor-intensive task for instructors, especially when the grading criteria are subjective and constantly evolving. Rubrics are often used to standardize grading, but they can be challenging to create and may not always capture the nuances of a particular assignment. Additionally, it can be difficult to articulate principles or constraints that define a "good" solution in less well-defined domains; like human-computer interaction (HCI) [3, 8] or user experience (UX) [7]. Instructors may delegate the task of grading and offering feedback to a number of graders. However, through our co-design study (section 2.1), we've found that inter-grader reliability, managing time constraints, and dealing with unclear rubrics are just some of the many issues faced by graders in this process.

Learning Experience Designers (LXDs) play an increasingly consequential role in the creation of courses and training materials that meet the needs of diverse learner populations and the growing class scope. Emerging design requests for scalable and effective courseware introduce new challenges in Learning Experience (LX) design practice while providing an opportunity for researchers to understand LX workflows and design new tools to improve them. This paper presents an interview study with 21 LXDs from 18 different organizations with the goal of understanding LXDs' collaborative relationships with subject matter experts (SMEs), data needs, and contextual challenges. We further perform a survey study to validate the challenges and probe into LXDs' attitudes toward a suite of data-driven solutions. We find that LXDs demonstrate a strong desire to collect data to inform their design - including target learners' prior knowledge and relevant design precedents. LXDs want support in better collaborating with SMEs, acquiring and processing diverse learner data, identifying relevant research studies to communicate their design decisions, understanding domain-specific material, and creating quality materials (especially questions). We discuss LXDs' concerns regarding automated solutions such as the lack of contextual understanding, over-reliance on automation, and data privacy before elaborating on the implications for future work.

While videoconferencing is prevalent, concurrent participation channels are limited. People experience challenges keeping up with the discussion, and misunderstanding frequently occurs. Through a formative study, we probed into the design space of providing real-time transcripts as an extra communication space for video meeting attendees. We then present MeetScript, a system that provides parallel participation channels through real-time interactive transcripts. MeetScript visualizes the discussion through a chat-alike interface and allows meeting attendees to make real-time collaborative annotations. Over time, MeetScript gradually hides extraneous content to retain the most essential information on the transcript, with the goal of reducing the cognitive load required on users to process the information in real time. In an experiment with 80 users in 22 teams, we compared MeetScript with two baseline conditions where participants used Zoom alone (business-as-usual), or Zoom with an adds-on transcription service (Otter.ai). We found that MeetScript significantly enhanced people's non-verbal participation and recollection of their teams' decision-making processes compared to the baselines. Users liked that MeetScript allowed them to easily navigate the transcript and contextualize feedback and new ideas with existing ones.

In this paper, we explore using Parsons problems to scaffold novice programmers who are struggling while solving write-code problems. Parsons problems, in which students put mixed-up code blocks in order, can be created quickly and already serve thousands of students while other types of programming support methods are expensive to develop or do not scale. We conducted two studies in which novices were given equivalent Parsons problems as optional scaffolding while solving write-code problems. We investigated when, why, and how students used the Parsons problems as well as their perceptions of the benefits and challenges. A think-aloud observational study with 11 undergraduate students showed that students utilized the Parsons problem before writing a solution to get ideas about where to start; during writing a solution when they were stuck; and after writing a solution to debug errors and look for better strategies. Semi-structured interviews with the same 11 undergraduate students provided evidence that using Parsons problems to scaffold write-code problems helped students to reduce the difficulty, reduce the problem completion time, learn problem-solving strategies, and refine their programming knowledge. However, some students found them less useful if the Parsons solution did not match their approach or if they did not understand the solution. We then conducted a between-subjects classroom study with 81 undergraduate students to investigate the effects on learning. We found that students who received Parsons problems as scaffolding during write-code problems spent significantly less time solving those problems. However, there was no significant learning gain in either condition from pretest to posttest. We also discuss the design implications of our findings.

NLP-powered automatic question generation (QG) techniques carry great pedagogical potential of saving educators' time and benefiting student learning.Yet, QG systems have not been widely adopted in classrooms to date.In this work, we aim to pinpoint key impediments and investigate how to improve the usability of automatic QG techniques for educational purposes by understanding how instructors construct questions and identifying touch points to enhance the underlying NLP models.We perform an in-depth need finding study with 11 instructors across 7 different universities, and summarize their thought processes and needs when creating questions.While instructors show great interests in using NLP systems to support question design, none of them has used such tools in practice.They resort to multiple sources of information, ranging from domain knowledge to students' misconceptions, all of which missing from today's QG systems.We argue that building effective human-NLP collaborative QG systems that emphasize instructor control and explainability is imperative for real-world adoption.We call for QG systems to provide process-oriented support, use modular design, and handle diverse sources of input.

Maximizing system scalability and quality are sometimes at odds. This work provides an example showing scalability and quality can be achieved at the same time in instructional design, contrary to what instructors may believe or expect. We situate our study in the education of HCI methods, and provide suggestions to improve active learning within the HCI education community. While designing learning and assessment activities, many instructors face the choice of using open-ended or close-ended activities. Close-ended activities such as multiple-choice questions (MCQs) enable automated feedback to students. However, a survey with 22 HCI professors revealed a belief that MCQs are less valuable than open-ended questions, and thus, using them entails making a quality sacrifice in order to achieve scalability. A study with 178 students produced no evidence to support the teacher belief. This paper indicates more promise than concern in using MCQs for scalable instruction and assessment in at least some HCI domains.

Practice is essential for learning. However, for many interpersonal skills, there often are not enough opportunities and venues for novices to repeatedly practice. Role-playing simulations offer a promising framework to advance practice-based professional training for complex communication skills, in fields such as teaching. In this work, we introduce ELK (Eliciting Learner Knowledge), a role-playing simulation system that helps K-12 teachers develop effective questioning strategies to elicit learners' prior knowledge. We evaluate ELK with 75 pre-service teachers through a mixed-method study. We find that teachers demonstrate a modest increase in effective questioning strategies and develop sympathy towards students after using ELK for 3 rounds. We implement a supplementary activity in ELK in which users evaluate transcripts generated from past role-play sessions. We have tentative evidence that a combination of role-play and evaluating conversation moves may be more effective for learning. We contribute design implications of using role-play systems for communication strategy training.

The outbreak of COVID-19 has led to a sharp transition from offline to online education in many countries and areas. This transition heightens the intensity of existing challenges of online education, such as student attendance and education equality. During this time of uncertainty, the vast disparities in teachers? online experience and technical backgrounds, students' education level and their families' economic status, and schools' support, further pose new challenges to teachers and students. In this work, we study how Chinese teachers and students addressed challenges during this transition. We interviewed 15 teachers and 18 students from diverse backgrounds at varying education levels (K-12 and college). Our work makes timely and new contributions to the literature of online education. For example, our results showed that teachers applied Live Video Streaming (LVS) on multiple social media platforms and re-purposed different entertainment features to deliver online teaching for better student engagement; some teachers came to enjoy this new form of instruction after being resistant to it in the beginning, and students developed a better sense of intimacy with their teachers after experiencing certain online interactions. Our work also reveals the remaining challenges and prospects of LVS-based online education and sheds light on the future design of collaborative technologies for online education.

In schools and colleges around the world, open-ended home-work assignments are commonly used. However, such assignments require substantial instructor effort for grading, and tend not to support opportunities for repeated practice. We propose UpGrade, a novel learnersourcing approach that generates scalable learning opportunities using prior student solutions to open-ended problems. UpGrade creates interactive questions that offer automated and real-time feedback, while enabling repeated practice. In a two-week experiment in a college-level HCI course, students answering UpGrade-created questions instead of traditional open-ended assignments achieved indistinguishable learning outcomes in ~30% less time. Further, no manual grading effort is required. To enhance quality control, UpGrade incorporates a psychometric approach using crowd workers' answers to automatically prune out low quality questions, resulting in a question bank that exceeds reliability standards for classroom use.

The human visual system processes complex scenes to focus attention on relevant items. However, blind people cannot visually skim for an area of interest. Instead, they use a combination of contextual information, knowledge of the spatial layout of their environment, and interactive scanning to find and attend to specific items. In this paper, we define and compare three cursor-based interactions to help blind people attend to items in a complex visual scene: window cursor (move their phone to scan), finger cursor (point their finger to read), and touch cursor (drag their finger on the touchscreen to explore). We conducted a user study with 12 participants to evaluate the three techniques on four tasks, and found that: window cursor worked well for locating objects on large surfaces, finger cursor worked well for accessing control panels, and touch cursor worked well for helping users understand spatial layouts. A combination of multiple techniques will likely be best for supporting a variety of everyday tasks for blind users.

Users of complex software applications often rely on inefficient or suboptimal workflows because they are not aware that better methods exist. In this paper, we develop and validate a hierarchical approach combining topic modeling and frequent pattern mining to classify the workflows offered by an application, based on a corpus of community-generated videos and command logs. We then propose and evaluate a design space of four different workflow recommender algorithms, which can be used to recommend new workflows and their associated videos to software users. An expert validation of the task classification approach found that 82% of the time, experts agreed with the classifications. We also evaluate our workflow recommender algorithms, demonstrating their potential and suggesting avenues for future work.

With the aim of better scaffolding discussion to improve learning in a MOOC context, this work investigates what kinds of discussion behaviors contribute to learning. We explored whether engaging in higher-order thinking behaviors results in more learning than paying general or focused attention to course materials. In order to evaluate whether to attribute the effect to engagement in the associated behaviors versus persistent characteristics of the students, we adopted two approaches. First, we used propensity score matching to pair students who exhibit a similar level of involvement in other course activities. Second, we explored individual variation in engagement in higher-order thinking behaviors across weeks. The results of both analyses support the attribution of the effect to the behavioral interpretation. A further analysis using LDA applied to course materials suggests that more social oriented topics triggered richer discussion than more biopsychology oriented topics.

In this paper, an inquiry-based Augmented Reality learning tool was implemented. Students could control, combine and interact with the 3D model of micro particles using markers, and conduct a series of inquiry-based experiments. The AR tool developed was tested in practice at a junior high school. Experiment result shows that the AR tool has significant supplemental learning effect as a computer-assisted learning tool and students generally have a positive attitude towards this software.