We study instructor perspectives on LLM-generated programming feedback in an introductory Python course. LLM tutors predominantly offered debugging help, while human instructors preferred more diverse feedback types, including conceptual reminders, revisiting the problem, and examples. Cases where LLM tutor feedback diverged from human instructors' intent required major edits with different feedback types, while cases with closer alignment needed only minor changes with similar feedback types. Findings highlight the need for LLM tutors to reflect on instructor intent to ensure pedagogically aligned feedback.

Large Language Models (LLMs) are increasingly used as programming tutors, but their feedback is often generic and prone to solution leakage. To address these issues, we present MisconceptionTutor, which grounds feedback in common student misconceptions. Through both pre-deployment analyses and a real-classroom deployment, we find that even simple prompting frameworks can meaningfully steer tutor behavior to be more pedagogically oriented and noticeably more satisfying to students.

Gauging a student's understanding of course concepts, at an arbitrary point during a course, can be challenging. Standardized exams offer only a snapshot of performance rather than a deep understanding of progress. However, with Large Language Models (LLMs) now deployed at scale in CS1 courses, we can track multiple attempts from each student for every homework problem. This data provides insights into how students learn and deploy concepts over time, presenting a unique opportunity to rethink how we track changes in individual student knowledge. Traditional Knowledge Tracing (KT) methods often lack explainability and are computationally expensive. In contrast, our framework leverages an LLM to identify student progress on labeled, problem-level concepts from a student homework code submission. Our initial results show that the student's knowledge state can be dynamically updated. This knowledge state can then be used to provide more targeted, effective feedback and create tailored study materials.

LLM-based chatbots enable students to get immediate, interactive help on homework assignments, but even a thoughtfully-designed bot may not serve all pedagogical goals. We report here on the development and deployment of a GPT-4-based interactive homework assistant ("61A Bot'') for students in a large CS1 course; over 2000 students made over 100,000 requests of our Bot across two semesters. Our assistant offers one-shot, contextual feedback within the command-line "autograder'' students use to test their code. Our Bot wraps student code in a custom prompt that supports our pedagogical goals and avoids providing solutions directly. Analyzing student feedback, questions, and autograder data, we find reductions in homework-related question rates in our course forum, as well as reductions in homework completion time when our Bot is available. For students in the 50th -80th percentile, reductions can exceed 30 minutes per assignment, up to 50% less time than students at the same percentile rank in prior semesters. Finally, we discuss these observations, potential impacts on student learning, and other potential costs and benefits of AI assistance in CS1.

While neural networks demonstrate a remarkable ability to model linguistic content, capturing contextual information related to a speaker's conversational role is an open area of research.In this work, we analyze the effect of speaker role on language use through the game of Mafia, in which participants are assigned either an honest or a deceptive role.In addition to building a framework to collect a dataset of Mafia game records, we demonstrate that there are differences in the language produced by players with different roles.We confirm that classification models are able to rank deceptive players as more suspicious than honest ones based only on their use of language.Furthermore, we show that training models on two auxiliary tasks outperforms a standard BERT-based text classification approach.We also present methods for using our trained models to identify features that distinguish between player roles, which could be used to assist players during the Mafia game.

Jessy Lin, Geza Kovacs, Aditya Shastry, Joern Wuebker, John DeNero. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

The workshop will allow participants to gain experience with a series of innovations developed at UC Berkeley that have enabled the teaching of undergraduate data science at scale to students from all backgrounds. Rather than beginning with established introductory strategies as the gateway to computer science, students in the "Foundations of Data Science" (data8.org) learn computational skills and concepts in relation to real-world issues and with attention to societal implications. By engaging with students' interest in the applications of computing on data, and integrating societal impact from the start, the program has developed a long-term commitment to advance computational skills for large numbers of students. These innovations in teaching not only convey important computational content, but also broaden participation beyond existing approaches to computer science. Goals include increasing diversity among students learning computer science, giving students a strong ethical foundation within their computer science work, and encouraging critical thinking in the application of inference and statistical techniques. Bringing a laptop is recommended. UC Berkeley has as over 1000 students in a large and open Data Science Major, where a range of Domain Emphases and backgrounds bring a broader set of students than the traditional CS major. UC Berkeley Data Science has been gathering over 500 educators in a summer workshop on sharing curricular innovation.

Word alignment was once a core unsupervised learning task in natural language processing because of its essential role in training statistical machine translation (MT) models.Although unnecessary for training neural MT models, word alignment still plays an important role in interactive applications of neural machine translation, such as annotation transfer and lexicon injection.While statistical MT methods have been replaced by neural approaches with superior performance, the twenty-year-old GIZA++ toolkit remains a key component of state-of-the-art word alignment systems.Prior work on neural word alignment has only been able to outperform GIZA++ by using its output during training.We present the first end-to-end neural word alignment method that consistently outperforms GIZA++ on three data sets.Our approach repurposes a Transformer model trained for supervised translation to also serve as an unsupervised word alignment model in a manner that is tightly integrated and does not affect translation quality.

We propose an efficient batching strategy for variable-length decoding on GPU architectures.During decoding, when candidates terminate or are pruned according to heuristics, our streaming approach periodically "refills" the batch before proceeding with a selected subset of candidates.We apply our method to variable-width beam search on a state-of-theart machine translation model.Our method decreases runtime by up to 71% compared to a fixed-width beam search baseline and 17% compared to a variable-width baseline, while matching baselines' BLEU.Finally, experiments show that our method can speed up decoding in other domains, such as semantic and syntactic parsing.

The Nifty Assignments special session is about promoting and sharing the ideas and ready-to-use materials of successful assignments. Each presenter will introduce their assignment, give a quick demo, and describe its niche in the curriculum and its strengths and weaknesses. The presentations (and the descriptions below) merely introduce the assignment. A key part of Nifty Assignments is the mundane but vital role of distributing the materials - handouts, data files, starter code, rubrics, autograders - that make each assignment ready to adopt. Each assignment presented has complete materials freely available on the Nifty Assignments home page nifty.stanford.edu. If you have an assignment that works well and would be of interest to the CSE community, please consider applying to present at Nifty Assignments.

We propose and compare methods for gradientbased domain adaptation of self-attentive neural machine translation models.We demonstrate that a large proportion of model parameters can be frozen during adaptation with minimal or no reduction in translation quality by encouraging structured sparsity in the set of offset tensors during learning via group lasso regularization.We evaluate this technique for both batch and incremental adaptation across multiple data sets and language pairs.Our system architecture-combining a state-of-the-art self-attentive model with compact domain adaptation-provides high quality personalized machine translation that is both space and time efficient.

No abstract available.

We apply phrase-based and neural models to a core task in interactive machine translation: suggesting how to complete a partial translation.For the phrase-based system, we demonstrate improvements in suggestion quality using novel objective functions, learning techniques, and inference algorithms tailored to this task.Our contributions include new tunable metrics, an improved beam search strategy, an n-best extraction method that increases suggestion diversity, and a tuning procedure for a hierarchical joint model of alignment and translation.The combination of these techniques improves next-word suggestion accuracy dramatically from 28.5% to 41.2% in a large-scale English-German experiment.Our recurrent neural translation system increases accuracy yet further to 53.0%, but inference is two orders of magnitude slower.Manual error analysis shows the strengths and weaknesses of both approaches.

We investigate the characteristics and quantifiable predispositions of both n-gram and recurrent neural language models in the framework of language generation.In modern applications, neural models have been widely adopted, as they have empirically provided better results.However, there is a lack of deep analysis of the models and how they relate to real language and its structural properties.We attempt to perform such an investigation by analyzing corpora generated by sampling from the models.The results are compared to each other and to the results of the same analysis applied to the training corpus.We carried out these experiments on varieties of Kneser-Ney smoothed n-gram models and basic recurrent neural language models.Our results reveal a number of distinctive characteristics of each model, and offer insights into their behavior.Our general approach also provides a framework in which to perform further analysis of language models.

Scaffolded projects with automated feedback are core instructional components of many massive courses. In subjects that include programming, feedback is typically provided by test cases constructed manually by the instructor. This paper explores the effectiveness of fuzz testing, a randomized technique for verifying the behavior of programs. In particular, we apply fuzz testing to identify when a student's solution differs in behavior from a reference implementation by randomly exploring the space of legal inputs to a program. Fuzz testing serves as a useful complement to manually constructed tests. Instructors can concentrate on designing targeted tests that focus attention on specific issues while using fuzz testing for comprehensive error checking. In the first project of a 1,400-student introductory computer science course, fuzz testing caught errors that were missed by a suite of targeted test cases for more than 48% of students. As a result, the students dedicated substantially more effort to mastering the nuances of the assignment.

In contrast to multiple-choice or selected response questions, constructed response questions can result in a wide variety of incorrect responses. However, constructed responses are richer in information. We propose a technique for using each student's constructed responses in order to identify a subset of their stable conceptual misunderstandings. Our approach is designed for courses with so many students that it is infeasible to interpret every distinct wrong answer manually. Instead, we label only the most frequent wrong answers with the misunderstandings that they indicate, then predict the misunderstandings associated with other wrong answers using statistical co-occurrence patterns. This tiered approach leverages a small amount of human labeling effort to seed an automated procedure that identifies misunderstandings in students. Our approach involves much less effort than inspecting all answers, substantially outperforms a baseline that does not take advantage of co-occurrence statistics, proves robust to different course sizes, and generalizes effectively across student cohorts.

"Be careful what you wish for, you just might get it." - Proverb In 2005, computing education was experiencing a crisis. Enrollments had "fallen to such an extent that some academic computing programs were facing significant reductions in staffing levels or even elimination". The community responded, with panels to investigate and highlight ways to infuse "passion, beauty, joy and awe" into the introductory experiences, the CS10K project to bring computing to 10,000 teachers and 100,000 students, and better messaging of career opportunities, to name a few of the initiatives to bring students back into our seats.

I suspect that students learn more from our programming assignments than from our much sweated-over lectures, with their slide transitions, clip art, and joke attempts. A great assignment is deliberate about where the student hours go, concentrating the student's attention on material that is interesting and useful. The best assignments solve a problem that is topical and entertaining, providing motivation for the whole stack of work. Unfortunately, creating great programming assignments is both time consuming and error prone. The Nifty Assignments special session is all about promoting and sharing the ideas and ready-to-use materials of successful assignments.

Recent work in neural machine translation has shown promising performance, but the most effective architectures do not scale naturally to large vocabulary sizes.We propose and compare three variable-length encoding schemes that represent a large vocabulary corpus using a much smaller vocabulary with no loss in information.Common words are unaffected by our encoding, but rare words are encoded using a sequence of two pseudo-words.Our method is simple and effective: it requires no complete dictionaries, learning procedures, increased training time, changes to the model, or new parameters.Compared to a baseline that replaces all rare words with an unknown word symbol, our best variable-length encoding strategy improves WMT English-French translation performance by up to 1.7 BLEU.

We present an incremental adaptation approach for statistical machine translation that maintains a flexible hierarchical domain structure within a single consistent model.Both weights and rules are updated incrementally on a stream of post-edits.Our multi-level domain hierarchy allows the system to adapt simultaneously towards local context at different levels of granularity, including genres and individual documents.Our experiments show consistent improvements in translation quality from all components of our approach.

Automatic assessment reduces the need for individual feedback in massive courses, but often focuses only on scoring solutions, rather than assessing whether students correctly understand problems. We present an enriched approach to automatic assessment that explicitly assists students in understanding the detailed specification of technical problems that they are asked to solve, in addition to evaluating their solutions. Students are given a suite of solution test cases, but they must first unlock each test case by validating its behavior before they are allowed to apply it to their proposed solution. When provided with this automated feedback early in the problem-solving process, students ask fewer clarificatory questions and express less confusion about assessments. As a result, instructors spend less time explaining problems to students. In a 1300-person university course, we observed that the vast majority of students chose to validate their understanding of test cases before attempting to solve problems. These students reported that the validation process improved their understanding.

Bidirectional models of word alignment are an appealing alternative to post-hoc combinations of directional word aligners.Unfortunately, most bidirectional formulations are NP-Hard to solve, and a previous attempt to use a relaxationbased decoder yielded few exact solutions (6%).We present a novel relaxation for decoding the bidirectional model of DeNero and Macherey (2011).The relaxation can be solved with a modified version of the Viterbi algorithm.To find optimal solutions on difficult instances, we alternate between incrementally adding constraints and applying optimality-preserving coarse-to-fine pruning.The algorithm finds provably exact solutions on 86% of sentence pairs and shows improvements over directional models.

We describe an effort to improve the composition quality of student programs: the property that a program can be understood effectively by another person. As a semester-long component of UC Berkeley's first course for majors, CS 61A, we gave students composition guidelines, scores, and qualitative feedback-all generated manually by a course staff of 10 graders for over 700 students. To facilitate this effort, we created a new online tool that allows instructors to provide feedback efficiently at scale. Our system differs from recently developed alternatives in that it is a branch of an industrial tool originally developed for internal code reviews at Google and used extensively by the open-source community. We found that many of the features designed for industrial applications are well-suited for instructional use as well. We extended the system with permissions controls and comment memories tailored for giving educational feedback.

A great CS assignment is a delight to all, but the path to one can be most roundabout. Many CS students have had their characters built up on assignments that worked better as an idea than as an actual assignment. Assignments are hard to come up with, yet they are the key to student learning. The Nifty Assignments special session is all about promoting and sharing the ideas and ready-to-use materials of successful assignments.

We address the problem of identifying multiword expressions in a language, focusing on English phrasal verbs.Our polyglot ranking approach integrates frequency statistics from translated corpora in 50 different languages.Our experimental evaluation demonstrates that combining statistical evidence from many parallel corpora using a novel ranking-oriented boosting algorithm produces a comprehensive set of English phrasal verbs, achieving performance comparable to a human-curated set. *Research conducted during an internship at Google.

Every time I re-use a handout, I look it over and make a few little "improvements". I play around with code demos and entertain myself with different slide transitions. However, inevitably, I return to the conclusion that most of what my students learn in my course comes from the assignments. Great assignments are hard to dream up and time-consuming to develop. With that in mind, the Nifty Assignments session is all about promoting and sharing the ideas and ready-to-use materials of successful assignments.