Background: Context personalization, the practice of adapting learning materials to students’ personal interests, has been shown to increase student learning and engagement. Within computer science education, research has found that LLMs can generate high-quality contextualized introductory programming exercises. Objective: In this paper, we evaluate the capability of LLMs to generate technically correct and thematically integrated contextualized algorithm design problems. Methods: In a series of three iterative studies, we use LLMs to generate contextualized algorithm design problems from a given base problem and theme, evaluating over 500 generated problems for technical and thematic alignment. Results: We find that LLM-generated algorithm design problems exhibit significantly more issues than prior work has found for introductory programming problems. We identify issues specific to the algorithm design context and then mitigate these issues with prompt engineering techniques and model choice. With these adjustments, we produce LLM-generated contextualized algorithm design problems that are technically strong, deeply themed, and largely realistic, though realism drops with more culturally and locally specific themes. Implications: We demonstrate a viable workflow for generating contextualized algorithm design problems using LLMs, including prompt design, model selection, and identification of specific issues to review for.

This paper introduces the PrairieLearn Question Sharing System (PQSS), which enables instructors to share question generators with other instructors, either as open educational resources or privately. PQSS is integrated into PrairieLearn, an open-source, problem-driven online learning platform. PQSS addresses a critical need for more open-source assessments by making it easier for instructors to share assessments and for instructors to use those assessments. Instructors often do not share questions due to the time it takes to publish them and the lack of recognition for their work. Because it is directly integrated into PrairieLearn, PQSS reduces the aforementioned friction of sharing and using shared questions, and we can report usage statistics to help question authors receive recognition for their work. In this paper, we share design and implementation details of the system, as well as experiences using it to share course content across courses and between universities.

Manually grading free response questions remains a persistent challenge in education. While such questions offer valuable opportunities for student learning and critical thinking, their evaluation often requires substantial time and effort from instructors or teaching assistants. In addition to the grading workload, open-ended responses are susceptible to inconsistencies in scoring and may reflect unclear expectations, both of which can undermine the effectiveness and fairness of the assessment process. To address these challenges, we employed an AI-based grading system integrated in PrairieLearn to automatically evaluate student submissions to free response questions using a predefined set of rubric items. This approach not only streamlines the grading process but also enables direct comparison between AI-generated rubric applications and human judgments, providing insight into alignment and potential discrepancies. These discrepancies provided valuable insight, allowing us to iteratively revise and clarify the rubric items. Our experiences with using the AI grading system across several computing courses suggest that even experienced educators face difficulties articulating rubrics that are both specific and interpretable. We furthermore argue that more attention should be given to the iterative development and evaluation of rubrics.

Concept inventories are standardized assessments that evaluate student understanding of key concepts within academic disciplines. While prevalent across STEM fields, their development lags for advanced computer science topics like dynamic programming (DP)---an algorithmic technique that poses significant conceptual challenges for undergraduates. To fill this gap, we developed and validated a Dynamic Programming Concept Inventory (DPCI). We detail the iterative process used to formulate multiple-choice questions targeting known student misconceptions about DP concepts identified through prior research studies. We discuss key decisions, tradeoffs, and challenges faced in crafting probing questions to subtly reveal these conceptual misunderstandings. We conducted a preliminary psychometric validation by administering the DPCI to 172 undergraduate CS students finding our questions to be of appropriate difficulty and effectively discriminating between differing levels of student understanding. Taken together, our validated DPCI will enable instructors to accurately assess student mastery of DP. Moreover, our approach for devising a concept inventory for an advanced theoretical computer science concept can guide future efforts to create assessments for other under-evaluated areas currently lacking coverage.

Introductory Programming courses teach multiple skills such as 1) explaining the purpose of code, 2) the ability to arrange lines of code in correct sequence, and 3) the ability to trace through the execution of a program, and 4) the ability to write code from scratch. Knowing if a programming skill is a prerequisite to another would assist instructors in organizing their materials such that students encounter and learn new topics using optimal skill sequences. In this study, we used the conviction measure from association rule mining to perform pair-wise comparisons of five skills: Write, Trace, Reverse trace, Sequence, and Explain code. We used the data from four exams with more than 600 participants in each exam from a public university in the United States, where students solved programming assignments of different skills for several programming topics. Our findings matched the previous finding that tracing is a prerequisite for students to learn to write code. But, contradicting the previous claims, our analysis suggested that writing code is a prerequisite skill to explaining code and that sequencing code is not a prerequisite to writing code. Our research can help instructors by systematically arranging the skills students exercise when encountering a new topic.

Background: Proof Blocks is a software tool that enables students to construct proofs by assembling prewritten lines and gives them automated feedback. Prior work on learning gains from Proof Blocks has focused on comparing learning gains from Proof Blocks against other learning activities such as writing proofs or reading.

Background: In CS1 courses, Parsons problems are a popular activity in which students are given blocks of code and asked to rearrange them into the correct order. Parsons problems often include incorrect blocks of code referred to as distractor blocks. Despite their widespread use, there have been few investigations into how distractor blocks impact student learning. Objectives: Our goals are to understand (1) the impact that including distractor blocks in Parsons problems has on learning and (2) the causality underlying that learning, if any. Methods: In this paper, we present the results of an explanatory sequential mixed methods study investigating the impact of distractor blocks on student learning. For the initial, quantitative stage, we use a randomized control trial to quantify the learning outcomes from practice with Parsons problems that include distractor blocks, as measured via post-test taken immediately after the practice activity and a retention test taken a week later. This study is followed by think-aloud interviews with 10 students practicing using a mix of Parsons problems that do and do not contain distractors to understand differences in how students approach those problems. Findings: Our findings show that students who practiced using Parsons problems that contained distractors performed 11 percentage points better on the immediate post-test (statistically significant) and 10 percentage points better on the retention test (approaching significance). The results of the think-aloud interviews indicate that grouping distractors with blocks of correct code causes students to more closely attend to the details of the code within those blocks. Implications: The results of this study indicate that distractors are essential when Parsons problems are used in a formative context. When they are not included, students may be able to successfully place blocks of code without attending to details of the code. This in turn limits their ability to learn new concepts or reinforce existing knowledge from those code blocks.

The command shell and Git are important tools for computer scientists to learn and is taught in many computer science curricula. Many tools used by computer scientists are primarily interfaced through the command shell, such as Git. However, there have been few studies and interventions designed to assist in understanding student behaviors in the command shell and better teaching of it. This paper aims to provide an overview and reflection of a novel intelligent tutoring system we developed, the Shell Tutor, which assists in teaching students the command shell and Git. This paper will also analyze features of this tool to better understand student behaviors within the command shell while using this intelligent tutoring system: a logging system which will enable researchers to better understand student behaviors in using the tool and the command shell in general. A study conducted with students who used this tool illuminates the perceived effects on student learning and their perspectives of the tool, which are overwhelmingly positive.

Algorithm design is a vital skill developed in most undergraduate Computer Science (CS) programs, but few research studies focus on pedagogy related to Algorithms coursework. To understand the work that has been done in the area, we present a systematic survey and characterization of existing studies in the CS Education literature related to the teaching of algorithm design at the undergraduate level. Across all papers in the ACM Digital Library, we only find 97 applicable papers. We classify these papers by topic, evaluation metric, evaluation methods, and intervention target. We present the results of these classifications alongside insights about existing knowledge, rigor, and contribution rates. We hope that this work not only provides a detailed representation of the current corpus of CS Education work related to algorithm design but also demonstrates that the body of knowledge is sparse and supports further research in the area. For future work, we intend to investigate and synthesize the conclusions reached by these papers.

Background : Proof Blocks is a software tool that enables students to construct proofs by assembling prewritten lines and gives them automated feedback. Prior research has shown that students learn as much from an activity where they use Proof Blocks as where they write proofs. However, in both cases students first read a book chapter. Prior research was not able to differentiate between the learning gains achieved from reading versus proof practice. Purpose : This study aims to measure learning gains from reading a book chapter versus completing Proof Blocks. Methods : We conducted a randomized controlled trial with three experimental groups: one that only read a book chapter, one that only completed Proof Blocks, and one that did both. Findings : The group that completed only Proof Blocks had the smallest learning gains. The group that read the book chapter and completed the Proof Blocks activity performed marginally better than students who only read the book chapter, but it is not clear if the source of this improvement was the Proof Blocks or just exposure to more examples.

Large language models (LLMs) have recently taken many fields, including computer science, by storm. Most recent work on LLMs in computing education has shown that they are capable of solving most introductory programming (CS1) exercises, exam questions, Parsons problems, and several other types of exercises and questions. Some work has investigated the ability of LLMs to solve CS2 problems as well. However, it remains unclear how well LLMs fare against more advanced upper-division coursework, such as proofs in algorithms courses. After all, while known to be proficient in many programming tasks, LLMs have been shown to have more difficulties in forming mathematical proofs.

Proof Blocks is a software tool that provides students with a scaffolded proof-writing experience, allowing them to drag and drop prewritten proof lines into the correct order instead of starting from scratch. In this paper we describe a randomized controlled trial designed to measure the learning gains of using Proof Blocks for students learning proof by induction. The study participants were 332 students recruited after completing the first month of their discrete mathematics course. Students in the study took a pretest and read lecture notes on proof by induction, completed a brief (less than 1 hour) learning activity, and then returned one week later to complete the posttest. Depending on the experimental condition that each student was assigned to, they either completed only Proof Blocks problems, completed some Proof Blocks problems and some written proofs, or completed only written proofs for their learning activity. We find that students in the early phases of learning about proof by induction are able to learn just as much from reading lecture notes and using Proof Blocks as by reading lecture notes and writing proofs from scratch, but in far less time on task. This finding complements previous findings that Proof Blocks are useful exam questions and are viewed positively by students.

In this software tool paper we present Proof Blocks, a tool which enables students to construct mathematical proofs by dragging and dropping prewritten proof lines into the correct order. We present both implementation details of the tool, as well as a rich reflection on our experiences using the tool in courses with hundreds of students. Proof Blocks problems can be graded completely automatically, enabling students to receive rapid feedback. When writing a problem, the instructor specifies the dependency graph of the lines of the proof, so that any correct arrangement of the lines can receive full credit. This innovation can improve assessment tools by increasing the types of questions we can ask students about proofs, and can give greater access to proof knowledge by increasing the amount that students can learn on their own with the help of a computer.

We present a psychometric evaluation of a revised version of theCybersecurity Concept Inventory (CCI), completed by 354 students from 29 colleges and universities. The CCI is a conceptual test of understanding created to enable research on instruction quality in cybersecurity education. This work extends previous expert review and small-scale pilot testing of the CCI. Results show that the CCI aligns with a curriculum many instructors expect from an introductory cybersecurity course, and that it is a valid and reliable tool for assessing what conceptual cybersecurity knowledge students learned.

Proof Blocks is a novel software tool which enables students to write mathematical proofs by dragging and dropping prewritten lines into the correct order, rather than writing a proof completely from scratch. We used Proof Blocks problems as exam questions for a discrete mathematics course with hundreds of students, allowing us to collect thousands of student responses to Proof Blocks problems. Using this data, we provide statistical evidence that Proof Blocks are easier than written proofs, which are typically very difficult. We also show that Proof Blocks problems provide about as much information about student knowledge as written proofs. Survey results show that students believe that the Proof Blocks user interface is easy to use, and that the questions accurately represent their ability to write proofs.

We analyze submissions for homework assignments of 527 students in an upper-level database course offered at the University of Illinois at Urbana-Champaign. The ability to query databases is becoming a crucial skill for technology professionals and academics. Although we observe a large demand for teaching database skills, there is little research on database education. Also, despite the industry's continued demand for NoSQL databases, we have virtually no research on the matter of how students learn NoSQL databases, such as MongoDB. In this paper, we offer an in-depth analysis of errors committed by students working on MongoDB homework assignments over the course of two semesters. We show that as students use more advanced MongoDB operators, they make more Reference errors. Additionally, when students face a new functionality of MongoDB operators, such as \texttt\$group operator, they usually take time to understand it but do not make the same errors again in later problems. Finally, our analysis suggests that students struggle with advanced concepts for a comparable amount of time. Our results suggest that instructors should allocate more time and effort for the discussed topics in our paper.

As data grow both in size and in connectivity, the interest to use graph databases in the industry has been proliferating. However, there has been little research on graph database education. In response to the need to introduce college students to graph databases, this paper is the first to analyze students' errors in homework submissions of queries written in Cypher, the query language for Neo4j---the most prominent graph database. Based on 40,093 student submissions from homework assignments in an upper-level computer science database course at one university, this paper provides a quantitative analysis of students' learning when solving graph database problems. The data shows that students struggle the most to correctly use Cypher's WITH clause to define variable names before referencing in the WHERE clause and these errors persist over multiple homework problems requiring the same techniques, and we suggest a further improvement on the classification of syntactic errors.

As data grow both in size and in connectivity, the interest to use graph databases in industry has been growing rapidly. However, there has been little research on graph database education. In response to the need to introduce college students to graph databases, this paper is the first to analyze students' errors in their submissions writing different types of queries in Cypher, the query language for Neo4j--the most prominent graph database. Based on 40,093 student submission from homework assignments in an upper-level computer science database course at University of Illinois at Urbana-Champaign, this paper provides qualitative insights and quantitative analysis about students' learning when solving graph database problems. The data shows that writing more complex queries initially takes students more time and more attempts to write more complex queries correctly. Additionally, students struggle to correctly use Cypher's WITH clause to define variable names before referencing in the WHERE clause, and these errors persist over multiple homework problems requiring the same techniques.

The substantive gender and racial gaps in spatial ability create unnecessary barriers to entry to STEM and computing fields [14,15]. Data structures courses in particular are difficult for those in underrepresented groups [8,12,13]. In Chemistry, this gender gap has been closed by teaching students both spatio-imagistic and spatio-analytic strategies to solving problems with diagrams. I seek to similarly close this gap in Data Structures by developing and evaluating further techniques for teaching data structures using diagrams.

We analyze the submissions of 286 students as they solved Structured Query Language (SQL) homework assignments for an upper-level databases course. Databases and the ability to query them are becoming increasingly essential for not only computer scientists but also business professionals, scientists, and anyone who needs to make data-driven decisions. Despite the increasing importance of SQL and databases, little research has documented student difficulties in learning SQL. We replicate and extend prior studies of students' difficulties with learning SQL. Students worked on and submitted their homework through an online learning management system with support for autograding of code. Students received immediate feedback on the correctness of their solutions and had approximately a week to finish writing eight to ten queries. We categorized student submissions by the type of error, or lack thereof, that students made, and whether the student was eventually able to construct a correct query. Like prior work, we find that the majority of student mistakes are syntax errors. In contrast with the conclusions of prior work, we find that some students are never able to resolve these syntax errors to create valid queries. Additionally, we find that students struggle the most when they need to write SQL queries related to GROUP BY and correlated subqueries. We suggest implications for instruction and future research.