Erroneous examples are structured problem examples which incorporate deliberate errors that students can identify and correct to reinforce their understanding. Erroneous examples have been shown to be beneficial for student learning in various domains. However, designing effective erroneous examples requires careful planning and a deep understanding of common student misconceptions, posing a time burden on educators. In this work, we introduce a framework that leverages Large Language Models (LLMs) to automatically generate erroneous programming examples for use in introductory computer science education. We systematically evaluate the performance of various LLMs on the generation task, and find that LLMs are generally capable of generating meaningful erroneous examples along with accurate explanations. We present our preliminary findings and outline next steps for this study and future research.

While question-asking platforms have provided a wide array of pedagogical benefits and have decreased grader workload in university classes, they are limited in what types of inputs are accepted. Recent work has explored using Large Language Models to expand what can be automatically graded. In this study, we examine their use for grading handwritten mathematics questions via rubrics. Although much work remains, our preliminary results suggest that using separate prompts to first extract text and then evaluate rubric items enables LLMs to distinguish fully correct solutions from those requiring further review, thereby reducing grader workload.

This paper introduces the PrairieLearn Question Sharing System (PQSS), which enables instructors to share question generators with other instructors, either as open educational resources or privately. PQSS is integrated into PrairieLearn, an open-source, problem-driven online learning platform. PQSS addresses a critical need for more open-source assessments by making it easier for instructors to share assessments and for instructors to use those assessments. Instructors often do not share questions due to the time it takes to publish them and the lack of recognition for their work. Because it is directly integrated into PrairieLearn, PQSS reduces the aforementioned friction of sharing and using shared questions, and we can report usage statistics to help question authors receive recognition for their work. In this paper, we share design and implementation details of the system, as well as experiences using it to share course content across courses and between universities.

Manually grading free response questions remains a persistent challenge in education. While such questions offer valuable opportunities for student learning and critical thinking, their evaluation often requires substantial time and effort from instructors or teaching assistants. In addition to the grading workload, open-ended responses are susceptible to inconsistencies in scoring and may reflect unclear expectations, both of which can undermine the effectiveness and fairness of the assessment process. To address these challenges, we employed an AI-based grading system integrated in PrairieLearn to automatically evaluate student submissions to free response questions using a predefined set of rubric items. This approach not only streamlines the grading process but also enables direct comparison between AI-generated rubric applications and human judgments, providing insight into alignment and potential discrepancies. These discrepancies provided valuable insight, allowing us to iteratively revise and clarify the rubric items. Our experiences with using the AI grading system across several computing courses suggest that even experienced educators face difficulties articulating rubrics that are both specific and interpretable. We furthermore argue that more attention should be given to the iterative development and evaluation of rubrics.

Computerized exams have benefits for large enrollment courses and computer science classes, specifically. In this research paper, we compare student self-reported test anxiety between two modes of administering computerized exams: a computer-based testing facility (CBTF) and a bring-your-own-device (BYOD) setup. We conducted crossover design experiments in two computer science courses, measuring trait anxiety, as well as students' test anxiety and their test performance after each exam.

Alternative grading practices, such as specifications grading, have been reported to increase student engagement, decrease instructor workload, and ensure a minimum mastery of all course topics. However, there are also reports of decreased student learning and there have been few comparisons between different policies. In this experience report, we explore the effects of adopting different types of specifications grading systems in a project-based Embedded Systems course taught at a South American institution. We tested three different grading policies for exams in three different course offerings, varying both the number of times mastery needed to be demonstrated and whether exams specified two (pass/fail) or five (I, D, C, B, A) proficiency levels. For each policy, we list potential problems we were trying to address and evaluate how the change impacted the course. We use a mixed-methods approach to do a retrospective analysis, focusing on three aspects: (i)exam attendance and pass rates,(ii) grades on labs and projects and, (iii) student perceptions. We find that attendance remains high and pass rates increase with the pass/fail model. Also, lab and project grades decrease after students receive a passing grade in the first exam, indicating a possible decrease in engagement with the course. In terms of student perception, students described a mix of procrastination, low engagement with the course, and a desire to learn and create a challenging project. Finally, we discuss how we believe these findings could be used to increase engagement and learning.

Delivery of affordable, secure, and scalable assessments is an essential component of large university courses, whether online or in-person. The transition to Computer-Based Testing (CBT) has a transformational effect on pedagogy. Modern CBT systems provide almost unlimited flexibility in the types of questions they can support for manual grading and autograding. In this BoF, faculty interested in learning about various components of CBT and how to implement it at their institution are invited to ask their questions and learn from others who have already done this. To facilitate these discussions, in this BOF we will break into four smaller groups to discuss CBT pedagogy, building and sharing question banks, technical or logistical considerations, and building buy-in from all levels of the institution.

In this paper, we explore three different methods for administering computer-based tests at scale: (1) a dedicated Computer-Based Testing Center (CBTC), (2) Bring Your Own Device (BYOD) exams proctored in person in the classroom, and (3) BYOD exams proctored online via Zoom. We conducted two randomized crossover experiments to compare pairs of modalities against each other (CBTC vs BYOD-in-person and CBTC vs BYOD-online). We found that testing modality did not impact students' exam performance or students' preparation before exams. However, we observed that students preferred the modalities in which they had recently received the highest scores. Our results indicate that several different modalities can be effectively used to administer testing at scale for CS courses.

Developing embedded microcontroller code is a complex task, especially for undergrad students new to this area. These students often make high-level conceptual mistakes beyond the scope of commercial standards like MISRA-C. These conceptual errors need to be checked manually through code feedback, a process that is time-consuming, error-prone, and does not scale well with an increasing number of students and/or assignments. In this paper, we present an embedded-check an automated tool that can detect common and critical errors students make when learning to code firmware. A set of 13 rules (baremetal and FreeRTOS) was devised based on our experience from several years of teaching Embedded systems. To validate our tool, we compared its results with manual code review of N=99 projects from the last 3 course offerings. We furthered our analysis by running our tool on N=1132 coding lab submissions that did not receive manual feedback and were used as part of classroom activities. We found that the top-3 errors flagged in the projects were already present when students completed the lab activities. We found that (i) our tool also identified all issues discovered during manual code feedback, (ii) our tool detected issues in 86% of student submissions, whereas manual code feedback only flagged 28% of the submissions as problematic, and (iii) 94.3% of students made some code quality error on individual assignments. Within this results, we believe that our tool can have a significant impact when used both as an formative assessment tool to support learning and as a learning analytics tool to improve teaching.

To address the challenges of running exams in large enrollment CS courses, we set up and operated an in-person testing center at a minority serving institution. We have run the testing center for two quarters, proctoring over 6,000 exams for eight CS courses with approximately 1,800 students. In this experience report, we discuss the motivation for the testing center, its set-up and operation, and the lessons that we have learned from our first two quarters of operation. In addition, we present student and instructor feedback regarding use of the testing center, future steps, and improvements.

Prior work has found that women tend to report lower sense of belonging compared to men in STEM and computing contexts, which may discourage women's persistence. Collaborative learning has been shown to improve students' sense of belonging in some STEM and computing courses relative to traditional lecturing; however, these studies tend to focus on a single course or the first implementation of such pedagogical changes. Our study explores whether these trends generalize by measuring students' sense of belonging across three non-introductory computing courses that have consistently used collaborative learning activities over three semesters. We ask the following research question: Is collaborative learning generally associated with an increased sense of belonging, especially for women? We found that while there were variations across courses, students' reported sense of belonging improved in all courses. Notably, women's reported sense of belonging improved 15% whereas men's reported sense of belonging improved by 11%. Our findings complement prior studies by providing evidence of a relationship between increased sense of belonging and collaborative learning, and suggest students' sense of belonging is malleable beyond the first year. These findings challenge critiques of past studies as being isolated to single courses or conducted only immediately after an effort to change a course, suggesting pedagogical changes may hold promise in improving students' affective outcomes.

Affordable, secure, and scalable assessment delivery is an essential component of large university courses, whether online or in-person. The switch to Computer-Based Testing (CBT) can have a surprising, and almost transformational effect on pedagogy. Modern CBT systems provide almost unlimited flexibility in the types of questions they can support, for both manual grading and autograding, and CBT has now been adopted at several universities and is under serious consideration at others. In this BoF, faculty interested in learning about CBT and how to implement it at their institution are invited to ask their questions. Faculty experienced with CBT are invited to share how CBT has changed their approach, pedagogy, and behavior and how to advocate for its adoption.

We compare the exam security of three proctoring regimens of Bring-Your-Own-Device, synchronous, computer-based exams in a computer science class: online un-proctored, online proctored via Zoom, and in-person proctored. We performed two randomized crossover experiments to compare these proctoring regimens. The first study measured the score advantage students receive while taking un-proctored online exams over Zoom-proctored online exams. The second study measured the score advantage of students taking Zoom-proctored online exams over in-person proctored exams. In both studies, students took six 50-minute exams using their own devices, which included two coding questions and 8--10 non-coding questions. We find that students score 2.3% higher on non-coding questions when taking exams in the un-proctored format compared to Zoom proctoring. No statistically significant advantage was found for the coding questions. While most of the non-coding questions had randomization such that students got different versions, for the few questions where all students received the same exact version, the score advantage escalated to 5.2%. From the second study, we find no statistically significant difference between students' performance on Zoom-proctored vs. in-person proctored exams. With this, we recommend educators incorporate some form of proctoring along with question randomization to mitigate cheating concerns in BYOD exams.

Allowing students to re-attempt assessments has been shown to be effective in traditional university-level courses in improving student mastery of course content. In this paper, we analyse an intensive programming introductory experience, where first semester university students' full load is a single semester-long course that teaches the basics of programming and software engineering. We study its use of mastery-based grading: offering five (formative) low-stakes quizzes (with retakes), each focused on a single topic, and five (summative) higher-stakes exams that assess all learning objectives. Our research questions are: (i) ''Do second chances help students to increase their performance over time in intensive courses?''; and (ii) ''Are second chances effective in reducing stress/mental load/weight of assessments in intensive courses?''. We find that (i) offering second chances on quizzes decreases the number of students at risk of failing before the first exam; (ii) students' proficiency in coding tasks (as measured by exam grades) improve during the semester; and (iii) that our schedule reduces anxiety and mental load for students, but only after students take the first chance.

Collaborative learning is an evidence-based instructional practice that has been widely used in higher education, but it is not a silver bullet and requires careful design and implementation to yield the desired benefits. Prior work has also shown that collaborative learning is especially effective for female students, who are historically underrepresented in Science, Technology, Engineering, Mathematics (STEM), and Computing fields, suggesting a promising prospect for applying collaborative learning in these disciplines. In spring and fall of 2022 at a large public university, an upper-level required computer science course using collaborative learning was offered in a hybrid format, with great flexibility in meeting time (students could meet and collaborate at the “instructor-scheduled” meeting time, or they could pick their preferred “student-scheduled” time) and team consistency (students were encouraged, but not required, to work with a fixed team throughout the semester). To understand the effects of meeting time preference and team consistency, we measured students' learning outcomes (exam performance) and experience (sense of belonging and satisfaction about team dynamics in collaborative learning), and conducted linear regression analyses. We also investigated whether the effects were different for male and female students. We found that students' meeting time preference had no significant effect on their exam performance, sense of belonging, or satisfaction, and the non-significant effects were homogeneous for both gender groups. We also found that having a consistent team had significantly positive effects on exam performance and sense of belonging, but no significant effect for satisfaction. Moreover, the effect of team consistency on exam performance was significantly stronger for female students than male students. Our findings justified the option to give students meeting time flexibility since it did not hurt their learning experience and outcome, and encouraged exploring effective approaches to forming consistent teams that make students intrinsically want to work with them. The gender difference in effects of team consistency on exam performance aligned with previous literature and served as evidence to use collaborative learning in computing and STEM classrooms.

"A's for All (as time and interest allow)" is a position that says it is increasingly possible to aim for a world in which students can achieve any grade (level of mastery) that they are willing to work for, even if some students take longer than others or require more practice to get there. Achieving this goal would have profound effects on fairness, equity, and participation in computing, to say nothing of student learning outcomes. We describe what this goal would entail, why it is worth pursuing, what the mechanism and policy requirements are for making progress, and why now is a good time to do it. We give specific and actionable recommendations, many based on our own experience so far, that our colleagues who are excited about the approach can put into immediate practice, and address a number of concerns and objections that our proposal may raise. Importantly, our proposed approach is not all-or-nothing, but all-or-something: there are many things instructors can do within existing policy frameworks and course constraints to move their course experience in this direction.

In recent years, a diverse body of research in computing education has discussed new pedagogies and curriculum changes to improve learning and students' experiences. Topics such as growth mindset, mastery learning, grading for equity, and specifications grading are important steps towards the Holy Grail: "A's for All" (as time and interest allow). In this new teaching approach, the "A" line does not move, but instead every student is given an opportunity to achieve proficiency and earn it, as long as they are willing to put in the time and effort it takes. In other words, students can all achieve the same learning outcomes at a different pace, instead of the traditional approach where students achieve different learning outcomes in a fixed amount of time. This workshop will provide educators and administrators with tools to implement the "A's for All" (as time and interest allow) approach in their courses and institutions. It will take attendees through elements of advocacy, hands-on randomized question generator design and implementation using a computer-based assessment system, best practices, and course policies to reduce friction.

A new computational linear algebra course was developed and offered at a large public university in the Midwest. This new course traded off some of the lecture time in the pre-existing traditional linear algebra course for applied computational materials taught in a flipped-classroom lab setting. We compare exam performance in a subsequent numerical methods course from students having taken either the new computational or traditional course, while controlling for student performance in prerequisite computer science and mathematics courses. We find that for students with less mathematics background (i.e., those who needed to take Calculus 2 at the university), taking the new computational linear algebra course has significant positive impact on their average exam performance in the subsequent course. The performance of students with more initial mathematics background (i.e., those who already had credit for Calculus 2) is not significantly affected by the computational vs. traditional course backgrounds.

Breast cancer (BC) treatments are often aggressive for the patients, causing severe impairments to their physical condition. Physical supervised exercise has proven to be beneficial for the patients' physical and emotional health recovery. In this work, we present an eHealth solution based on optical fiber sensors (OFSs), to monitor the physical activity of BC patients, during their exercises. Such solution will allow the patients to exercise from the comfort of their home, with the continuous supervision of the physiotherapist, and with a possibility to connect remotely with other patients. The use case, presented in this work, is focused on the design of optical fiber based eHealth enablers to monitor the exercises prescribed in the recovery of the handgrip force, often affected by the treatments that BC patients are subjected to.

This paper presents the results of a controlled crossover experiment designed to measure the score advantage that students have when taking exams asynchronously (i.e., the students can select a time to take the exam in a multi-day window) compared to synchronous exams (i.e., all students take the exam at the same time). The study was performed in an upper-division undergraduate computer science course with 321 students. Stratified sampling was used to randomly assign the students to two groups that alternated between the two treatments (synchronous versus asynchronous exams) across a series of four exams during the semester. These non-programming exams consisted of a mix of multiple choice, checkbox, and numeric input questions. For some questions, the parameters were randomized so that students received different versions of the question and some questions were identical for all students. In our results, students taking the exams asynchronously had scores that were on average only 3% higher (0.2 of a standard deviation). Furthermore, we found that the score advantage was decreased by the use of randomized questions, and it did not significantly differ based on the type of question. Thus, our results suggest that asynchronous exams can be a compelling alternative to synchronous exams.