We present results of an intervention integrating a tool promoting metacognitive study behaviors, CompassX, in a Data Structures and Algorithms (CS2) course. Metacognition---commonly referred to as ''thinking about thinking''---has been consistently linked to improved learning strategies and student achievement. However, no prior literature has practiced an intervention that addresses all three commonly-accepted phases of metacognition. Thus in this work, we share the key features of CompassXthat promote metacognitive study behaviors, how our users engaged with those features, and how continued practice of metacognition using those features is related to improved learning outcomes. Students used CompassX voluntarily and some students did not fully engage with all metacognition-based features. However, continued engagement with a metacognitive feature appears to be indicative of higher exam scores. This also implies the possibility of utilizing a metacognitive tool to improve the performance of student outcome modeling by collecting a new type of behavioral data.

Academic procrastination is a prevalent issue among students, detrimentally impacting their academic performance and well-being. Although existing Acceptance and Commitment Therapy (ACT) interventions have aimed to alleviate this problem, they often follow generic ACT protocols, lacking customizations to address the challenges of academic procrastination. In this study, we introduce the ClearMind Workshop, a unique workshop that utilizes web-based ACT content and additional content to explore the nature, causes, and management strategies of procrastination. We assessed the effectiveness of the workshop by collecting pre- and post-survey data from workshop participants (N=19) and a control group (N=38) on their level of procrastination, anxiety, and subjective happiness and their use of coping strategies. Results reveal that the workshop reduced academic procrastination and anxiety in computing students. Participants adopted healthier coping strategies such as positive reframing and active coping strategies, while practicing less unhealthy ones such as self-blame and avoidance strategies. These findings encourage future directions on assessing long-term effects with a larger sample size, exploring additional variables, and integrating technology-based or online interventions.

This panel will highlight the experiences of Black, Asian, and Latinx women and non-binary instructors as junior faculty. Despite efforts to broaden participation, CS departments are often quite homogeneous, forcing faculty of marginalized identities to attenuate certain aspects of, or entire identities, to better "fit in" with their colleagues. We will discuss the impacts on faculty and engage with the audience to identify avenues of individual (i.e., colleagues) and systemic (i.e., department culture and policies) allyship.

This work piloted an international replication and generalization study on an existing prediction model called PreSS. PreSS has been developed and validated over nearly two decades and can predict student performance in CS1 with nearly 71% accuracy, at a very early stage in the learning module. Motivated by a prior validation study and its competitive modelling accuracy, we chose PreSS for such an international replication and generalization study. The study took place in two countries, with two institutions in Ireland and one institution in the US, totalling 472 students throughout the academic year 2020-21. In doing so, this study addressed a call from the 2015 ITiCSE working group for the educational data mining and learning analytics community: systematically analyse and verify previous studies using data from multiple contexts to tease out tacit factors that contribute to previously observed outcomes. This pilot study achieved 90% accuracy, which is higher than the prior work's. This encouraging finding sets the foundations for a larger scale international study. This paper describes in detail the pilot replication and generalization study and our progress on the larger scale study which is taking place across six continents.

A Concept Inventory (CI) is an assessment to measure student conceptual understanding of a particular topic. This article presents the results of a CI for basic data structures (BDSI) that has been previously shown to have strong evidence for validity. The goal of this work is to help researchers or instructors who administer the BDSI in their own courses to better understand their results. In support of this goal, we discuss our findings for each question of the CI using data gathered from 1,963 students across seven institutions.

There is little research on why underrepresented minorities are less likely to specifically study Machine Learning and Artificial Intelligence (ML/AI). We surveyed 159 undergraduate students about their interest in, exposure to, and personal views on ML/AI in order to explore variations in responses by self-reported gender and race/ethnicity groups. We found that students underrepresented by race/ethnicity are ~6 times less likely to take a traditional ML/AI course than those not underrepresented by race/ethnicity, but no significant difference was found between gender representation. Additionally, students underrepresented by race/ethnicity are more likely to report interest in social, cultural, and political impacts of ML/AI rather than the more technical aspects of ML/AI itself, which is a prevalent interest of students not underrepresented by race/ethnicity. We explore potential reasoning for this difference through further analysis of their survey responses. Encouragingly, we find that regardless of representational status 72.0% of students who report lack of interest in a traditional introductory course are interested in a ML/AI course that focuses more on the political, philosophical, and ethical issues raised by ML/AI and its impacts on society. Our findings suggest that a 'CS Principles" style introductory ML/AI course, emphasizing social and political impacts, could be an effective way to promote diversity in ML/AI.

Our prior work found differences in study habits between high- and low-performers in a small-scale qualitative study, and this work seeks to verify and extend these findings by examining the study habits of a larger population of CS1 students. To do this, we devised a survey based on the findings of our prior qualitative study. The responses of CS1 students reveals that some study habits are more frequently practiced by higher-performers then lower-performers or vice versa. One concern with these findings is that the differences in study habits might simply be explained by prior experience. As such, we compare study habits between students with and without prior experience as well. We find that although prior experience translates to better class performance, it is not associated with the same study habits as lower- and higher-performers, suggesting that prior experience and study habits are separately associated with better student performance. These findings encourage further inquiry into the role of study habits in student success and whether explicit instruction on better study habits might be the basis for successful future interventions.

Many prior work from various disciplines, including computing education, investigated how students interact with quizzes and how their interactions impact their learning outcomes. However, most of their results are based on quantitative analysis which does not offer detailed nuances behind the actual behaviors of students. Some prior studies took a qualitative approach but they focused mainly on students' perception on quizzes in general or student behaviors strictly while they work on the quizzes. Thus, our work conducted a qualitative study on how students interact with quizzes in a broader context, by also including their motivation for doing quizzes and the next steps they take after completing a quiz. By investigating observed student behaviors from the interviews with respect to their performance on the midterm exam, our results revealed a variety of student interactions and how they are related to performance. Our findings also provide some suggestions on how instructors should engage students with quizzes for effective learning.

Computer Science now has a number of validated instruments available for measuring student knowledge or interest in computing including the Second CS1 Assessment (SCS1), The Basic Data Structures Inventory (BDSI), the Computing Attitudes Survey (CAS), and the Digital Logic CI. These instruments can be used by instructors to assess their students and/or their own teaching. They can also be used by researchers to measure students' learning or attitudes. The goal of this BOF is to help instructors and researchers gain a better understanding of how to use these instruments, whether that be to get started in education research, to compare student learning across terms/curricular revisions, or just to learn more about student misconceptions. We will begin by discussing the available instruments, their purpose, and how to obtain them. Then we will open the discussion to the group on what they would like to measure, how these instruments might work for them, and how to best employ them with their students.

Being able to accurately self-assess one's own understanding is a crucial metacognitive skill that enables students to allocate their study time and energy more effectively. Prior works have explored different metacognition-based interventions but they were either not reliably effective or heavy-weight. In this work, we present Compass, an intervention composed of self-assessment quizzes that additionally ask students to self-report their confidence level per answer in order to automatically recommend prioritized sets of resources. We found that although frequent self-assessment quiz taking correlated with higher exam performance, the repeated practice of self-reporting confidence levels did not seem to benefit students' metacognitive accuracy over time. Our findings also challenge the commonly accepted hypothesis that high-performing students have high metacognitive accuracy.

Computer Science now has a number of validated instruments available for measuring student knowledge or interest in computing (SCS1, BDSI, CAS, Digital Logic CI, etc.). But when and how should instructors and researchers use these instruments? In this BOF, we will begin by discussing the available instruments, their purpose, and how to obtain them. Then we will open the discussion to the group on what they would like to measure, how these instruments might work for them, and how to best employ them with their students.

A Concept Inventory (CI) is a validated assessment to measure student conceptual understanding of a particular topic. This work presents a CI for Basic Data Structures (BDSI) and the process by which the CI was designed and validated. We discuss: 1) the collection of faculty opinions from diverse institutions on what belongs on the instrument, 2) a series of interviews with students to identify their conceptions and misconceptions of the content, 3) an iterative design process of developing draft questions, conducting interviews with students to ensure the questions on the instrument are interpreted properly, and collecting faculty feedback on the questions themselves, and 4) a statistical evaluation of final versions of the instrument to ensure its internal validity. We also provide initial results from pilot runs of the CI.

As computer science enrollments continue to surge, exam grading requires significant instructional resources. Online grading platforms have been developed in recent years and have been adopted at a number of institutions; however, their effectiveness compared with traditional grading on paper is not fully known. This study is the first in CS to compare online and paper grading. Comparing overall time to grade, including all factors, online grading doesn't show a consistent advantage compared with paper grading. We observed that online grading is much faster during the actual grading phase, but some of this benefit is offset by the additional overhead prior to grading (e.g., scanning) for online grading. Examining student and grader preferences based on feedback, both groups show a strong preference for the online format, predominately due to the convenience of the online platform. Graders report being able to grade more accurately online with the ability to modify rubrics, but that grading on paper tends to result in more social interactions among graders.

Although recent work in computing has discovered multiple techniques to identify low-performing students in a course, it is unclear what factors contribute to those students' difficulties. If we were able to better understand the characteristics of such students, we may be better able to help those students. This work examines the characteristics of low- and high-performing students through interviews with students from an introductory computing class. We identify a number of relevant areas of student behavior including how they approach their exam studies, how they approach completing programming assignments, whether they sought help after identifying misunderstandings, how and from whom they sought help, and how they reflected on assignments after submitting them. Particular behaviors within each area are coded and differences between groups of students are identified.

A number of recent studies in computer science education have explored the value of various data sources for early prediction of students' overall course performance. These data sources include responses to clicker questions, prerequisite knowledge, instrumented student IDEs, quizzes, and assignments. However, these data sources are often examined in isolation or in a single course. Which data sources are most valuable, and does course context matter? To answer these questions, this study collected student grades on prerequisite courses, Peer Instruction clicker responses, online quizzes, and assignments, from five courses (over 1000 students) across the CS curriculum at two institutions. A trend emerges suggesting that for upper-division courses, prerequisite grades are most predictive; for introductory programming courses, where no prerequisite grades were available, clicker responses were the most predictive. In concert, prerequisites and clicker responses generally provide highly accurate predictions early in the term, with assignments and online quizzes sometimes providing incremental improvements. Implications of these results for both researchers and practitioners are discussed.

As enrollments and class sizes in postsecondary institutions have increased, instructors have sought automated and lightweight means to identify students who are at risk of performing poorly in a course. This identification must be performed early enough in the term to allow instructors to assist those students before they fall irreparably behind. This study describes a modeling methodology that predicts student final exam scores in the third week of the term by using the clicker data that is automatically collected for instructors when they employ the Peer Instruction pedagogy. The modeling technique uses a support vector machine binary classifier, trained on one term of a course, to predict outcomes in the subsequent term. We applied this modeling technique to five different courses across the computer science curriculum, taught by three different instructors at two different institutions. Our modeling approach includes a set of strengths not seen wholesale in prior work, while maintaining competitive levels of accuracy with that work. These strengths include using a lightweight source of student data, affording early detection of struggling students, and predicting outcomes across terms in a natural setting (different final exams, minor changes to course content), across multiple courses in a curriculum, and across multiple institutions.

To be effective instructors and CS education researchers, we must identify and understand student difficulties surrounding core computing topics. This study examines student difficulties with the basic data structures commonly found in CS2 courses. Initial exploration of student thinking began with think-aloud interviews with students. These interviews centered on open-ended questions that were iteratively improved upon based on analysis of interview transcripts. The revised open-ended questions were then posed to 249 students during an end-of-term final exam study session. Using the explanations and justifications included by students, responses to the questions were coded and summarized. This work characterizes the difficulties revealed by student responses, and provides details of their prevalence among the examined student population.

Since computing education began, we have sought to learn why students struggle in computer science and how to identify these at-risk students as early as possible. Due to the increasing availability of instrumented coding tools in introductory CS courses, the amount of direct observational data of student working patterns has increased significantly in the past decade, leading to a flurry of attempts to identify at-risk students using data mining techniques on code artifacts. The goal of this work is to produce a systematic literature review to describe the breadth of work being done on the identification of at-risk students in computing courses. In addition to the review itself, which will summarize key areas of work being completed in the field, we will present a taxonomy (based on data sources, methods, and contexts) to classify work in the area.

Jigsaw learning is a cooperative learning technique enabling students to teach and learn from their peers. Although prior studies investigated the efficacy of Jigsaw learning in computing education by measuring student performance after Jigsaw activities, this work reports on student and instructor experiences with Jigsaw learning. Jigsaw activities were performed in lectures throughout the term and student experience data was collected through student surveys. The survey results reveal that 72% of survey respondents thought the Jigsaw activities helped their learning. Furthermore, 75% of respondents acknowledged their role in class as a more engaged learner and 44% of those students identified themselves as taking on the role of a teacher. The instructor found Jigsaw learning to be labor intensive but observed higher levels of student engagement.

Peer Instruction (PI) is an active learning pedagogy that has been shown to improve student outcomes in computing, including lower failure rates, higher exam scores, and better retention in the CS major. PI's key classroom mechanism is the PI question: a formative multiple choice question on which students vote, then discuss, then vote again. While research indicates that PI questions lead to learning gains for students, relatively little is known about the questions themselves and how faculty employ them. Additionally, much of the work has examined PI data collected by researchers operating in a quasi-experimental setting. We examine data collected incidentally by multiple instructors using PI as a pedagogical technique in their classroom. We look at how many questions instructors use in their courses, the difficulty level of the questions, and normalized gain, a metric that looks at increases in student correctness between individual and group votes. We find normalized gain levels similar to those in existing literature, indicating that students are learning, and that most questions, even those developed by instructors new to PI, fall within recommended difficulty levels, indicating instructors can create good PI questions with little training. We also find that instructors add PI questions over the first several iterations of a new PI course, showing that they find PI questions valuable and suggesting that full development of PI materials for a course may take multiple semesters.

Peer Instruction is an active learning pedagogy that has been shown to improve student outcomes in computing, including lower failure rates, higher exam scores, and better retention in the CS major. A key classroom mechanism for Peer Instruction is the "clicker question": a formative multiple-choice question on which students vote, then discuss, then vote again. While research indicates that clicker questions lead to learning gains for students, relatively little is known about the questions themselves and how faculty employ them. Additionally, much of the work has examined clicker data collected by CS Education researchers operating in a quasi-experimental setting. In this project, we examine clicker data collected incidentally by multiple instructors using clickers as a pedagogical technique in their classroom. This work represents a first effort to systematically evaluate how instructors use clicker questions, including how many clicker questions are used in a course, how difficult the questions used are, and whether instructors add or modify questions over time.

As student enrollments in computer science increase, there is a growing need for pedagogies that scale. Recent evidence has shown Peer Instruction (PI) to be an effective in-class pedagogy that reports high student satisfaction even with large classes. Yet, the question of the scalability of traditional lecture versus PI is largely unexplored. To explore this question, this work examines publicly available student evaluations of computer science courses across a wide range of class sizes (50--374 students) over a four year period. It first compares evaluations regardless of size and confirms prior work that PI classes are better appreciated by students than traditional lecture. It then examines how course evaluations change with class size and provides evidence that PI achieves a smaller decline in evaluations as class size increases.

Identifying at-risk students early in the term is valuable. It is because an instructor can have more time to provide extra support, and students can also estimate how much extra effort they should put on to succeed in class. Prior work showed it is possible to predict at-risk students, but they either did not provide a specific prediction method or are too onerous to implement. Thus, my dissertation will develop and evaluate more robust, universal, and simple prediction methodology to classify at-risk students and propose how to automatically generate customized practice materials for early intervention. Once the methodology becomes robust, I will implement publicly accessible educational software application so that other CS instructors can easily adopt this method.

Being able to identify low-performing students early in the term may help instructors intervene or differently allocate course resources. Prior work in CS1 has demonstrated that clicker correctness in Peer Instruction courses correlates with exam outcomes and, separately, that machine learning models can be built based on early-term programming assessments. This work aims to combine the best elements of each of these approaches. We offer a methodology for creating models, based on in-class clicker questions, to predict cross-term student performance. In as early as week 3 in a 12-week CS1 course, this model is capable of correctly predicting students as being in danger of failing, or not, for 70% of the students, with only 17% of students misclassified as not at-risk when at-risk. Additional measures to ensure more broad applicability of the methodology, along with possible limitations, are explored.