We report our development of a survey measuring students' sense of belonging in computing courses. Existing definitions and surveys are helpful in identifying and measuring perceptions of belonging (e.g., feeling accepted, included, and valued) but give little insight into why students feel they do (not) belong. To address this limitation, we use a conceptual framework that breaks sense of belonging into four components: motivations, opportunities, competencies, and perceptions. Survey item development was based on interviews with undergraduate computing students that were analyzed using this conceptual framework. We revised survey items based on 1) cognitive interviews and 2) psychometric properties of pilot survey data with participants from multiple required courses typically taken by first- and second-year undergraduate computing students. The resulting survey consists of 12 items with three items per component of the conceptual framework. A four-factor confirmatory factor analysis model indicated good empirical fit with the proposed structure of the framework (N=514). Cronbach's alpha indicated good internal consistency within each component (>0.70) as well as across all 12 items (>0.90). The inter-component correlations were relatively high for all pairs of components (>0.60), aligning with previous interview findings that indicate the components may influence each other. Altogether, we have sufficient evidence of validity for the survey's use in undergraduate computing courses and discuss avenues for future work.

To make cybersecurity concepts more accessible to non-technical students, we present our classroom-based study of CyberGuardian, an educational game designed to teach cryptographic primitives through simplified, real-world scenarios. Players assume the role of a cybersecurity advisor, providing guidance to simulated clients and analyzing authentic messages to assure confidentiality, data integrity, and authentication through three cryptographic primitives: symmetric encryption, asymmetric encryption, and digital signature. CyberGuardian is freely available, easy to deploy at scale, and implemented as a single-player offline game compatible with macOS and Windows. Our learning objectives are to enable students to: (1) Identify how an adversary can interfere with communications; (2) Explain how cryptographic primitives can mitigate dangers; and (3) Apply cryptographic primitives to reduce vulnerabilities.

This paper introduces the PrairieLearn Question Sharing System (PQSS), which enables instructors to share question generators with other instructors, either as open educational resources or privately. PQSS is integrated into PrairieLearn, an open-source, problem-driven online learning platform. PQSS addresses a critical need for more open-source assessments by making it easier for instructors to share assessments and for instructors to use those assessments. Instructors often do not share questions due to the time it takes to publish them and the lack of recognition for their work. Because it is directly integrated into PrairieLearn, PQSS reduces the aforementioned friction of sharing and using shared questions, and we can report usage statistics to help question authors receive recognition for their work. In this paper, we share design and implementation details of the system, as well as experiences using it to share course content across courses and between universities.

MeetingMayhem is a web-based educational game focused on adversarial thinking in the context of network security. In particular, this game gently and non-technically introduces students without prior cybersecurity knowledge to the Dolev-Yao network intruder model. In MeetingMayhem, three students take on the roles of two agents and an adversary. Two agents need to agree on a time and location to exchange an essential asset. The agents communicate through a network controlled by the adversary without knowing the adversary's identity. The adversary can insert, block, or modify messages. Students can play different roles, try different strategies, interact with different players, and send different messages. MeetingMayhem is available as a Docker Image, with open-source code. MeetingMayhem provides an engaging and innovative way for novice students to learn about adversarial thinking and cryptography within 2 hours.

CyberGuardian is an interactive educational game that introduces students to foundational concepts in cybersecurity through authentic scenarios. Specifically, the game teaches students about cryptographic primitives and adversarial thinking. Targeted at students without computer science backgrounds, the game immerses players in practical cybersecurity challenges using simplified technical concepts. Players assume the role of a cybersecurity advisor, providing guidance to simulated clients and analyzing authentic messages to assure confidentiality, data integrity, and authentication through three cryptographic primitives: symmetric encryption, asymmetric encryption, and digital signature. Implemented as a single-player offline experience on Mac and Windows, CyberGuardian is scalable and broadly available.

ILDBug is a novel debugging approach inspired by the pedagogical technique Interactive Lecture Demonstrations (ILDs). During ILDs, students predict a demonstration's result, experience the demonstration, and then reflect on their experience. We adapted this process to teach debugging (ILDBug) by having students engage in detecting the bug for prediction, then locating and correcting the bug for experience, and then reflecting on the debugging process. The ILDBug approach is designed to be a lightweight technique to create a debugging exercise that is adaptable to many contexts. This demo will cover the high-level pieces the audience needs to create their own ILDBug using our approach, an example ILDBug in an introductory programming context so they can see it in action, and finally tips on how to adapt our technique to fit their own classroom context.

Graph data structures and algorithms play an essential role in computer science, and one of the ultimate goals of learning graphs is to solve more complicated algorithm design problems with them. A common way to solve a novel, complex problem is to reduce the problem to a standard graph problem, which often requires modeling a graph, and one essential way to model a graph is a technique called graph layering. Graph layering is often considered difficult by students and rarely studied by computer science education researchers despite its significance in algorithm design. To understand students' struggles with graph layering and improve teaching of algorithm designs, we conducted this qualitative study using think-aloud interviews with current students from an algorithm course. Participants were asked to solve algorithm design problems meant to be solved with graph layering. We used thematic analysis to extract difficulties observed in these interviews. We share our preliminary findings in this poster, and propose next steps for this study and future research.

Computerized exams have benefits for large enrollment courses and computer science classes, specifically. In this research paper, we compare student self-reported test anxiety between two modes of administering computerized exams: a computer-based testing facility (CBTF) and a bring-your-own-device (BYOD) setup. We conducted crossover design experiments in two computer science courses, measuring trait anxiety, as well as students' test anxiety and their test performance after each exam.

Background: Proof Blocks is a software tool that enables students to construct proofs by assembling prewritten lines and gives them automated feedback. Prior work on learning gains from Proof Blocks has focused on comparing learning gains from Proof Blocks against other learning activities such as writing proofs or reading.

Sense of belonging correlates with retention rates for university-level students especially for those in their earlier years [5, 9, 10] and for those in science, technology, engineering, and math (STEM) and computing majors [2, 8]. In computing, women and students of color (i.e., Black, Latinx, and Indigenous) tend to report lower sense of belonging compared to men [6, 7]. Unfortunately, we lack consensus on how sense of belonging should be defined, and why students come to feel like they do or do not belong. Past definitions (e.g., [3, 4] help identify educational contexts and students’ feelings, they do not necessarily provide coherent insights into why students develop these feelings.

We report on our experiences fielding MeetingMayhem, an interactive game that we developed, which introduces students to fundamental concepts in network security, cybersecurity, and adversarial thinking. The game is intended for students who do not necessarily have any prior background in computer science. Assuming the role of agents, two players exchange messages over a network to try to agree on a meeting time and location, while an adversary interferes with their plan. Following the Dolev-Yao model, the adversary has full control of the network: they can see all messages and modify, block, or forward them. We designed the game as a web application, where groups of three students play the game, taking turns being the adversary. The adversary is a legitimate communicant on the network, and the agents do not know who is the other agent and who is the adversary. Through gameplay, we expect students to be able to (1) identify the dangers of communicating through a computer network, (2) describe the capabilities of a Dolev-Yao adversary, and (3) apply three cryptographic primitives: symmetric encryption, asymmetric encryption, and digital signatures. We conducted surveys, focus groups, and interviews to evaluate the effectiveness of the game in achieving the learning objectives. The game helped students achieve the first two learning objectives, as well as using symmetric encryption. We found that students enjoyed playing Meeting Mayhem. We are revising MeetingMayhem to improve its user interface and to better support students to learn about asymmetric encryption and digital signatures.

Prior work has found that women tend to report lower sense of belonging compared to men in STEM and computing contexts, which may discourage women's persistence. Collaborative learning has been shown to improve students' sense of belonging in some STEM and computing courses relative to traditional lecturing; however, these studies tend to focus on a single course or the first implementation of such pedagogical changes. Our study explores whether these trends generalize by measuring students' sense of belonging across three non-introductory computing courses that have consistently used collaborative learning activities over three semesters. We ask the following research question: Is collaborative learning generally associated with an increased sense of belonging, especially for women? We found that while there were variations across courses, students' reported sense of belonging improved in all courses. Notably, women's reported sense of belonging improved 15% whereas men's reported sense of belonging improved by 11%. Our findings complement prior studies by providing evidence of a relationship between increased sense of belonging and collaborative learning, and suggest students' sense of belonging is malleable beyond the first year. These findings challenge critiques of past studies as being isolated to single courses or conducted only immediately after an effort to change a course, suggesting pedagogical changes may hold promise in improving students' affective outcomes.

Background : Proof Blocks is a software tool that enables students to construct proofs by assembling prewritten lines and gives them automated feedback. Prior research has shown that students learn as much from an activity where they use Proof Blocks as where they write proofs. However, in both cases students first read a book chapter. Prior research was not able to differentiate between the learning gains achieved from reading versus proof practice. Purpose : This study aims to measure learning gains from reading a book chapter versus completing Proof Blocks. Methods : We conducted a randomized controlled trial with three experimental groups: one that only read a book chapter, one that only completed Proof Blocks, and one that did both. Findings : The group that completed only Proof Blocks had the smallest learning gains. The group that read the book chapter and completed the Proof Blocks activity performed marginally better than students who only read the book chapter, but it is not clear if the source of this improvement was the Proof Blocks or just exposure to more examples.

Collaborative learning is an evidence-based instructional practice that has been widely used in higher education, but it is not a silver bullet and requires careful design and implementation to yield the desired benefits. Prior work has also shown that collaborative learning is especially effective for female students, who are historically underrepresented in Science, Technology, Engineering, Mathematics (STEM), and Computing fields, suggesting a promising prospect for applying collaborative learning in these disciplines. In spring and fall of 2022 at a large public university, an upper-level required computer science course using collaborative learning was offered in a hybrid format, with great flexibility in meeting time (students could meet and collaborate at the “instructor-scheduled” meeting time, or they could pick their preferred “student-scheduled” time) and team consistency (students were encouraged, but not required, to work with a fixed team throughout the semester). To understand the effects of meeting time preference and team consistency, we measured students' learning outcomes (exam performance) and experience (sense of belonging and satisfaction about team dynamics in collaborative learning), and conducted linear regression analyses. We also investigated whether the effects were different for male and female students. We found that students' meeting time preference had no significant effect on their exam performance, sense of belonging, or satisfaction, and the non-significant effects were homogeneous for both gender groups. We also found that having a consistent team had significantly positive effects on exam performance and sense of belonging, but no significant effect for satisfaction. Moreover, the effect of team consistency on exam performance was significantly stronger for female students than male students. Our findings justified the option to give students meeting time flexibility since it did not hurt their learning experience and outcome, and encouraged exploring effective approaches to forming consistent teams that make students intrinsically want to work with them. The gender difference in effects of team consistency on exam performance aligned with previous literature and served as evidence to use collaborative learning in computing and STEM classrooms.

Structured Query Language (SQL) plays a pivotal role in the effective management of relational databases and is a key skill across domains that engage with database systems, including research, development, and business management. However, mastering SQL can be challenging. To comprehend the approaches employed by students when solving SQL problems and address the challenges they faced during the learning process, our study analyzes submissions from the Database Systems course at the University of Illinois Urbana-Champaign during the Fall 2022 semester. We extend prior research involving line chart visualizations that facilitate instructors in identifying struggling students and understanding their submission behaviors. Yet, we acknowledge the limitations of this approach in providing timely feedback and actionable insights due to the sheer volume of visualizations. To address this, we developed an innovative technique using global sequence alignment scores and regular expression algorithms to compress student submission sequences. Our approach reveals submission patterns and pattern elements, leading to recommendations for instructors to enhance database education. By integrating student performance data, such as the number of submission attempts on a particular SQL problem and whether the student arrived at a correct final solution query, we aim to empirically support these recommendations, thereby enabling instructors to more accurately differentiate between struggling and excelling students.

Structured Query Language (SQL) is an essential skill to acquire for those who interact with databases, such as researchers, developers, and people involved in businesses. However, the challenges that these users face while learning SQL requires further research. In particular, the types of errors that students encounter on various assignment types or under exam conditions are an area that we are interested in to determine an optimal arrangement of coursework materials for improved learning. In this paper, we analyze 156,513 student SQL submissions to homework assignments, collaborative assignments, and exams of the Database Systems course available to 730 upper-level undergraduate and graduate students offered in the Fall 2022 semester at the University of Illinois Urbana-Champaign. We look at the ratio of syntax and semantic errors, and correct submissions for each of these assignment problem types as well as the most frequent syntax error codes. We visualize our data findings and draw recommendations for future coursework arrangements from the comparisons between the assignment types for a more effective acquisition of SQL as a skill. We found that although students most commonly encountered syntax error codes 1064 and 1054 regardless of the assignment type, they made more syntax errors (and fewer semantic errors) on exam problems compared with homework and collaborative assignment problems. We recommend instructors place a higher emphasis on non-timed SQL programming problems, targeted syntax drills during instruction, and syntax support during exams.

Training the growing number of Computer Science Education (CSEd) PhD students is a pressing concern for our community. To meet the needs of our CSEd PhD students at University of Illinois Urbana-Champaign, we have developed a new course designed to strengthen students’ foundation in relevant fields. Through a collaborative process, we developed a reading list that covers the educational theory and perspectives that most inform our own work, as well as concepts that prepare our graduates to engage with the broader CSEd community.

No abstract available.

We present a psychometric evaluation of the Cybersecurity Curriculum Assessment (CCA), completed by 193 students from seven colleges and universities. The CCA builds on our prior work developing and validating a Cybersecurity Concept Inventory (CCI), which measures students' conceptual understanding of cybersecurity after a first course in the area. The CCA deepens the conceptual complexity and technical depth expectations, assessing conceptual knowledge of students who had completed multiple courses in cybersecurity. We review our development of the CCA and present our evaluation of the instrument using Classical Test Theory and Item-Response Theory. The CCA is a difficult assessment, providing reliable measurements of student knowledge and deeper information about high-performing students.

Proof Blocks is a software tool that provides students with a scaffolded proof-writing experience, allowing them to drag and drop prewritten proof lines into the correct order instead of starting from scratch. In this paper we describe a randomized controlled trial designed to measure the learning gains of using Proof Blocks for students learning proof by induction. The study participants were 332 students recruited after completing the first month of their discrete mathematics course. Students in the study took a pretest and read lecture notes on proof by induction, completed a brief (less than 1 hour) learning activity, and then returned one week later to complete the posttest. Depending on the experimental condition that each student was assigned to, they either completed only Proof Blocks problems, completed some Proof Blocks problems and some written proofs, or completed only written proofs for their learning activity. We find that students in the early phases of learning about proof by induction are able to learn just as much from reading lecture notes and using Proof Blocks as by reading lecture notes and writing proofs from scratch, but in far less time on task. This finding complements previous findings that Proof Blocks are useful exam questions and are viewed positively by students.

This Research Full Paper presents an observation protocol to explore group processing in cooperative learning. Use of structured roles, such as Process Oriented Guided Inquiry Learning (POGIL) and pair programming, can help facilitate cooperative learning and help courses scale to large classroom sizes, decrease attrition and failure rates, and improve student performance. Our observation protocol was created to capture how groups work together in online, POGIL-inspired activities. A team of graduate student researchers developed the observation protocol for a variety of courses by observing three different computer science courses during the Spring 2021 semester. A total of 77 groups across all three courses were recorded, and percent agreement using a subset of the recordings suggested good interrater reliability (91.29%). We also extend a previous equality metric to quantify the rates of student participation, and found that it offered good differentiation between groups where one student contributed the most and groups where students contributed equally. We present example applications of our observation protocol related to general participation trends, the kinds of contributions students make, student-student bonding, and help-seeking patterns. Finally, we discuss future directions for use of our coding scheme as well as implications for implementing structured role-based cooperative learning online in the future.

In this software tool paper we present Proof Blocks, a tool which enables students to construct mathematical proofs by dragging and dropping prewritten proof lines into the correct order. We present both implementation details of the tool, as well as a rich reflection on our experiences using the tool in courses with hundreds of students. Proof Blocks problems can be graded completely automatically, enabling students to receive rapid feedback. When writing a problem, the instructor specifies the dependency graph of the lines of the proof, so that any correct arrangement of the lines can receive full credit. This innovation can improve assessment tools by increasing the types of questions we can ask students about proofs, and can give greater access to proof knowledge by increasing the amount that students can learn on their own with the help of a computer.

We present a psychometric evaluation of a revised version of theCybersecurity Concept Inventory (CCI), completed by 354 students from 29 colleges and universities. The CCI is a conceptual test of understanding created to enable research on instruction quality in cybersecurity education. This work extends previous expert review and small-scale pilot testing of the CCI. Results show that the CCI aligns with a curriculum many instructors expect from an introductory cybersecurity course, and that it is a valid and reliable tool for assessing what conceptual cybersecurity knowledge students learned.

This full research paper explores students' attitudes toward second-chance testing and how second-chance testing influences students' behavior. Second-chance testing refers to giving students the opportunity to take a second instance of each exam for some sort of grade replacement. Previous work has demonstrated that second-chance testing can lead to improved student outcomes in courses, but how to best structure second-chance testing to maximize its benefits remains an open question. We complement previous work by interviewing a diverse group of 23 students that have taken courses that use second-chance testing. From the interviews, we sought to gain insight into students' views and use of second-chance testing. We found that second-chance testing was almost universally viewed positively by the students and was frequently cited as helping to reduce test takers' anxiety and boost their confidence. Overall, we find that the majority of students prepare for second-chance exams in desirable ways, but we also note ways in which second-chance testing can potentially lead to undesirable behaviors including procrastination, overreliance on memorization, and attempts to game the system. We identified emergent themes pertaining to various facets of second-chance test-taking, including: 1) concerns about the time commitment required for second-chance exams; 2) a belief that second-chance exams promoted fairness; and 3) how second-chance testing incentivized learning. This paper will provide instructors and other stakeholders with detailed insights into students' behavior regarding second-chance testing, enabling instructors to develop better policies and avoid unintended consequences.

As part of teaching SQL, instructors often rely on auto-grading systems for marking students' assignments. However, such systems lack essential insights into the approaches students use to solve these assignments, allowing subtle flaws in student intuition to go unseen. Further, manual analysis of students' code submissions ranges from costly to impossible, depending on the assessments' frequency. In this paper, we present a system capable of extracting features that instructors deem significant from students' SQL queries and using them to generate clusters that capture the key approaches taken. To supplement this, we project the extracted information to an interactive dashboard and demonstrate its usefulness in allowing database systems professors and teaching staff to quickly identify trends in students' solutions.

Proof Blocks is a novel software tool which enables students to write mathematical proofs by dragging and dropping prewritten lines into the correct order, rather than writing a proof completely from scratch. We used Proof Blocks problems as exam questions for a discrete mathematics course with hundreds of students, allowing us to collect thousands of student responses to Proof Blocks problems. Using this data, we provide statistical evidence that Proof Blocks are easier than written proofs, which are typically very difficult. We also show that Proof Blocks problems provide about as much information about student knowledge as written proofs. Survey results show that students believe that the Proof Blocks user interface is easy to use, and that the questions accurately represent their ability to write proofs.

Structured Query Language (SQL), the standard language for relational database management systems, is an essential skill for software developers, data scientists, and professionals who need to interact with databases. SQL is highly structured and presents diverse ways for learners to acquire this skill. However, despite the significance of SQL to other related fields, little research has been done to understand how students learn SQL as they work on homework assignments. In this paper, we analyze students' SQL submissions to homework problems of the Database Systems course offered at the University of Illinois at Urbana-Champaign. For each student, we compute the Levenshtein Edit Distances between every submission and their final submission to understand how students reached their final solution and how they overcame any obstacles in their learning process. Our system visualizes the edit distances between students' submissions to a SQL problem, enabling instructors to identify interesting learning patterns and approaches. These findings will help instructors target their instruction in difficult SQL areas for the future and help students learn SQL more effectively.

We construct and evaluate a modular assessment for students' knowledge about CPU cache memories. Caches play a key role in improving performance in modern computing. They are difficult for students to learn, but we have little conceptual or empirical evidence about why. Building on prior frameworks, we propose six underlying knowledge components that we believe students need to robustly evaluate how a cache can affect the performance of code on a processor. We constructed a modular assessment using these components that can be used as a diagnostic instrument to find the concepts students are struggling to understand. Because different institutions teach caches at varying depths of detail, individual modules of the assessment can be used by instructors and researchers as appropriate for their context. We evaluated the assessment using a combination of Classical Test Theory, Exploratory Factor Analysis, and Confirmatory Factor Analysis. Our results suggest that the assessment is reliable and can be used modularly to assess various components of students' knowledge about caches, though future work needs to be done to evaluate the validity of these modules at different institutions. This assessment can help instructors and researchers design more precisely targeted instructional interventions to help students learn caches. The creation of similar modular assessments may help us in improving instruction in other difficult topics in computing.

This Innovative Practice Full Paper presents a framework for generating computer-based exams for complex engineering systems (such as cache memories) that can be machine graded while still offering partial credit for students. Complex multi-faceted engineering systems often require long, multi-part problems to fully assess students' understanding of those systems. Cache memories represent one such system in computer architecture courses. Traditionally, we assessed students' understanding of caches using comprehensive, multipart questions in a paper-based exam. Grading these exams was time-consuming and relied on subjective grading. To cope with rising enrollment, we sought to address these issues by developing machine administered and gradable exams that did not heavily rely on multiple-choice questions or exact numerical responses. Additionally, this system needed to provide partial credit, a common expectation of our students. We developed a cache simulator to use as a back-end for our questions. We used the simulator to develop exam questions and new homework assignments to help students practice cache memory concepts. To give students access to fair partial credit, we allowed multiple submissions for the exam questions with limited feedback. We also awarded partial credit for answers within certain tolerance of the correct answer. The partial credit awarded reduced as deviation from the correct answer increased. Consequently, students could correct minor mistakes or propagating errors which are common reasons for awarding partial credit. To evaluate the effect of the switch from paper-based to computerized exam, we ported questions from one of our paper-based exams to a computerized exam. We evaluated the differences in student performance on paper-based version and the computerized version of the questions and found mixed results with students performing comparably or better than the paper-based exam on the computer-based exam. We also surveyed students about their experience with the computer-based exam. Students overwhelmingly indicated a preference for the computer-based exam. We believe that ideas from our work can be used to automate generation, administration, and grading of complex multi-part questions in engineering disciplines beyond computer architecture.

We explore how course policies affect students' studying and learning when a second-chance exam is offered. High-stakes, one-off exams remain a de facto standard for assessing student knowledge in STEM, despite compelling evidence that other assessment paradigms such as mastery learning can improve student learning. Unfortunately, mastery learning can be costly to implement. We explore the use of optional second-chance testing to sustainably reap the benefits of mastery-based learning at scale. Prior work has shown that course policies affect students' studying and learning but have not compared these effects within the same course context. We conducted a quasi-experimental study in a single course to compare the effect of two grading policies for second-chance exams and the effect of increasing the size of the range of dates for students taking asynchronous exams. The first grading policy, called 90-cap, allowed students to optionally take a second-chance exam that would fully replace their score on a first-chance exam except the second-chance exam would be capped at 90% credit. The second grading policy, called 90-10, combined students' first- and second-chance exam scores as a weighted average (90% max score + 10% min score). The 90-10 policy significantly increased the likelihood that marginally competent students would take the second-chance exam. Further, our data suggests that students learned more under the 90-10 policy, providing improved student learning outcomes at no cost to the instructor. Most students took exams on the last day an exam was available, regardless of how many days the exam was available.

We analyze the submissions of 286 students as they solved Structured Query Language (SQL) homework assignments for an upper-level databases course. Databases and the ability to query them are becoming increasingly essential for not only computer scientists but also business professionals, scientists, and anyone who needs to make data-driven decisions. Despite the increasing importance of SQL and databases, little research has documented student difficulties in learning SQL. We replicate and extend prior studies of students' difficulties with learning SQL. Students worked on and submitted their homework through an online learning management system with support for autograding of code. Students received immediate feedback on the correctness of their solutions and had approximately a week to finish writing eight to ten queries. We categorized student submissions by the type of error, or lack thereof, that students made, and whether the student was eventually able to construct a correct query. Like prior work, we find that the majority of student mistakes are syntax errors. In contrast with the conclusions of prior work, we find that some students are never able to resolve these syntax errors to create valid queries. Additionally, we find that students struggle the most when they need to write SQL queries related to GROUP BY and correlated subqueries. We suggest implications for instruction and future research.

While substantial research has demonstrated that active learning pedagogies are better for learning than passive lectures, we need to understand the trade-offs between different active learning pedagogies. Computer Architecture at Midwestern University has historically been taught using active lectures, introducing content with a few clicker questions. In Fall 2018 (N = 363), short video lectures were made available to students as a supplemental resource. In Spring 2019, the instructor flipped the course, requiring students to watch the video lectures and complete an assignment before attending class. Two versions of the course were taught concurrently, using the same homework assignments, machine problems, and examinations but with different in-class pedagogies. Version SP19PI (N = 179) was taught using peer instruction. Version SP19CP (N = 73) was taught using collaborative problem solving, organizing students into teams of 3 to work on problems. Students completed surveys that measured their perceptions of time spent on the course, course difficulty, perceptions of stress, and sense of belonging. We compare students' performance on midterm exams and their non-cognitive outcomes to examine the relative effects of these different active learning pedagogies. We find that both flipped offerings (peer instruction and collaborative problem solving) benefited students beyond active lectures. Peer instruction (SP19PI) made learning more efficient. Collaborative instruction (SP19CP) provided greater social support for learning and eliminated gender grade disparities.

For the past five years we have been operating a Computer-Based Testing Facility (CBTF) as the primary means of summative assessment in large-enrollment STEM-oriented classes. In each of the last three semesters, it has proctored over 50,000 exams for over 6,000 unique students in 25–30 classes. Our CBTF has simultaneously improved the quality of assessment, allowed the testing of computational skills, and reduced the recurring burden of performing assessment in a broad collection of STEM-oriented classes, but it does require an up-front investment to develop the digital exam content. We have found our CBTF to be secure, cost-effective, and well liked by our faculty, who choose to use it semester after semester. We believe that there are many institutions that would similarly benefit from having a Computer-Based Testing Facility.

In this research category full paper, we present our findings on the effects of different course policies for second-chance testing on students' studying and exam taking behavior. Second-chance testing, where students are allowed to take a second instance of an exam for some form of grade replacement, is a less expensive approximation of mastery-based learning that can be easily integrated into a broad range of college course structures. It encourages students to review course material after poor performance on an examination but limits the amount of resources instructors must invest in the creation of examinations or in grading them. There exists, however, a large space of potential course policies for integrating second-chance testing into a course and little prior research on how these policies affect student behavior. This paper analyzes three different grading policies in use at Midwestern University. All of the policies attempt to encourage students to prepare adequately for the first-chance exam and review the material again before the second-chance exam, if they elect to take it. The first policy used a partial grade replacement policy with insurance: students' grades could only improve by taking the second-chance exam but the first-chance exam always counts for at least one-third of a students' grade on the examination. The second policy is identical, but required students to complete a zero-credit, online-homework assignment before being allowed to take the second-chance exam. The third policy implemented full grade replacement (even if the second score is lower) and capped the score that could be achieved on the second exam. By comparing these different course policies, we show that grading policies have a significant effect on whether students take second-chance exams. We also performed a quasi-experimental study, adding second-chance exams to a course. We present data from students' exam performance and from the course's learning management system that suggest that adding a second-chance exam had no effect on student performance or study habits for the first-chance exam. However, the total amount of time that students studied did increase substantially as students who took the second-chance exam studied an additional 60% of their original effort.

We analyze expert review and student performance data to evaluate the validity of the Cybersecurity Concept Inventory (CCI) for assessing student knowledge of core cybersecurity concepts after a first course on the topic. A panel of 12 experts in cybersecurity reviewed the CCI, and 142 students from six different institutions took the CCI as a pilot test. The panel reviewed each item of the CCI and the overwhelming majority rated every item as measuring appropriate cybersecurity knowledge. We administered the CCI to students taking a first cybersecurity course either online or proctored by the course instructor. We applied classical test theory to evaluate the quality of the CCI. This evaluation showed that the CCI is sufficiently reliable for measuring student knowledge of cybersecurity and that the CCI may be too difficult as a whole. We describe the results of the expert review and the pilot test and provide recommendations for the continued improvement of the CCI.

While most efforts to improve students' learning in computer science education have focused on designing new pedagogies or tools, comparatively little research has focused on redesigning examinations to improve students' learning. Cognitive science research, however, has robustly demonstrated that getting students to practice using their knowledge in testing environments can significantly improve learning through a phenomenon known as the testing effect. The testing effect has been shown to improve learning more than rehearsal strategies such as re-reading a textbook or re-watching lectures. In this paper, we present a quasi-experimental study to examine the effect of using frequent, automated examinations in an advanced computer science course, "Programming Languages and Compilers" (CS 421). In Fall 2014, students were given traditional paper-based exams, but in Fall 2015 a computer-based testing facility enabled the course to offer more frequent examinations while other aspects of the course were held constant. A comparison of 292 student scores across the two semesters revealed a significant change in the distribution of students' grades with fewer students failing the final examination, and proportionately more students now earning grades of B and C instead. This data suggests that focusing on redesigning the nature of examinations may indeed be a relatively untapped opportunity to improve students' learning.

We present a methodological improvement for calculating Grade Point Averages (GPAs). Heterogeneity in grading between courses systematically biases observed GPAs for individual students: the GPA observed depends on course selection. We show how a logistic model can account for course selection by simulating how every student in a sample would perform if they took all available courses, giving a new “modeled GPA.” We then use 10 years of grade data from a large university to demonstrate that this modeled GPA is a more accurate predictor of student performance in individual courses than the observed GPA. Using Computer Science (CS) as an example learning analytics application, it is found that required CS courses give significantly lower grades than average courses. This depresses the recorded GPAs of CS majors: modeled GPAs are 0.25 points higher than those that are observed. The modeled GPA also correlates much more closely with standardized test scores than the observed GPA: the correlation with Math ACT is 0.37 for the modeled GPA and is 0.20 for the observed GPA. This implies that standardized test scores are much better predictors of student performance than might otherwise be assumed.

Mathematics courses are a major source of engineering student attrition. Many engineering students drop out before even taking an engineering course. While the mismatch between mathematics content and following engineering coursework is already a concern, it is often studied heuristically by talking to engineering faculty. Few rigorous studies empirically document when students actually need to use calculus knowledge in their coursework. We mapped how foundational calculus concepts are used to solve assigned problems in topics covered in the widely required Engineering Statics course. To create this mapping, we used the mathematics-in-use technique of Czocher et al. We present preliminary evidence of the match, or mismatch, between what calculus is taught and learned in mathematics courses and what is expected of students in following engineering coursework. For this work-in-progress, we focused on presenting the alignment between calculus concepts/skills and topics taught in Statics.

While the concept of state is foundational to computing, students possess a myriad of misconceptions about it and the role it plays within computing systems. Research on students' misconceptions reveals that their ability to use conceptually appropriate information varies based on the task they are performing and the representational tools they are provided. Critically, the tacit information in these representations influences this process, hindering or helping students. In this paper, we present a qualitative research study, in which we interviewed 24 students as they transformed finite state machines into synchronous, sequential logic circuits. We found that students generally had profound skill with procedures. However, their ability to reason about the four components of state, next-state, inputs, and outputs, were constrained by the representations that they were given or created themselves. Conversely, the order in which students produced their drawings provided complementary insights into students conceptual understanding. These findings revealed that students possess conceptions of computers as input-output systems rather than state-based systems. We suggest potential interventions and future research based on these findings.

Sketches and other forms of graphical communication are central to both the practice and learning of engineering. Visual representations play a critical role in helping students learn engineering concepts, socialize them into the engineering discipline, and facilitate or hinder the design process. Despite the importance of graphical communication and visual representations, our understanding of how students learn these representations and use them is poor. This paper describes part of a larger expert-novice comparison study to document how sketches moderate engineering problem solving. We present preliminary findings on how digital logic instructors use sketches to solve canonical problems.

Creating systemic change in undergraduate engineering and STEM education is difficult to achieve and just as difficult to study. It has been proposed that organizational learning and change theories can be coupled with social network analysis to achieve both of these goals. In this paper, we describe an institutional change effort designed around principles from Communities of Practice. We then present the design of a social analysis network study that we are executing to study and analyze whether this change effort has been successful in achieving its goals. We present some preliminary data to demonstrate the promise of this approach for executing and studying institutional change in engineering education and STEM education more broadly.

Engaging underrepresented populations of women and minorities in engineering represents our greatest untapped resource for increasing the STEM workforce and its productivity. Guided by the research that shows that students' performance and experience in a first course in the discipline is predictive of retention of these students, we are adopting Merit workshops into our Introduction to Electronics course (one of our department's first-year courses). Merit workshops engage students in collaborative learning communities to improve students' sense of belonging in the department. In this work in progress paper, we will describe how we implemented our Merit workshops and provide some preliminary evaluations of the workshop.

Despite the documented need to train and educate more cybersecurity professionals, we have little rigorous evidence to inform educators on effective ways to engage, educate, or retain cybersecurity students. To begin addressing this gap in our knowledge, we are conducting a series of think-aloud interviews with cybersecurity students to study how students reason about core cybersecurity concepts. We have recruited these students from three diverse institutions: University of Maryland, Baltimore County, Prince George's Community College, and Bowie State University. During these interviews, students grapple with security scenarios designed to probe student understanding of cybersecurity, especially adversarial thinking. We are analyzing student statements using a structured qualitative method, novice-led paired thematic analysis, to document student misconceptions and problematic reasonings. We intend to use these findings to develop Cybersecurity Assessment Tools that can help us assess the effectiveness of pedagogies. These findings can also inform the development of curricula, learning exercises, and other educational materials and policies.

We present a study that explores the grade-point average (GPA) penalties that students face when taking introductory STEM courses. Previous work has found that there is a large and significant grade point penalty for women and minorities in most STEM classes (that is, these students perform worse in these classes than their overall GPA would suggest). We recreated this work using a new, large data set (63,012 students over 10 years) of student performance, and found that the initial results held when using the original approach. We argue that there are methodological shortcomings to the original approach, however, as there is no attempt to control for individual student program difficulty (STEM majors and non-STEM majors share some classes, but have very different overall suites of courses that determine their overall GPA). As the female/male and racial ratios vary across majors it is therefore likely that a division by gender is not comparing equivalent sample populations. By controlling for student test scores or major most of the penalty is removed. The initial findings of large GPA penalties in STEM courses appears to be an example of “Simpson's Paradox”.

Many engineering courses are transitioning from traditional paper textbooks to online and multimedia instructional modules to present content to students outside of class time. As the use of these online resources expands, research about the effective use and production of these resources should grow in tandem. We study the effect of three different educational interventions: expandable worked examples, assertion headings, and hand-drawn figures on students' learning and affective responses in online instructional texts for an introductory electrical engineering course. Although measures of students' performance on technical content showed few significant changes, affective measures of student course satisfaction with the materials had improved.

The adoption of research-based instruction strategies (RBIS) by faculty is generally perceived as being slow. Faculty resist change, reject data that demonstrates the effectiveness of RBIS, and prioritize research over teaching. Even when faculty attempt to use RBIS, they often do so with poor fidelity to the original design of the RBIS. By organizing faculty into communities of practice, we are observing a sudden surge in the adoption of RBIS across STEM departments. Through this work in progress we present preliminary data that we have collected to understand exactly why many faculty are now adopting RBIS. While this effort has led to the adoption of many RBIS, we focus on the adoption of personal response systems (clickers) and Peer Instruction. Preliminary data suggests that faculty adoption is being driven by community and collaborative instruction.

STEM educators have devoted increasing attention to discrete mathematics in recent years due, in part, to its strong connections with subjects like computer science, probability and statistics, and business management. Combinatorics problems, in particular, while useful for modeling concrete situations, are often considered to be tricky for students. To develop a better understanding of students' conceptions regarding problems involving permutations and combinations, a secondary data analysis using a grounded theory approach was performed on transcripts of student interviews obtained during an earlier study. Participants had recently completed a college-level discrete mathematics course with a passing grade. Analysis focused on answering two research questions: 1) What patterns of responses do students generate while producing solutions to combinatorics word problems? 2) What underlying conceptual ideas lead to these patterns?

A hybrid project and service based learning course has been introduced in the Department of Civil and Environmental Engineering (CEE) at the University of Illinois. The primary objectives of the course are to develop engineering problem solving, professional, and business skills earlier in our CEE undergraduate curriculum by having student teams identify open-ended, ill-defined campus/community problems or opportunities, develop a feasible project scope, and propose sustainable solutions. The unique features of our course include (1) a blend of service and project learning with faculty instructors partnering with the campus engineering staff to assist with team project mentoring, campus data collection, local field trips, and case studies, (2) formal course assessments through pre and post-class survey and student focus group interviews, and (3) weekly instructor meetings that consist of faculty, teaching assistants, department administrators and engineering staff updating the course during the semester and planning major changes for the next course offering. In this paper, we describe the course organization and its curricular evolution along with evaluation data from student surveys and focus groups as well as the impact of routine instructor community of practice meetings.

The purpose of this proposed mini-workshop is to connect our research in conceptual change [Collaborative research: Getting Engineers to Talk Across Disciplines, EEC 1129447] to instructional practice. Our research suggests that one potential indicator of students' conceptual understanding across several engineering disciplines is their use (or misuse) of language. During this mini-workshop participants will be able to read excerpts from interviews we conducted, discuss how to draw inferences about student learning from these interviews, and then brainstorm how they can be more explicit about the use of language in their classrooms.

This session is a chance for researchers studying concept inventories (CIs)--low-cost assessments highlighting student misconceptions in a field--and CS education practitioners to communicate about advances in concept inventories in an engaging and utterly ridiculous way.

Recent large-scale data breaches such as the credit card scandals of Target and Home Depot have significantly raised the public awareness of the importance of the security of their data and personal information. These incidents highlight a growing need and urgency to develop the cybersecurity infrastructure of our country and in the world. The development of ACM's Computer Science Curriculum 2013 and the National Initiative for Cybersecurity Education framework further highlight the growing importance of cybersecurity in computing education. Critically, recent studies predict that there will be a significant demand for cybersecurity professionals in the coming years, yet there is a lack of rigorous evidence-based infrastructure to advise educators on how best to engage, inform, educate, nurture, and retain cybersecurity students and how best to structure cybersecurity curricula to prepare new professionals for careers in this field. The development of validated assessment tools of student learning provide one means for increasing the rigor with which we make pedagogical and curricular decisions. During this Birds of a Feather session, participants will engage in a structured dialogue to identify what assessment tools are needed to improve cybersecurity education. Further, participants will provide feedback on initial efforts to identify a core set of concepts and skills that will be essential for students' success in cybersecurity fields.

Online courses have the promise of extending the horizons of today's academic landscape with their cost-effective and convenient model compared to traditional learning environments. Despite the promising nature of the learning model, there continue to be several challenges that hinder learning, one of which is lack of instructor presence. This study aims at understanding the effect of instructor presence on student satisfaction in an online setting of a course in engineering. We conducted a student-centered pilot experiment to assess engineering students' perceptions of two modes of online minilectures: the first, a presentation with the instructor appearing in window, created using an off-the-shelf screen-capture software; the second, a presentation with the instructor overlaid in the slides created using recent visual communication technology that overlays the video of the instructor without any background images or outline boxes. The instructor was the same in both the presentations. Our focus here is on the following factors: 1. Comparing overall student satisfaction after watching the two modes; 2. Comparing the perceived non-verbal immediacy factors of the instructor; and, 3. Comparing the preference of video mode for future online courses. Preliminary results suggest a preference of the video mode with the instructor overlaid over that with the instructor in a box. The effect sizes of the differences in overall satisfaction between the experimental groups and their perceived levels of non-verbal immediacy factors when viewing the online lecture in the two modes are encouraging enough to pursue more longitudinal studies with the set-up.

The low-cost intrinsic motivation (IM) course conversion project is an effort to create a new system of course design that focuses on creating scalable and sustainable courses that emphasize promoting students' IM to learn. Unlike many course design methods such as idea-based learning, project- or problem-driven learning, or “flipped” classrooms, which first ask, “How do we help students learn X better,” we ask “how do we foster intrinsically-motivated learners who want to learn X?” While this course design method still uses theories of cognition to design course structures, it uses motivational constructs such as purpose, autonomy, relatedness, and competence as the primary design considerations of a course. Secondarily, the course design method considers and documents the financial, time, political, and psychological costs of course design. In this paper, we present a preliminary attempt to formalize this IM-driven course design method as well as a system for evaluating the short- and long-term costs of implementing a specific course design.

The low-cost intrinsic motivation (IM) course conversion project aims to promote the adoption of education innovations, lowering the costs of these innovations by promoting students' intrinsic motivation to learn and to invest in their own education. We have piloted and scaled the IM course conversion of a large enrollment, sophomore-level engineering course. As part of a broader evaluation, we interviewed 31 students to better understand how the IM course conversion affected students' motivations to learn. During these interviews, students described their experiences in the course as a story from the beginning to the end of the semester. Interviews were transcribed and analyzed with an open coding scheme focused on motivational and affective statements. Preliminary analysis indicates that strategic choices, positive team dynamics, and productive failures in the learning process all contribute to promoting students' intrinsic motivation to learn in both pilot and full scale IM course conversions.

Conceptual change is often approached with a concrete epistemology in which students must replace their misconceptions with correct conceptions. Drawing upon the literature and our research, we propose that we need a new, discursive epistemology that describes conceptual change as language acquisition.

In an effort to create education reform by promoting students' intrinsic motivation (EVI) to learn, we are developing a program to train IM-supportive teaching assistants (TAs). We present our preliminary analysis of the first two TAs' reflective teaching journals and discuss what IM-supportive instruction looks like and how we can improve our training program.

The study of students' preconceptions and how they affect their learning in science, technology, engineering and mathematics (STEM) fields is of nationally recognized importance. There are, however, various and contradictory theoretical approaches to conceptual change, and none of them have been rigorously applied in the context of engineering education. This paper is part of a larger study drawing on existing sets of data from a wide range of engineering content areas to develop a theoretical explanation of conceptual change in engineering education. In the work reported here we re-analyze students' understanding of concepts about axially loaded members (from mechanics of materials) and Boolean logic (from digital logic). Previously published analyses of these data argue that the context of a problem or question effects students' reasoning about that concept. These contexts can range from the presence or absence of figures or diagrams to the social contexts of the problem. We explored three potential theoretical explanations for the context-sensitivity of student reasoning: (1) a perceptual cues theory, (2) a domain specificity theory, and (3) a language-based theory. It is argued that these competing theoretical explanations do not contradict each other as much as they overlap, and potentially productive syntheses of the theories are proposed as directions for future work.

The ability to reason with formal logic is a foundational skill for computer scientists and computer engineers that scaffolds the abilities to design, debug, and optimize. By interviewing students about their understanding of propositional logic and their ability to translate from English specifications to Boolean expressions, we characterized common misconceptions and novice problem-solving processes of students who had recently completed a digital logic design class. We present these results and discuss their implications for instruction and the development of pedagogical assessment tools known as concept inventories.

Cognitive research in digital logic has shown that students struggle to organize their knowledge of concepts and tools into meaningful categories that facilitate problem solving or design. When students learn about digital logic design tools, they must practice using these tools independently and often come to think of these tools as “problems-unto-themselves,” rather than as tools that can be used to solve more complicated problems. To help students properly organize their knowledge of digital logic design tools, we propose the creation of a digital logic calculator that helps students learn the purposes and functionality of the different digital logic design tools.

A concept inventory (CI) is a standardized assessment tool that evaluates how well a student's conceptual framework matches the accepted conceptual framework of a discipline. In this paper, we present our process in creating and evaluating the alpha version of a CI to assess student understanding of digital logic. We have checked the validity and reliability of the CI through an alpha administration, follow-up interviews with students, analysis of administration results, and expert feedback. So far the feedback on the digital logic concept inventory is positive and promising.

Computing educators are often baffled by the misconceptions that their CS1 students hold. We need to understand these misconceptions more clearly in order to help students form correct conceptions. This paper describes one stage in the development of a concept inventory for Computing Fundamentals: investigation of student misconceptions in a series of core CS1 topics previously identified as both important and difficult. Formal interviews with students revealed four distinct themes, each containing many interesting misconceptions. Three of those misconceptions are detailed in this paper: two misconceptions about memory models, and data assignment when primitives are declared. Individual misconceptions are related, but vary widely, thus providing excellent material to use in the development of the CI. In addition, CS1 instructors are provided immediate usable material for helping their students understand some difficult introductory concepts. Copyright 2010 ACM.

A concept inventory is a standardized assessment tool intended to evaluate a student’s understanding of the core concepts of a topic. In order to create a concept inventory it is necessary to accurately identify these core concepts. A Delphi process is a structured multi-step process that uses a group of experts to achieve a consensus opinion. We present the results of three Delphi processes to identify topics that are important and difficult in each of three introductory computing subjects: discrete mathematics, programming fundamentals, and logic design. The topic rankings can not only be used to guide the coverage of concept inventories, but can also be used by instructors to identify what topics merit special attention.

The ability to reason with formal logic is a foundational skill for computer scientists and computer engineers that scaffolds the abilities to design, debug, and optimize. By interviewing students about their understanding of propositional logic and their ability to translate from English specifications to Boolean expressions, we characterized common misconceptions and novice problem-solving processes of students who had recently completed a digital logic design class. We present these results and discuss their implications for instruction and the development of pedagogical assessment tools known as concept inventories.

A Delphi process is a structured multi-step process that uses a group of experts to achieve a consensus opinion. We present the results of three Delphi processes to identify topics that are important and difficult in each of three introductory computing subjects: discrete math, programming fundamentals, and logic design. The topic rankings can be used to guide both the coverage of standardized tests of student learning (i.e., concept inventories) and can be used by instructors to identify what topics merit emphasis.