This full research paper describes the analysis of common software vulnerabilities that are introduced by students enrolled in four-year computing and cybersecurity majors from two different higher education institutions in Georgia. As the demand for secure coding education continues to grow, pedagogical improvements need to be made in identifying key software vulnerabilities students commit during code development (from the first programming course to the exit senior design capstone) which in turn can be analyzed to inform the pedagogical interventions focused at preparing students with skill sets for writing secure code and entering the professional workforce. While code security is emphasized throughout the computing curriculum, this research is focused on training individuals to be aware of common vulnerabilities and tailoring programming concept knowledge that has been shown to have a positive effect on code security. Existing research has mainly focused on developing vulnerability analysis tools rather than collecting data (and subsequently analyzing) regarding the types of vulnerabilities produced by students at their institutions. In this paper, we analyzed student code across different courses and reported the types of vulnerabilities produced by students in their assignment submission code from two different higher education institutions across different levels of the four-year curriculum. The reported vulnerabilities are grouped by CWE-ID, which is a standard and common way to categorize and identify software vulnerabilities. The resulting CWE-IDs are then grouped per student submission and per semester (across curriculum levels) to discover the common types of software vulnerabilities committed across cross sections of students. Our results from the analysis of vulnerabilities (ranging from CS1 courses to capstone courses) are organized around the following research questions: 1) What are the most common software vulnerabilities produced by computing majors at different levels through the computing curriculum?; and 2) Do these vulnerabilities persist throughout their curriculum as they advance into higher-level courses? We report that students commonly make mistakes related to variable usage, null pointer checks, hard-coding sensitive information, and improperly validating input. Vulnerabilities such as CWE-489 (“ Active Debug Code”) and CWE-215 (“Insertion of Sensitive Information Into Debugging Code”) tend to persist across multiple course levels and may need to be focused in the computing curriculum. The number of vulnerabilities introduced in assignment code increases as course complexity increases. We also find that vulnerabilities produced by students have little overlap with what software vulnerability researchers commonly study, potentially leading to a mismatch in priority for secure coding topics. Our findings have implications for computer science and cybersecurity curriculum design and delivery.

This Research to Practice Full Paper presents the use of data collected by our Non-Intrusive Classroom Attention Tracking System (NiCATS) to evaluate student comprehension. Quantifying students' cognitive processes in classrooms in a non-intrusive way is challenging. By analyzing various aspects of the eye metrics against defined regions of interest (ROI), instructors can better understand students’ cognitive processes as they acquire new knowledge. Eye-tracking studies primarily define ROIs based on commonly used metrics (source code complexity, significant fixation durations, etc.). While helpful, these metrics, when used independently, do not accurately represent their comprehension patterns. This paper contributes an alternative, multilayered approach for calculating gaze metrics against automatically defined ROIs. The work utilizes the AI-based Non-Intrusive Classroom Attention Tracking System (NiCATS - developed by the researchers), collecting raw-gaze data in real-time as information is presented on a computer screen. This paper reports the results of a study in which undergraduate students in a CS programming course were asked to identify defects seeded in Java programs. Each JAVA program included its own unique sets of ROIS defined using two different granularities: lexer-based and line-based. The ROI sets were then used to calculate relevant eye metrics in the context of each ROI layout. The results of the eye metric analysis at specific ROIs w.r.t their code review task provide insights into the cognitive processes students undergo when trying to comprehend new material. Subdividing this region into lexer-based regions, we determined “content topics” students struggled with (e.g., using complex data types) in a specific area. This feedback is valuable to the instructor as it enables the ability to identify hard-to-comprehend content topics post-hoc and gives the ability to validate student learning in the classroom. While this experiment focused on students in introductory programming courses, we intend to conduct experiments in other learning settings where students are expected to read material on a computer screen or solve actual problems. To summarize, the analysis of these eye metrics using more fine-grained ROIs (lexer-based, line-based) as an extension of complexity-based ROIs provides instructors with deeper insights into the cognitive processes used by students when compared to the current state-of-the-art techniques.

This Research to Practice Full Paper presents our Non-intrusive Classroom Attention Tracking System (NiCATS) and discusses the data collected through it. Academic instructors and institutions desire the ability to accurately and autonomously measure students' attentiveness in the classroom. Generally, college departments use unreliable direct communication from students, observational sit-ins, and end-of-semester surveys to collect feedback regarding their courses. Each of these methods of collecting feedback is useful but does not provide automatic feedback regarding the pace and direction of lectures. It has been widely reported that attention levels during passive classroom lectures generally drop after about ten to thirty minutes and can be restored to normal levels with regular breaks, novel activities, mini-lectures, case studies, or videos. Tracking these “drops” in attention can be crucial for the accurate timing of these change-ups in activities. This allows for maximal attention and a greater amount of deeply learned material. Autonomously collected data can also be used either in real-time or post-hoc to alter the design and presentation of lectures. Keeping track of student attention is vital to having confidence in delivering material. Even if lectures do not break up presentation slides with attention-raising activities, they can still show more important information during periods of high attention and less important information during periods of low attention. This area of research has applications both in in-person classrooms and online learning environments. The long-term goals of this research can prove invaluable for large in-person classrooms or classrooms where students’ faces are obscured, such as behind computer monitors.

This Innovative Practice Full-Paper presents a system for real-time accurate detection of classroom attentiveness using monitor-mounted webcams and eye trackers. Academic institutions and instructors cannot accurately assess the moment-to-moment attentiveness of students in classrooms where students' faces are obscured by computer monitors. This can cause the lectures of Computer Science, Information Technology, or other lab-based courses to be incorrectly paced, which leads to students having overall poorer grasps of the subject material. We present a system for accurate detection of classroom attentiveness using monitor-mounted webcams and eye trackers. To determine correlations for the attentiveness judging system, we compare an initial attentiveness score produced by trained labelers using an image of the student's face with a series of calculated eye metrics to determine a final attentiveness score. Because the student webcam images and eye coordinates are synchronously collected with the lecture, this final attentiveness score is used to provide post-hoc feedback to instructors on the status of their students via time-series graphs displayed on the instructor's computer monitor. The proposed system is invaluable for institutions seeking to improve student education, instructors striving to improve the flow of lectures, and students seeking a more accommodating learning environment. The primary source of innovation from this system comes from the correlation of extracted eye metrics with the face images labeled for attentiveness. Research exists about determining attentiveness using a convolutional neural network trained on face images and even determining attentiveness by correlating face-image-trained outputs, each of which we plan to incorporate to make our system real-time in the future. This novel research could prove helpful for the field of education.