MeetingMayhem is a web-based educational game focused on adversarial thinking in the context of network security. In particular, this game gently and non-technically introduces students without prior cybersecurity knowledge to the Dolev-Yao network intruder model. In MeetingMayhem, three students take on the roles of two agents and an adversary. Two agents need to agree on a time and location to exchange an essential asset. The agents communicate through a network controlled by the adversary without knowing the adversary's identity. The adversary can insert, block, or modify messages. Students can play different roles, try different strategies, interact with different players, and send different messages. MeetingMayhem is available as a Docker Image, with open-source code. MeetingMayhem provides an engaging and innovative way for novice students to learn about adversarial thinking and cryptography within 2 hours.

We report on our experiences fielding MeetingMayhem, an interactive game that we developed, which introduces students to fundamental concepts in network security, cybersecurity, and adversarial thinking. The game is intended for students who do not necessarily have any prior background in computer science. Assuming the role of agents, two players exchange messages over a network to try to agree on a meeting time and location, while an adversary interferes with their plan. Following the Dolev-Yao model, the adversary has full control of the network: they can see all messages and modify, block, or forward them. We designed the game as a web application, where groups of three students play the game, taking turns being the adversary. The adversary is a legitimate communicant on the network, and the agents do not know who is the other agent and who is the adversary. Through gameplay, we expect students to be able to (1) identify the dangers of communicating through a computer network, (2) describe the capabilities of a Dolev-Yao adversary, and (3) apply three cryptographic primitives: symmetric encryption, asymmetric encryption, and digital signatures. We conducted surveys, focus groups, and interviews to evaluate the effectiveness of the game in achieving the learning objectives. The game helped students achieve the first two learning objectives, as well as using symmetric encryption. We found that students enjoyed playing Meeting Mayhem. We are revising MeetingMayhem to improve its user interface and to better support students to learn about asymmetric encryption and digital signatures.

We present a psychometric evaluation of the Cybersecurity Curriculum Assessment (CCA), completed by 193 students from seven colleges and universities. The CCA builds on our prior work developing and validating a Cybersecurity Concept Inventory (CCI), which measures students' conceptual understanding of cybersecurity after a first course in the area. The CCA deepens the conceptual complexity and technical depth expectations, assessing conceptual knowledge of students who had completed multiple courses in cybersecurity. We review our development of the CCA and present our evaluation of the instrument using Classical Test Theory and Item-Response Theory. The CCA is a difficult assessment, providing reliable measurements of student knowledge and deeper information about high-performing students.

We present a psychometric evaluation of a revised version of theCybersecurity Concept Inventory (CCI), completed by 354 students from 29 colleges and universities. The CCI is a conceptual test of understanding created to enable research on instruction quality in cybersecurity education. This work extends previous expert review and small-scale pilot testing of the CCI. Results show that the CCI aligns with a curriculum many instructors expect from an introductory cybersecurity course, and that it is a valid and reliable tool for assessing what conceptual cybersecurity knowledge students learned.

We analyze expert review and student performance data to evaluate the validity of the Cybersecurity Concept Inventory (CCI) for assessing student knowledge of core cybersecurity concepts after a first course on the topic. A panel of 12 experts in cybersecurity reviewed the CCI, and 142 students from six different institutions took the CCI as a pilot test. The panel reviewed each item of the CCI and the overwhelming majority rated every item as measuring appropriate cybersecurity knowledge. We administered the CCI to students taking a first cybersecurity course either online or proctored by the course instructor. We applied classical test theory to evaluate the quality of the CCI. This evaluation showed that the CCI is sufficiently reliable for measuring student knowledge of cybersecurity and that the CCI may be too difficult as a whole. We describe the results of the expert review and the pilot test and provide recommendations for the continued improvement of the CCI.

Despite the documented need to train and educate more cybersecurity professionals, we have little rigorous evidence to inform educators on effective ways to engage, educate, or retain cybersecurity students. To begin addressing this gap in our knowledge, we are conducting a series of think-aloud interviews with cybersecurity students to study how students reason about core cybersecurity concepts. We have recruited these students from three diverse institutions: University of Maryland, Baltimore County, Prince George's Community College, and Bowie State University. During these interviews, students grapple with security scenarios designed to probe student understanding of cybersecurity, especially adversarial thinking. We are analyzing student statements using a structured qualitative method, novice-led paired thematic analysis, to document student misconceptions and problematic reasonings. We intend to use these findings to develop Cybersecurity Assessment Tools that can help us assess the effectiveness of pedagogies. These findings can also inform the development of curricula, learning exercises, and other educational materials and policies.