In the contemporary landscape of cybersecurity, the importance of effective penetration testing is underscored by NIS2, emphasizing the need to assess and demonstrate cyber resilience. This paper introduces an innovative approach to penetration testing that employs interactive checklists, supporting both manual and automated tests, as demonstrated within the Penterep environment. These checklists, functioning as a quantifiable measure of test completeness, guide pentesters through methodological testing, addressing the inherent challenges of the security testing domain. While some may perceive a limitation in the dependency on predefined checklists, the results from a presented case study underscore the criticality of methodological testing. The study reveals that relying solely on fully automated tools would be inadequate to identify all vulnerabilities and flaws without the inclusion of manual tests. Our innovative approach complements established methodologies, such as PTES, OWASP, and NIST, providing crucial support to penetration testers and ensuring a comprehensive testing process. Implemented within the Penterep environment, our approach is designed with deployment flexibility (both on-premises and cloud-based), setting it apart through an overview comparison with existing tools aligned with state-of-the-art penetration testing approaches. This flexible and scalable approach effectively bridges the gap between manual and automated testing, meeting the increasing demands for effectiveness and adaptability in penetration testing. • An innovative approach to penetration testing using adaptable interactive checklists. • Automation of testing to increase its effectiveness and reduce repetitive tasks. • Penetration testing environment design and implementation using high customizability. • Showcase using the provided case study, concluded with lessons learned.

This Research To Practice Full Paper presents an instructor guide and a tool to improve the creation of cybersecurity hands-on training with adaptive learning support. Adaptive learning uses students' performance and skills to assign suitable tasks to improve their learning experience. While it is well-established in many domains, it is rarely used in operating systems, networking, and cybersecurity. In this paper, we improve and present how to ease the creation and optimization process of adaptive hands-on training by instructors. To the best of our knowledge, this paper is one of the first works investigating the process of creating cybersecurity training with adaptive learning. The training uses metrics such as pre-training assessment and performance during the previous tasks in training to assign suitable tasks for each student. With the help of the developed tool, we demonstrate how metrics settings influence the students' transitions between training tasks. The instructors can easily visualize students' transitions throughout the training. This approach helps the instructors adapt the metrics to predict students' transitions between tasks for each training session. The results from performed simulations show that our tool might increase the efficiency of the adaptive training and students' experience even more. Using the experience from the simulations and past training sessions, we propose the design process for the whole creation of adaptive training. This design process is general enough to be adopted by other domains such as operating systems and networking that may use adaptive learning techniques for their hands-on assignments. We have released the tool and all the software components under an open-source license, so other instructors can freely use and adopt them.

Networking, operating systems, and cybersecurity skills are exercised best in an authentic environment. Students work with real systems and tools in a lab environment and complete assigned tasks. Since all students typically receive the same assignment, they can consult their approach and progress with an instructor, a tutoring system, or their peers. They may also search for information on the Internet. Having the same assignment for all students in class is standard practice efficient for learning and developing skills. However, it is prone to cheating when used in a summative assessment such as graded homework, a mid-term test, or a final exam. Students can easily share and submit correct answers without completing the assignment. In this paper, we discuss methods for automatic problem generation for hands-on tasks completed in a computer lab environment. Using this approach, each student receives personalized tasks. We developed software for generating and submitting these personalized tasks and conducted a case study. The software was used for creating and grading a homework assignment in an introductory security course enrolled by 207 students. The software revealed seven cases of suspicious submissions, which may constitute cheating. In addition, students and instructors welcomed the personalized assignments. Instructors commented that this approach scales well for large classes. Students rarely encountered issues while running their personalized lab environment. Finally, we have released the open-source software to enable other educators to use it in their courses and learning environments.

This Research To Practice Full Paper presents how learning experience influences students' capability to learn and their motivation for further learning. Although each student is different, standard instruction methods do not adapt to individual students. Adaptive learning reverses this practice and attempts to improve the student experience. While adaptive learning is well-established in programming, it is rarely used in cybersecurity education. This paper is one of the first works investigating adaptive learning in cybersecurity training. First, we analyze the performance of 95 students in 12 training sessions to understand the limitations of the current training practice. Less than half of the students (45 out of 95) completed the training without displaying any solution, and only in two sessions, all students completed all phases. Then, we simulate how students would proceed in one of the past training sessions if it would offer more paths of various difficulty. Based on this simulation, we propose a novel tutor model for adaptive training, which considers students' proficiency before and during an ongoing training session. The proficiency is assessed using a pre-training questionnaire and various in-training metrics. Finally, we conduct a case study with 24 students and new training using the proposed tutor model and adaptive training format. The results show that the adaptive training does not overwhelm students as the original static training format. In particular, adaptive training enables students to enter several alternative training phases with lower difficulty than the phases in the original training. The proposed adaptive format is not restricted to particular training used in our case study. Therefore, it can be applied to practicing any cybersecurity topic or even in other related computing fields, such as networking or operating systems. Our study indicates that adaptive learning is a promising approach for improving the student experience in cybersecurity education. We also highlight diverse implications for educational practice that improve students' experience.

This Innovative Practice full paper describes a technical innovation for scalable teaching of cybersecurity hands-on classes using interactive learning environments. Hands-on experience significantly improves the practical skills of learners. However, the preparation and delivery of hands-on classes usually do not scale. Teaching even small groups of students requires a substantial effort to prepare the class environment and practical assignments. Further issues are associated with teaching large classes, providing feedback, and analyzing learning gains. We present our research effort and practical experience in designing and using learning environments that scale up hands-on cybersecurity classes. The environments support virtual networks with full-fledged operating systems and devices that emulate realworld systems. The classes are organized as simultaneous training sessions with cybersecurity assignments and learners' assessment. For big classes, with the goal of developing learners' skills and providing formative assessment, we run the environment locally, either in a computer lab or at learners' own desktops or laptops. For classes that exercise the developed skills and feature summative assessment, we use an on-premises cloud environment. Our approach is unique in supporting both types of deployment. The environment is described as code using open and standard formats, defining individual hosts and their networking, configuration of the hosts, and tasks that the students have to solve. The environment can be repeatedly created for different classes on a massive scale or for each student on-demand. Moreover, the approach enables learning analytics and educational data mining of learners' interactions with the environment. These analyses inform the instructor about the student's progress during the class and enable the learner to reflect on a finished training. Thanks to this, we can improve the student class experience and motivation for further learning. Using the presented environments KYPO Cyber Range Platform and Cyber Sandbox Creator, we delivered the classes on-site or remotely for various target groups of learners (K-12, university students, and professional learners). The learners value the realistic nature of the environments that enable exercising theoretical concepts and tools. The instructors value time-efficiency when preparing and deploying the hands-on activities. Engineering and computing educators can freely use our software, which we have released under an open-source license. We also provide detailed documentation and exemplary hands-on training to help other educators adopt our teaching innovations and enable sharing of reusable components within the community.