When employing the Socratic method of teaching, instructors guide students toward solving a problem on their own rather than providing the solution directly. While this strategy can substantially improve learning outcomes, it is usually time-consuming and cognitively demanding. Automated Socratic conversational agents can augment human instruction and provide the necessary scale, however their development is hampered by the lack of suitable data for training and evaluation. In this paper, we introduce a manually created dataset of multi-turn Socratic advice that is aimed at helping a novice programmer fix buggy solutions to simple computational problems. The dataset is then used for benchmarking the Socratic debugging abilities of a number of language models, ranging from fine-tuning the instruction-based text-to-text transformer Flan-T5 to zero-shot and chain of thought prompting of the much larger GPT-4. The code and datasets are made freely available for research at the link below.

Without a sense of belonging, students may become disheartened and give up when faced with new challenges. Moreover, with the sudden growth of remote learning due to COVID-19, it may be even more difficult for students to feel connected to the course and peers in isolation. Therefore, we propose a recommendation system to build connections between students while recommending solutions to challenges. This pilot system utilizes students’ reflections from previous semesters, asking about learning challenges and potential solutions. It then generates sentence embeddings and calculates cosine similarities between the challenges of current and prior students. The possible solutions given by previous students are then recommended to present students with similar challenges. Self-reflection encourages students to think deeply about their learning experiences and benefit both learners and instructors. This system has the potential to allow reflections also to help future learners. By demonstrating that previous students encountered and overcame similar challenges, we could help improve students’ sense of belonging. We then perform user studies to evaluate this system’s potential and find that participants rated 70% of the recommended solutions as useful. Our findings suggest an increase in students’ sense of membership and acceptance, and a decrease in the desire to withdraw.

Abstract Writing software exploits is an important practice for offensive security analysts to investigate and prevent attacks. In particular, shellcodes are especially time-consuming and a technical challenge, as they are written in assembly language. In this work, we address the task of automatically generating shellcodes, starting purely from descriptions in natural language, by proposing an approach based on Neural Machine Translation (NMT). We then present an empirical study using a novel dataset ( Shellcode_IA32 ), which consists of 3200 assembly code snippets of real Linux/x86 shellcodes from public databases, annotated using natural language. Moreover, we propose novel metrics to evaluate the accuracy of NMT at generating shellcodes. The empirical analysis shows that NMT can generate assembly code snippets from the natural language with high accuracy and that in many cases can generate entire shellcodes with no errors.

Writing exploits for security assessment is a challenging task. The writer needs to master programming and obfuscation techniques to develop a successful exploit. To make the task easier, we propose an approach (EVIL) to automatically generate exploits in assembly/Python language from descriptions in natural language. The approach leverages Neural Machine Translation (NMT) techniques and a dataset that we developed for this work. We present an extensive experimental study to evaluate the feasibility of EVIL, using both automatic and manual analysis, and both at generating individual statements and entire exploits. The generated code achieved high accuracy in terms of syntactic and semantic correctness.

Data sharing is a common contribution to open science. The creation of open datasets can speed up research advancements by allowing researchers to focus efforts on developing and validating analytical techniques, rather than on obtaining data. Open datasets also allow researchers to benchmark new analytical approaches against a known standard, and increase the reproducibility of research. The field of higher education learning analytics could benefit from the creation of open, shared datasets on higher education students as these data do not currently exist in open and accessible formats. Here, we propose the use of synthetic data generators to create open access versions of student data. Synthetic datasets have an advantage over real data, as private student data is protected by federal laws. We compare the characteristics of the synthetic data to the original data and illustrate a model for how the synthetic data can be leveraged for developing and optimizing a common learning analytics algorithm.

This work in progress presents a model for first undergraduate software engineering course as a core course of the computer science curriculum. The course is designed to be offered in the fourth or fifth semesters (i.e., end of sophomore or beginning of the junior year) for students who have completed the introductory programming and data structures courses. In addition, they may have some basic knowledge of databases and web technologies. Moreover, in our curriculum, students at this level have not been exposed to any codebase of real-world application and particularly of large size (>10,000 lines) codebases yet. Based on such situations, the major focus of this course is on teaching the fundamentals of software engineering as a methodology of developing real-world software with an emphasis on: 1) software systems in the enterprise level, 2) basic modeling using functional, flow, and behavioral diagrams, and 3) team-based agile project development. The paper discusses our novel course configuration of the three emphasized elements above, where students work with open source software in this class as part of course activities and assignments to simulate working on an enterprise project and learn agile development Our observation as well as our industry partners indicated that students enjoy the open source challenges and demonstrate professional competency after this course. Our initial findings include a positive impact of open source and team work on our students.

We introduced a model of activity-based active learning class which has been practiced for a few years in [4]. While it may seem an easy task, designing an effective activity-based active learning can be quite challenging. Active learning adds new benefits to teaching including increased student involvement, social interaction, and hands-on learning, etc. [1]. However, in some situations, it may not provide an efficient learning environment. For example, we may assume that it is effective for educators to provide students with an environment where they can perform peer instruction and social learning. It could be further assumed that students will naturally find their way around the activities by following the instructions or peer instruction [2]. Such issues depend on the way activities are designed and executed. In this work, we discuss how the scaffolding of activities can help students stay engaged with the course without feeling lost or disconnected. Scaffolding refers to methods used to help students progress towards stronger understandings and eventually more independence in the learning process [3]. Our scaffolding methods smoothly transition students from lower to higher levels of challenge through an appropriate breakdown of course contents into activities of various types, proper sequencing of concepts and formative assessment questions. We believe that such proper breakdown is essential to keeping students engaged. We don't want students to feel bored (when challenge level is too low) or overwhelmed (when the challenge level is too high). Our findings indicate statistically significant differences in participation and engagement when using scaffolded activities in our introductory programming course (CS1).