It is generally accepted amongst CS educators that creating ''good'' variable names is important. However, it is difficult to understand what makes a variable name ''good''. This makes it challenging to both educate and assess students on this topic. This is particularly true for high school teachers, many of whom are not experienced programmers. We therefore develop a decision-tree taxonomy of variable identifiers, with the goal of better equipping teachers to explain, understand, and assess what is ''good'' when it comes to their students' variable naming conventions and behaviours. Our taxonomy is based on 152 real high school student submissions to an automated assessment system used for a New Zealand high school programming standard in the Python programming language. From this, we perform an inductive analysis of 831 variable identifiers, and create a classification scheme made up of 25 distinct categories. We demonstrate that this taxonomy is usable outside of the developers, by scoring high inter-rater reliability with someone not involved in its development, and discuss how it could be adopted by high school educators.

This working group aims to develop a research-based AI concept inventory (AI CI) to assess the understanding of foundational AI concepts among non-experts. By identifying core concepts and common misconceptions through literature reviews, expert consultations, and iterative validation, the group will create a user-friendly assessment tool that can be used to capture snapshots of AI understanding, support benchmarking across contexts, and inform educational initiatives and policy. Designed for diverse non-expert audiences, including educators, students, and the general public, this tool can provide valuable insights into how AI knowledge evolves over time, contributing to the broader goal of promoting AI literacy in everyday contexts.

Teaching computer programming has become common in high schools, and often students' programming work needs to be assessed formally. In Aotearoa New Zealand this formal assessment happens primarily in the last three years of high school, and is currently carried out manually by teachers. Here we explore the possibility of automating this assessment with the goal of decreasing teacher workload, and increasing fairness and transparency. To do this, we first report on a multi-part survey of teachers in New Zealand schools. The first part of the survey identifies the range of languages that need to be supported. The second part of the survey, in conjunction with a literature review of automated assessment, gives us a range of tools that are designed to automatically assess programming. We evaluate this group of assessment tools based on a wide range of criteria gathered from the survey of schools and the literature review. This includes the programming languages these tools can check, with a priority given to programming languages teachers are already using in the classroom. We also consider their resilience to common programming mistakes (such as infinite while loops), whether the tools can check style and quality, whether they include gamification, whether they use closure to encourage students, what kinds of programming constraints we can put on questions, and if they are open source. Based on this analysis, CodeRunner was the contender that most strongly stood out for our context, and is the tool we will move forward with in implementing automated assessment in a New Zealand high school environment.

Randomization is a technique that can be used with programming assessments to discourage academic misconduct by making it unlikely for two colluding students to get the exact same questions. Previous research about randomization has shown it to be an effective tool for addressing academic misconduct, but this work often focuses on randomization broadly, with few considering specific techniques. In contrast, we consider different randomization techniques and the contexts that they are best suited to. In addition, we investigate the effectiveness of randomization techniques against emerging AI technologies. This is done by exploring randomization in the context of an online quiz system that evaluates student responses to pro-gramming challenges, specifically the CodeRunner system for the Moodle learning management system. We provide a classification of techniques, and discuss the benefits of each. This classification starts with simpler techniques, such as shuffling question order, shuffling multi-choice question options, and question pooling. We then move on to more advanced techniques, including simple substitution, altering expected output, switching logic, and steganography. We also investigate two approaches to generating randomized questions, considering the benefits and drawbacks of each. These approaches are generating the questions beforehand (pre-generation) and generating the questions when the quiz is started (on-the-fly generation). We then identify four categories of assessment based on assessment that is formative/summative, and proctored/non-proctored, then identify which randomization techniques are suited for each category. Finally, we test randomized questions against OpenAI's Codex, to see if these techniques could prevent this new opportunity for academic dishonesty. We found that there are some types of questions that Codex currently performs poorly on, such as program reasoning, and creating complex classes, but overall randomization was not effective in defeating it, with Codex scoring 79.7% on questions that were created after it was trained, and 85.3 % on questions that could have been available to it when it was trained.

Programming plays a paramount role in many educational policies and initiatives. However, the current focus on coding skills poses a risk of giving pupils an over simplistic and impoverished idea of what programming means and involves. Their experiences would be much more significant if learning were aimed at understanding the richness of the nature of programs. In fact, programs are strange creatures that escape simple definitions. They are real, in that they affect our real lives; they are abstract, in that they process abstract entities; and they are concrete, in that they take up space in digital devices memory, and can be copied, transferred, corrupted. Thus, understanding the multifaceted nature of programs is crucial knowledge for all citizens of the digital era, and a fundamental component of such an understanding is getting a sense of how programs are created and work (i.e., the programming process). To the best of our knowledge, there is no Nature of Programs framework (e.g., a set of statements that describe what the nature of programs is), that teachers and policy makers can use to shape their practice and targets. The goal of the WG is developing such a framework, by collecting and organizing contributions from CER, CS experts, and educators.