This work in progress Innovative Practice paper presents our work evaluating how Large Language Models (LLMs) can be utilized to aid in the development of scalable and authentic programming exam questions for students in upper-division courses. Traditionally, students' understanding of computing concepts are often verified using written exams. This is especially the case in larger universities due to their overall enrollment sizes, making more authentic, involved assignments particularly hard to facilitate. Since the rise of Chat-GPT and other LLMs, the use of such exams has seen a resurgence in popularity in classrooms globally. Fears have recently grown that programming assignments, particularly those that are take home assignments, are not adequate assessments of students' understanding given the ease at which students can use LLMs such as Chat-GPT to obtain answers. multiple-choice exams have been a traditionally popular programming exam format, where-in students are required to choose the correct answer to a prompt or the segment of code that best fits within a larger block of code. Another form of exam question that has been traditionally popular is providing students with blocks of code and ask students to either interpret the code's meaning or find the errors in the code. Both of these styles of assignments provide opportunities for students to demonstrate a deep understanding of code syntax and structure, while also tasking them to demonstrate their understanding of what the intended purpose of the code is. However, the development of these exam problems has traditionally required a substantial amount of work for instructors and teaching assistants to develop. Our goal with this work was to determine if LLMs represent effective tools for developing exam questions. Additionally, we wanted to see if LLMs could effectively design problems based on learning outcomes, allowing the instructor to become the evaluator of the exam questions rather than the original author. Our motivation for this was to develop a reproducible and easier to implement methodology for developing exams at scale for instructors while alleviating the workload imposed on instructors during the development of exams. Our initial research has indicated the use of these LLMs for exam problem generation greatly reduces the workload for instructors while allowing for the creation of far richer programming questions for students that require them to apply more of their knowledge to individual problems. We have had a good amount of success in developing these problems for upper-division courses, as well as introductory-level courses. This work represents initial steps towards the use of these LLMs for generating exams and more work is needed to determine the actual efficacy and long-term benefits and reliability of these tools. Regardless, we are confident that LLMs in their current form represent incredibly powerful tools for instructors to utilize in the development of their course exams.

This innovative practice paper explores two differing approaches to the use of artificial intelligence tools in computer science classrooms. Artificial Intelligence (AI), while not a new technology, has seen a rise in popularity over the past two years, with companies such as OpenAI, Google, and Microsoft making readily available generative AI tools for anyone to use. This surge in AI popularity has led to a rise in its use in educational settings, in some cases allowed and in others discouraged or disallowed. A debate has risen among academics, particularly in higher education, about what AI's place in education is, with some educators actively encouraging its use while others view the use of AI as a form of academic dishonesty. Given AI is only advancing and is becoming more prevalent, it will only continue to become a more dominant force throughout education and the world at large. This paper presents two educators' distinct viewpoints and experiences on how AI should be handled in computer science courses (absolutely forbidden vs. decriminalized). The goal of this paper is to present different perspectives as well as concrete experiences we have had with AI in our own classrooms to encourage others to consider their own positions on its use and its implications for their own learning environments. While the debate on the place of AI in education is a long way from being settled, educators need to think about making choices, clearly articulating policies, and evaluating the positives and negatives of positions about AI in their classrooms.

WIP Research Paper: Identifying the Threshold Concepts within a discipline illuminates the key concepts or components within the curriculum. Once students have overcome the barrier of learning these concepts, they often will be better able to identify as members of that community, and understanding a threshold concept opens the door to learning additional concepts. Within computer science, there has been much debate over what concepts could potentially be threshold concepts. Meyer and Land originally defined threshold concepts as resulting in an individual being placed into a state of uncertainty or liminality, and successfully traversing this liminal state results in a transformation of the individual with potential feelings of accomplishment. While there has been some work attempting to identify threshold concepts within the first year or beginning stages of programming, little work has considered the intermediate years (years 2 and 3) of university study and what potential threshold concepts exist during this time period. Our goal with this work is to help address this gap that exists by answering the following research question: What do intermediate students identify as being troublesome and/or ‘uncomfortable to learn’ within their computer science coursework?” A first cohort of participants were interviewed in late 2022 and coding began in the first quarter of 2023. The coding of these interviews proved challenging. The students who were interviewed often did not give enough information about a concept for the coders to identify whether the concepts had the key characteristics of threshold concepts. It was considered and accepted that the original interview protocol was not supporting the participants well in eliciting the types of information needed to identify a concept as threshold. The interview protocol was redesigned, and new interviews commenced. The work presented here is a continued discussion of the initial findings and the subsequent change in interview protocol, with the primary improvement being an inclusion of concept mapping. Concept maps, or a graphical representation of the interrelationship of topics and ideas, coupled with an intentional simplification of associated terminology, are expected to reduce cognitive load as participants reflect on their learning experiences. Interviews with the revised protocol including concept maps have been more engaging and productive in identifying potential threshold concepts within the intermediate computer science curricula.

Since the 1960s, the ACM has provided routinely updated guidelines for what concepts constitute a computer science curriculum, with the latest version currently in development in 2023. These guidelines have traditionally provided a model curriculum from which universities can choose to adopt or modify for their own purposes. What is unclear, however, is to what degree schools follow the curriculum recommendations that the ACM provides. While most faculty and students likely have knowledge of their own institution's curriculum, as well as what courses are offered at a small selection of other schools, the goal of the work presented in this poster is to distill a cohesive view of what computer science curriculums in their second and third years look like across a broad range of universities across a range of institutions. Our goal with this work was to answer the following question: What do computer science course requirements look like at a wide range of different institutions? We believe the work will help those who are trying to develop curriculum changes within their own institutions and aims to provide a more cohesive view of what trends and patterns exist in course offerings and degree requirements for computer science in the second and third years across a wide range of universities.

WIP Research Paper: The ability to identify threshold concepts within a discipline recognizes key components within a curriculum which, when learned, enables an individual to demonstrate as a member of that community. Within computer science, potential threshold concepts have sparked debates among researchers. According to the original definition by Meyer and Land, threshold concepts result in an individual being placed into a state of uncertainty or liminality and successful traversal of this liminal state results in a potentially irreversible transformation. When seeking to identify threshold concept, researchers often search for what concepts students feel are ‘troublesome’ or ‘difficult to learn’ since this state is often difficult to describe or understand when currently inside it or just past it. While threshold concepts have been identified for the beginning stages of programming, there is little to no work on the intermediate years of university computing education (years 2 & 3) and what potential threshold concepts exist during that time for computer science students. Our goal with this research is to help address this gap by answering the following research question: What concepts do intermediate students identify as being troublesome and/or ‘uncomfortable to learn’ within their computer science coursework?

Computing Education Research (CER) has spent a large effort focusing on the difficulties faced by first year students, particularly in the first programming course students take (commonly referred to as CS1), as well as final-year students as they transition from their undergraduate studies to professional positions. There exists little if any research into middle-year students and fails to consider what difficulties students face and potential retention issues that exist within the middle-years. Recent data has suggested that retention is not a problem limited to first-year students as has been previously assumed, indicating that there is an even greater need for research into supporting middle-years students. This work attempts to fill this gap in our understanding of middle-year student difficulties framed around the retention issues they face. Retention has been shown to be influenced by three factors: students’ sense of belonging, their identity, and their self-efficacy. These three retention factors are impacted negatively when students run into troublesome learning experiences while not feeling as though they have the support needed to overcome them. This work looks to identify what these troublesome learning experiences consist of in terms of content knowledge and develop and pilot an intervention to support their experiences with learning, retention factors, and retention. The proposed work will contribute to CER by providing a framework for retention, a concept inventory of middle-year troublesome knowledge, and potential interventions to support one or more of the concepts.

Academic Integrity (AI) violations have long been a concern of educators and academic institutions within all fields of study. AI violations can consist of a broad range of student behaviors that are considered dishonest, including but not limited to plagiarism, copying others' assignments, and paying for others to complete their work [1]. As the risks for academic integrity issues become ever more prevalent, particularly as more academic institutions rely on online course formats, researchers are continually attempting to identify what underlying factors lead students to commit AI violations. The COVID-19 pandemic among other factors has contributed to the rise in online learning formats. This change in course modality combined with easier access to digital resources has lowered the boundary for many students for violating AI, thus increasing our need to understand what factors influence students to consider violating AI.

This work-in-progress research paper discusses issues of academic integrity which have long been a concern of education researchers and academic institutions within all fields of study. Academic integrity (AI) violations can consist of a broad range of student behaviors that are considered dishonest, including but not limited to plagiarism, copying others’ assignments, and paying for others to complete their work. A plethora of researchers have attempted to identify what underlying factors lead students to commit AI violations, and have identified several potential factors, including a lack of self-control, students’ ethical views of AI, perceived opportunities to commit AI violations, involvement in extracurricular activities, and students’ social groups.Another topic that has in recent times become a focal point of education research is students’ sense of belonging within their field of study. Researchers have identified several factors that contribute to students feeling less welcome within higher education, particularly within Engineering and Computer Science. Students who feel a lower sense of belonging have been identified as being at higher risk of performing poorly with their studies and retention rates for these students are historically lower. Despite this, little research has been conducted to examine where issues with students’ sense of belonging and their incidences of AI violations overlap. In this study, we attempt to try and better understand this relationship between students’ sense of belonging and AI violations by attempting to answer the following question: Can students’ sense of belonging within their discipline influence their propensity to violate academic integrity? We take up a student centered, restorative position, and choose to understand the cognitive underpinnings behind students’ choices to violate AI, with the goal of identifying if student outreach and more inclusive practices within Engineering and Computer Science can be utilized to prevent instances of AI violations. To accomplish this, we are employing a qualitative interview-based study of first-year students studying Computer Science at a large public university in the northeastern United States. We plan to analyze transcript data collected during interviews using Grounded Theory and Narrative Analysis methodologies. Our goal with this study is to draw awareness to additional underlying causes behind students deciding to violate AI, with the hope that this research will encourage academic institutions to employ a more preventative approach to handling AI issues by ensuring all students feel welcome and included within their chosen field of study, thereby helping prevent AI violations before they happen.

Recursion is often cited as both an essential and problematic topic for students to learn, though often without justifications for these claims. The goals of this research are focused on forming a framework from which to evaluate how students' form effective mental models for recursion, including the role of misconceptions, teaching strategies and experiences in the formation of these mental models. As a preliminary step in this research, a systematic literature review was performed to better understand the existing literature on the teaching and learning of recursion. The review consists of a total of 224 papers from several sources, including the ACM Digital Library, IEEE, ScienceDirect, Taylor & Francis, and Wiley. The literature review was designed to answer the following research question: What methods for teaching recursion have been evaluated and have been determined to be effective? The results show that while many methods for teaching recursion have been developed, there is a surprising lack of evidence for the effectiveness of many of the methods presented, with many articles simply presenting the teaching method without providing any evidence for the method's efficacy.

One of the most ubiquitous techniques for evaluating research, particularly in educational settings, has been Likert-style questionnaires. Having a participant rank their level of engagement, agreement, or interest on a scale can provide powerful insight into an individual's perspective and attitude. However, as computing education matures as a discipline it becomes important for us to examine our practices and ensure that we are employing the techniques of evaluation properly. Likert-style questionnaires can be prone to unintentional biases and noise which, from the perspective of the researcher, may affect the study in unknown, unexpected, and potentially undesirable ways. In this research we seek to aid the computing education researcher not only avoid biases and noise but also improve the reproducibility of their work. First, we establish best practices for these instruments by synthesizing recommendations from the original creator, Rensis Likert, the Center for Disease Control (CDC), the Association of American Medical Colleges (AAMC), and the American Association for Public Opinion Research (AAPOR). We then considered additional sources of unintentional biasing and noise resulting from the measurement scales/response options, specifically possible biasing resulting from how response options are presented to the study participant. With these recommendations, we then examined 121 evaluation instruments used in computer science education research and curated in the csedresearch.org online database to see how often researchers unintentionally fall victim to the pitfalls of ambiguity, awkward phrasing, use of conjunctions, leading or biased statements, and double negatives. We found that the occurrence of at least one of these problematic statements to be in 82.6% of all instruments. We also examine demographic information for the intended study participants, number of response options, and how those options are presented. Overall, while we see many instrument authors falling victim to a few of these common pitfalls, we believe that increased awareness of potentially problematic statements/measurement scales and their impact on research bias and reproducibility will help insulate computing education researchers from avoidable complications and strengthen the discipline throughout.

This WIP Research paper is a follow-up to a study conducted in 2013 by McGill, Decker, and Settle that investigated the effects of pre-college computing experiences on students' decisions to study computer science at university. Their results indicated that the exposure to pre-college computing activities impacted students in different ways, particularly when looking at perceived impact by men and women participants [1], [2]. After six years and a myriad of changes to the K-12 computing landscape, it was time to see if impacts to current undergraduates were different than previously observed. This current study is a pilot qualitative study in which we interviewed seven undergraduate students about their experiences with computing prior to college. We were particularly interested in finding out about the nature of their computing experiences, whether they enjoyed them, and what they would change to make such experiences better for future participants. We used grounded theory and thematic coding to encode the interview transcripts, enabling us to look for common themes among the interview subjects. We are looking for elements in the interviews that could point to key differences in the pre-college computing landscape that have impacted student experiences that are different from those previously observed and thus impacting their experiences in university with computing. The goal of this preliminary study was to get a sense of what aspects of students' exposure to pre-college computing experiences have changed since 2013 and what changes should be made when creating a follow-up to this initial study.