Introductory programming courses often rely on small code-writing exercises that have clearly specified problem statements. This limits opportunities for students to practice how to clarify ambiguous requirements -- a critical skill in real-world programming. In addition, the emerging capabilities of large language models (LLMs) to produce code from well-defined specifications may harm student engagement with traditional programming exercises. This study explores the use of ``Probeable Problems'', automatically gradable tasks that have deliberately vague or incomplete specifications. Such problems require students to submit test inputs, or `probes', to clarify requirements before implementation. Through analysis of over 40,000 probes in an introductory course, we identify patterns linking probing behaviors to task success. Systematic strategies, such as thoroughly exploring expected behavior before coding, resulted in fewer incorrect code submissions and correlated with course success. Feedback from nearly 1,000 participants highlighted the challenges and real-world relevance of these tasks, as well as benefits to critical thinking and metacognitive skills. Probeable Problems are easy to set up and deploy at scale, and help students recognize and resolve uncertainties in programming problems.

Learners must be able to critique AI-generated code, which is often plausible but not always appropriate. Refute Questions have been proposed as one way to develop this ability. Each such question has two components: a task specification and a purported solution (typically, a single function) for that task. Our prior work [1, 4, 5] has focused on Refute Questions where purported solutions are inappropriate because they are logically incorrect. Given such a question, learners must demonstrate their understanding of both the buggy function and the specification by providing an input on which the function’s behavior does not satisfy the specification.

To broaden participation, competitive programming contests may include beginner-level problems that do not require knowledge of advanced Computer Science concepts (e.g., algorithms and data structures). However, since most participants have easy access to AI code-generation tools, these problems often become trivial to solve. For beginner-friendly programming contests that do not prohibit the use of AI tools, we propose Probeable Problems: code writing tasks that provide (1) a problem specification that deliberately omits certain details, and (2) a mechanism to probe for these details by asking clarifying questions and receiving immediate feedback. To evaluate our proposal, we conducted a 2-hour programming contest for undergraduate Computer Science students from multiple institutions, where each student was an active member of their institution’s ACM student chapter. The contest comprised of six Probeable Problems for which a popular code-generation tools (e.g., GitHub Copilot) were unable to generate accurate solutions due to the absence of details. Students were permitted to work individually or in groups, and were free to use AI tools. We obtained consent from 26 groups (67 students) to use their submissions for research. To determine whether Probeable Problems are suitable for such contests, we analyze the extent to which the code submitted by these groups identifies missing details.

GitHub Copilot is an artificial intelligence tool for automatically generating source code from natural language problem descriptions. Since June 2022, Copilot has officially been available for free to all students as a plug-in to development environments like Visual Studio Code. Prior work exploring OpenAI Codex, the underlying model that powers Copilot, has shown it performs well on typical CS1 problems thus raising concerns about its potential impact on how introductory programming courses are taught. However, little is known about the types of problems for which Copilot does not perform well, or about the natural language interactions that a student might have with Copilot when resolving errors. We explore these questions by evaluating the performance of Copilot on a publicly available dataset of 166 programming problems. We find that it successfully solves around half of these problems on its very first attempt, and that it solves 60% of the remaining problems using only natural language changes to the problem description. We argue that this type of prompt engineering, which we believe will become a standard interaction between human and Copilot when it initially fails, is a potentially useful learning activity that promotes computational thinking skills, and is likely to change the nature of code writing skill development.

Bug-eecha is a web-based game that seeks to help novice programmers in two ways: (1) in comprehending programming problems, and (2) in developing a thorough set of test cases for such problems. At present, we restrict problems to pure functions with one or more arguments and a single output. The instructor seeds the game with questions, each with four hidden implementations: one must be correct and three must be buggy. Students must specify sufficiently many test cases (input/output examples) to eliminate all buggy implementations. The game allows the instructor to set various penalties on a per-question basis. These include a penalty for providing an incorrect output for a given input (which may correspond to a mistake in parsing the problem correctly), and a separate penalty for providing a test case that does not eliminate additional buggy implementations (i.e., for failing to improve the thoroughness of the test suite). We are testing this game in our CS1 (Fall 2022) course.

To help students develop a critical eye for human and (increasingly) machine-generated artifacts such as code, documentation, and more, this workshop proposes Refute questions. Students are given an artifact created for a stated purpose and asked: Why does the artifact fail to serve that purpose? Students must provide evidence demonstrating this failure. After a hands-on introduction to Refute questions in their originally proposed context (an alternative to 'Explain in Plain English' questions), participants will receive and review a richer variety of Refute questions for autograded formative and summative assessments, targeting a CS1 course in Python. Then, based on their interest, they will join a subgroup to create theme-specific Refute questions and identify theme-specific challenges such as question-generation and evaluation strategies. Finally, as a group, participants will identify a wish-list of Refute-related tools to be created and research questions to be answered. The workshop caters to a diverse audience: CS instructors at all levels, designers of Intelligent Tutoring Systems and interactive e-books, and CS education researchers.

The introduction of Large Language Models (LLMs) has generated a significant amount of excitement both in industry and among researchers. Recently, tools that leverage LLMs have made their way into the classroom where they help students generate code and help instructors generate learning materials. There are likely many more uses of these tools -- both beneficial to learning and possibly detrimental to learning. To help ensure that these tools are used to enhance learning, educators need to not only be familiar with these tools, but with their use and potential misuse. The goal of this BoF is to raise awareness about LLMs and to build a learning community around their use in computing education. Aligned with this goal of building an inclusive learning community, our BoF is led by globally distributed discussion leaders, including undergraduate researchers, to facilitate multiple coordinated discussions that can lead to a broader conversation about the role of LLMs in CS education.

Ensuring that programs are syntactically correct imposes a cognitive load on novice programmers. ‘Gradual programming’ languages such as Hedy seek to reduce this load. This paper describes AryaBota, a visual and tolerant approach to gradual programming. Unlike Hedy, which enforces syntax transitions when moving from one level to the next, AryaBota supports forward and backward tolerance – allowing novices to learn syntax at their own pace and eliminating the need to unlearn syntax. It also provides visual feedback on program correctness, helping novices catch errors easily. AryaBota was evaluated via an online Python workshop for students in grades 6-8, during which 1410 programs were executed. The analysis of these programs and the final survey were indicative of tolerance being a useful feature to the students while learning programming.

Executable examples enable students to check their comprehension of programming problems at any time before, during, or after implementation. Students express their understanding of a problem by specifying input-output pairs, and they receive immediate feedback on their understanding when these pairs are executed against correct and buggy solutions specified by the instructor. These solutions are typically executed on a server, and we find evidence that some students in an introductory programming course are wary of revealing their fragile problem comprehension in exchange for feedback. We propose a student-side mechanism that enables students to receive the same feedback privately. We also conduct a study to investigate differences in student ability to create valid and thorough examples using the server-side or the student-side feedback mechanisms.

poster Share on Refute: An Alternative to 'Explain in Plain English' Questions Author: Viraj Kumar Dayananda Sagar University, India Dayananda Sagar University, IndiaView Profile Authors Info & Claims ICER 2021: Proceedings of the 17th ACM Conference on International Computing Education ResearchAugust 2021 Pages 438–440https://doi.org/10.1145/3446871.3469791Online:17 August 2021Publication History 0citation42DownloadsMetricsTotal Citations0Total Downloads42Last 12 Months42Last 6 weeks3 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Most of the 2.5 million undergraduates in CS and related programs in India are enrolled in institutions outside the top tier. While official policies broadly acknowledge that many instructors at these institutions lack the requisite domain competence, there is no precise understanding of their limitations. In this study, we analyse the ability of CS instructors from 10 mid-tier institutions in Karnataka (India) to perform CS1-level programming tasks. While instructors can perform simple code tracing tasks with reasonable accuracy, we find a sharp fall in their ability to reason about code abstractly, to modify given code, and to write code. Our findings have immediate implications for faculty training initiatives, which must initially focus on these foundational skills.

We generally accept that some students will struggle on simple tasks at the end of an introductory programming course. We investigate the disturbing possibility that many CS faculty in India also struggle on such tasks. Based on initial data from a pan-India unproctored online programming test, we find that the 193 self-identified faculty participants performed poorly on this test. Our preliminary findings are consistent with recent national policies that acknowledge the lack of requisite expertise and depth among many faculty in India.

Whilst today's computing educators are very successful at preparing students in the technologies of computing, they are also entrusted with the responsibility of preparing them to face the complex issues surrounding ethics. These issues are perpetually changing, unpredictable, and challenging. Twenty-first century computing students graduate into a world in which many of the greatest ethical issues roiling society are centered on their field. A recent surge of scholarly interest has produced some rich overviews of computing ethics pedagogy. This paper seeks to go beyond the Western perspectives typically centered in that work to consider how these problems may be understood in quite different ways through the value systems of diverse professional groups, political systems, and cultures. The breadth of computing ethics as a subject makes a universal overview of its education here unrealistic. Instead, a selective review of relevant international literatures is presented to illustrate a range of influences on the teaching of computing ethics. Different cultural conceptualizations of ethics and contrasting curricula from a set of global regions are outlined. Other influences include (i) differing levels of professional association from global down to national; and (ii) subfields and topics of computing (including diversity and social responsibility). This position paper considers these perspectives on computing ethics education relative to each other without attempting to reconcile any differences, and emphasizes the importance of adopting culturally relevant dimensions if students are to take ownership of the ethics of their future work.

SIGCSE is a global organization with members from well over one hundred countries, but attendance at SIGCSE conferences is not always reflective of membership as a whole. Attendance at the SIGCSE Technical Symposium is overwhelmingly from the United States, with more than 92% of all attendees in recent years having a U.S. affiliation. This panel, which includes members of the SIGCSE Board and the Symposium International Committee, will present the state of Symposium participation from outside the U.S. in an effort towards understanding what can be done to expand global participation in the Symposium. The panelists span the range from students to long-time faculty and together have computing education experience from seven countries across six continents. The panel will also solicit views from attendees on barriers to global Symposium participation and ways to improve non-U.S. participation in SIGCSE's flagship conference.

Climate change is the defining environmental challenge now facing our planet. Babies born today will be 22 when global warming reaches 1.5 C, according to the latest IPCC report and forecasts. What will life be like in 2040? How do we as educators respond today, to the challenges that lie ahead for the next generation?

Instructors and learners in every country have increasing access to smartphones, and a variety of pedagogical tools and techniques have been developed to leverage the affordances of these devices in classroom settings. Two common patterns of communication between such devices are many-to-one (e.g., when each learner's device sends data to the instructor's device) and one-to-many (e.g., when the instructor's device sends data to each learner's device). Every classroom usually requires additional infrastructure (such as a Wi-Fi router) to reliably transmit this data, which can be challenging for institutions with limited resources. We therefore present BeaconNet, a hybrid Bluetooth/Wi-Fi beacon-based smartphone ad-hoc network that requires no infrastructure beyond commodity smartphones already owned by learners and instructors. BeaconNet supports many-to-one and one-to-many communication between these devices. For applications where the volume of communication is small (a few tens of bytes per message), it is faster and scales to large classrooms (up to 80 learners) more reliably than existing infrastructure-less solutions.

Intelligent Tutoring Systems (ITS) that match or exceed human tutoring are often the product of expensive research efforts in specific domains, and the lack of high-quality ITS in other domains is partly explained by this high creation cost. In this paper, we propose a hybrid ITS, where human instructors perform two critical tasks within an otherwise automated system: (1) giving learners specific types of feedback, and (2) scaffolding students’ reasoning. We argue that such a hybrid ITS can be cost-effective when the pool of learners is too small to justify the cost of creating an ITS. To illustrate these arguments concretely, we have implemented a hybrid ITS for a component of the undergraduate Computer Science curriculum: the construction of DFA (deterministic finite automata).

Instructors who wish to disseminate digital content to learners in an economical manner must consider the costs of (1) content creation (software and hardware), (2) content dissemination (data transfer costs), and (3) content access (hardware and software needed to view content). In this paper, we present a novel technique – photocasting – to greatly reduce these three costs in a one-to-one learning environment. Our technique has minimal requirements: a low-end smartphone for instructors and learners, and a modest additional effort by instructors after recording their digital notes. The delivery cost is primarily determined by the size of the audio, and is approximately 100 times smaller than the size of a traditional video of comparable quality. We have implemented our technique as a prototype for the Android platform, and we report initial results of a small-scale user study with three participants: a school teacher, a private tutor, and a high-school student.

Students in the developing world are frequently cited as being among the most important beneficiaries of online education initiatives such as massive open online courses (MOOCs). While some predict that online classrooms will replace physical classrooms, our experience suggests that blending online and in-person instruction is more likely to succeed in developing regions. However, very little research has actually been done on the effects of online education or blended learning in these environments. In this paper we describe a blended learning initiative that combines videos from a large online course with peer-led sessions for undergraduate technical education in India. We performed a randomized controlled trial (RCT) that indicates our intervention was associated with a small but significant improvement in performance on a summative exam. We discuss the results of the RCT and an ethnographic study of the intervention to make recommendations for future, scalable blended learning initiatives for places such as India.

This paper describes an ongoing research project to develop micro-notes: digital footnotes created by a community of instructors and learners to enhance the utility of educational videos. Specifically, we demonstrate that a suitably motivated group of undergraduates were able to create over 1,400 annotations for over 30 hours of video lectures in just ten days. Furthermore, over 57% of these micro-notes were rated by at least two peers in two more days. The quality of these micro-notes was manually assessed in two ways: (1) an independent group of undergraduates evaluated a large sample (35%) of micro-notes that had been peer-rated above average by the original group, and (2) the author, who was also the instructor for a similar course, evaluated a smaller sample of the top 6% of peer-rated notes. Our results indicate that peer-ratings are a useful (but not infallible) indication of the quality of micro-notes, and can be used to effectively generate useful annotations of video lectures for beginner students.

Due to the recent emergence of massive open online courses (MOOCs), students and teachers are gaining unprecedented access to high-quality educational content. However, many questions remain on how best to utilize that content in a classroom environment. In this small-scale, exploratory study, we compared two ways of using a recorded video lecture. In the online learning condition, students viewed the video on a personal computer, and also viewed a follow-up tutorial (a quiz review) on the computer. In the blended learning condition, students viewed the video as a group in a classroom, and received the follow-up tutorial from a live lecturer. We randomly assigned 102 students to these conditions, and assessed learning outcomes via a series of quizzes. While we saw significant learning gains after each session conducted, we did not observe any significant differences between the online and blended learning groups. We discuss these findings as well as areas for future work.

We propose the study of visibly pushdown automata (VPA) for processing XML documents. VPAs are pushdown automata where the input determines the stack operation, and XML documents are naturally visibly pushdown with the VPA pushing onto the stack on open-tags and popping the stack on close-tags. In this paper we demonstrate the power and ease visibly pushdown automata give in the design of streaming algorithms for XML documents.