As undergraduate computer science classes grow in size, institutions increasingly rely on asynchronous computer-based assessments. To investigate whether exam timing reveals evidence of cheating, we analyze 21,403 submissions from 51 asynchronous exams across two undergraduate courses in this retroactive study. We extend prior research on proctored multiday exams by introducing a comparison in student performance trends between two distinct assessment modes: on-site proctored and off-site unproctored. We find that performance declines throughout the exam window in both modes. We observe a weak negative correlation between start time and performance, with standardized scores decreasing by 0.14 points per hour (on-site proctored) and 0.61 points per hour (off-site unproctored). In addition, start-time distributions and student surveys reveal behavioral differences. On-site proctored exams follow a centered start-time distribution, likely influenced by a reserved lecture hour. In contrast, off-site unproctored exams show a left-tailed distribution, with most students starting later than intended. This pattern suggests that greater scheduling flexibility leads to later exam starts, potentially exacerbating performance declines due to academic procrastination.

The popularity of autograding has grown due to increasing class sizes and the need to reduce grading load while ensuring quality. Autograding has conventionally been used for multiple choice and fill in the blank questions, or to check code correctness. In this work, we discuss the use of autograders at UBC and some non-conventional autograding implementations in our curricula. We reflect upon our autograder use in our courses and discuss the benefits, implications, and considerations of this pedagogical choice.

The rise of online assessments has motivated the development of randomized question banks, with randomization referring to the generation of different variants of a question. Although not all randomization efforts are equally effective in generating question isomorphs, the current classification of questions solely as randomized or not fails to address the varying degrees of randomization. To address this limitation in describing the diversity of randomization designs, we introduce a framework that outlines six distinct randomization levels. Additionally, we designed practical guides to assist educators in effectively using the framework, aligning with their pedagogical objectives. Through our application of this framework to classify around 200 questions from two courses, we further highlight the generalizability of the framework and reveal insights into the considerations and challenges associated with incorporating question randomization into computer science curricula.

PrairieLearn is an open source, extensible online assessment platform built on modern web technologies. In this workshop, we will focus on how PrairieLearn can be used to improve student learning in undergraduate computer science classes. However, the platform is also more than suitable for use as an assessment engine in a variety of courses including the humanities, social, physical, and life sciences. In the first part of the workshop, we will showcase multiple question styles that highlight PrairieLearn's abilities as an online platform, including deploying automatically and manually graded questions at scale in large classes. In the second part of the workshop, we will discuss the anatomy of a PrairieLearn question, create several custom questions, and design assessments in PrairieLearn. In the third part, we will share strategies on adopting PrairieLearn at your institution. In particular, how algorithmically generated questions can be used in support of alternative grading schemes such as Mastery- or Specifications-Grading. Finally, we will share how PrairieLearn can be extended to support other coding languages and paradigms with custom and external autograders. There will be plenty of opportunities for questions throughout the workshop, and we intend to leave plenty of time for additional 1:1 support and training. Attendees will be able to attend the session virtually and are recommended to bring a web-connected computing device. By the end of the session, attendees will know enough to run a whole class on PrairieLearn including designing questions appropriate for homework, labs, and tests.

The identification of student struggles has drawn increasing interests from computing education and learning analytics communities in recent years, considering the high failure rate and fast enrollment growth of computer science courses. Prior studies on this topic employed a multitude of data sources and methodologies with varying degrees of success. Nearly all studies attempted to predict low overall course performance to identify struggling students, risking oversimplifying student learning and struggles. Additionally, many studies utilize data sources that are limited to their original contexts or local student demographics, making it difficult to replicate or put the findings into practice. To address these gaps, we studied the feasibility of identifying student struggles at the topic level using features that are agnostic to courses and contexts. Our results show that it is possible to identify student struggles at a more fine-grained level within days. Our findings contribute new insights into automatic identification of student struggles at the topic level on a large scale, which can be used to guide meaningful interventions on student learning.

We propose a novel, inverted two-stage exam format that encourages anticipation of transfer problems. We report on its design, use, and initial assessment for low-stakes quizzes in an algorithms course. A typical two-stage exam, where the group stage comes after the individual stage, emphasizes retrospective learning: reflecting on already-solved problems. Our inverted two-stage format places the group stage first, and incentivizes prospective learning: preparing for transfer to novel problems that will appear on the individual stage and subsequent assessments. TAs reported the new format leads to reliable engagement. In surveys, most students preferred inverted two-stage quizzes to individual quizzes plus a TA walkthrough. Students who preferred this format cited the value of learning from peers, brainstorming in the problem domain, and working out ambiguities in the domain and problems.

Cloud-based JupyterHub installations support easy access to computing environments for intro computing using Jupyter Notebooks. We propose an interface and technical design to smoothly integrate JupyterHub with Canvas for intro students.

In this workshop, we as a group use Appreciative Inquiry (AI) techniques to explore and develop our strengths as CS educators. As a participant, you will gain appreciation for your strengths as an educator, with concrete plans for building on these strengths. You will also learn about AI as a qualitative research methodology that is complementary to more common CS research methodologies, and that you can apply to evaluate and improve your own educational practice. Appreciative Inquiry drives change by building on what's already working well in an organization. Similarly to other qualitative methods, AI generates rich, deep feedback that is grounded in stakeholders' experiences, but in contrast to other methods its focus on strengths and positives surfaces unique, strength-based findings and make it an energizing and fulfilling approach to professional development and the scholarship of teaching and learning. AI is commonly used in education and organizational research and is an effective and community-building way to drive organizational or program change and positively impact participants' morale. We will share our materials and key tips to enable you to apply Appreciative Inquiry in your own work. You may wish to run Appreciative Inquiry workshops with students as an evaluation method, or run them with colleagues for professional development or for promoting positive change in your unit or program, or take smaller steps integrating the appreciative mindset into your teaching or other professional work.

In a session with live music and collaborative parody, we celebrate the long tradition of computing parody songs ("filks") and their potential to contribute to education and a fun environment in CS courses. We perform several new and classic CS filks, such as "Like it Called on Me (QuickSort)" [1], interspersing discussion of how and why these parodies were written. We also propose a song for the audience to parody and walk them through a structured small-group activity to help them brainstorm topics and lyrical phrases and fit them to the existing lyrics and music. Attendees should expect to laugh (and possibly cry) with the singing and to leave inspired to incorporate filks and other creative activities into their computing education practice: playing, performing, or writing songs themselves, and encouraging their students to do so as well.

A Concept Inventory (CI) is a set of multiple choice questions used to reveal student's misconceptions related to some topic. Each available choice (besides the correct choice) is a distractor that is carefully developed to address a specific misunderstanding, a student wrong thought. In computer science introductory programming courses, the development of CIs is still beginning, with many topics requiring further study and analysis. We identify, through analysis of open-ended exams and instructor interviews, introductory programming course misconceptions related to function parameter use and scope, variables, recursion, iteration, structures, pointers and boolean expressions. We categorize these misconceptions and define high-quality distractors founded in words used by students in their responses to exam questions. We discuss the difficulty of assessing introductory programming misconceptions independent of the syntax of a language and we present a detailed discussion of two pilot CIs related to parameters: an open-ended question (to help identify new misunderstandings) and a multiple choice question with suggested distractors that we identified.

This session is a chance for researchers studying concept inventories (CIs)--low-cost assessments highlighting student misconceptions in a field--and CS education practitioners to communicate about advances in concept inventories in an engaging and utterly ridiculous way.

No abstract available.

In this paper, we triangulate evidence for five misconceptions concerning binary search trees and hash tables. In addition, we design and validate multiple-choice concept inventory questions to measure the prevalence of four of these misconceptions. We support our conclusions with quantitative analysis of grade data and closed-ended problems, and qualitative analysis of interview data and open-ended problems. Instructors and researchers can inexpensively measure the impact of pedagogical changes on these misconceptions by using these questions in a larger concept inventory.

We present a highly reusable "inverted" project in which students learn asymptotic and practical behaviour of dictionary data structures--linked-lists, arrays, balanced trees, and hash tables--in an atmosphere of mild competition. Much like David Levine's Nifty Assignment "Sort Detective", rather than implementing the dictionaries, students' programs generate input to our (unlabeled) implementations, and students use timing data to label the implementations. Much like Bryant and O'Halloran's computer architecture labs, students also compete to "convince" a web-based, automated system that their input generators distinguish the dictionaries based on trend-line behaviour. Initial assessment results suggest the project makes substantially improves students' understanding of practical performance of various dictionary data structures, particularly hash tables. UBC has used the project in three terms, and we plan to use it at UBC and U Toronto in coming terms.

We report a study of how dual display screens were used in classroom lectures for university-level courses across a variety of disciplines during five academic terms over a 2-year period. Our goal was to understand the pedagogical consequences of using more than a single electronic display screen to support classroom lectures. We deployed an in-house software system (MultiPresenter) in real classrooms. We examined the use of MultiPresenter by 8 university instructors who taught 15 courses with a total of 1,147 students during 13-week regular terms or 6-week summer terms. We observed classroom lectures, interviewed instructors, collected screen images and log files of MultiPresenter usage, and administered questionnaires to students about their subjective impressions. Based on these data, we analyzed how instructors used MultiPresenter in order to identify examples of how multiple display screens might best be used for educational purposes. The analysis revealed that the following practices are beneficial: the ability to keep information persistent for extended periods, the increased flexibility in where and when information is shown, capability for side-by-side comparison of full screens of information, simultaneous visibility of both overview (“roadmap”) and detailed (“content”) information, and extra space to annotate information. Possible hazards include difficulty focusing on specific information amidst a large amount of information and too much information changing too quickly without proper indication of the changes.

We report on best practices we have established to teach first-year computer science students in closed laboratories, founded on over three years of action research in a large introductory discrete mathematics and digital logic course. Our practices have resulted in statistically significant improvements in student and teaching assistant perception of the labs. Specifically, we discuss our practices of streamlining labs to reduce load on students that is extraneous to the lab's learning goals; establishing a positive first impression for students and TAs in the early weeks of the term; and effectively managing the teaching staff, including weekly preparation meetings for TAs using and a gradual, iterative curriculum development cycle that engages all stakeholders in the course.

Most CS1 assignments are known answer assignments, requiring students to implement already-solved problems with no user but their grader and themselves. In this paper, we present Poogle: a freely available framework for designing unknown-answer assignments. Poogle assignments are open-ended, graphical, and multi-user. Poogle makes it easy for students to share their creations with their users: fellow students and the general public. We present two freely available CS1 assignments based on Poogle and discuss outcomes from use of one of them in a CS1 course.

No abstract available.

Digital inking systems are becoming increasingly popular across a variety of domains. In particular, many systems now allow instructors to write on digital surfaces in the classroom. Yet, our understanding of how people actually use writing in these systems is limited. In this paper, we report on classroom use of writing in one such system, in which the instructor annotates projected slides using a Tablet PC. Through a detailed analysis of lecture archives, we identify key use patterns. In particular, we categorize a major use of ink as analogous to physical gestures and present a framework for analyzing this ink; we explore the relationship between the ephemeral meaning of many annotations and their persistent representation; and we observe that instructors make conservative use of the system's features. Finally, we discuss implications of our study to the design of future digital inking systems.

In this paper, we report on an empirical exploration of digital ink and speech usage in lecture presentation. We studied the video archives of five Master's level Computer Science courses to understand how instructors use ink and speech together while lecturing, and to evaluate techniques for analyzing digital ink. Our interest in understanding how ink and speech are used together is to inform the development of future tools for supporting classroom presentation, distance education, and viewing of archived lectures. We want to make it easier to interact with electronic materials and to extract information from them. We want to provide an empirical basis for addressing challenging problems such as automatically generating full text transcripts of lectures, matching speaker audio with slide content, and recognizing the meaning of the instructor's ink. Our results include an evaluation of handwritten word recognition in the lecture domain, an approach for associating attentional marks with content, an analysis of linkage between speech and ink, and an application of recognition techniques to infer speaker actions.

Computer science instructors frequently teach using slides displayed with a computer and a data projector. This has many advantages, e.g., ability to present prepared materials and ease of switching the display to a development environment during mid-presentation. However, existing computer-based presentation systems severely limit flexibility in delivery, hindering instructors' extemporaneous adaptation of their presentations to match their audiences. One major limitation of computer-based systems is lack of support for high-quality handwriting over slides, as with overhead projectors and other manual presentation systems. We developed and deployed Classroom Presenter, a Tablet PC-based presentation system that (1) combines the advantages of existing computer-based and manual presentation systems and (2) builds on these systems, introducing novel affordances. Classroom Presenter has been used in 25 Computer Science courses at three universities. In this paper we describe the system, summarize results from its deployment, and detail several novel uses of the system by instructors in computer science courses.

We have developed and deployed a lecture presentation system in which the instructor uses a Tablet PC as a presentation device. The system was deployed in six university courses in Autumn 2002 and has been favorably received by students and instructors. In our system, the instructor holds a pen-based computer that is wirelessly networked with another computer driving the classroom projector. The instructor displays slides from the tablet and can write on top of them. Various navigation and control facilities are available.

Pedagogy of large lecture classes has traditionally focussed on deemphasizing the problems their size creates. This approach has yielded valuable practical advice for instructors. However, this paper argues that there are pedagogical advantages to the large lecture format and that exploiting these advantages can further improve classroom instruction. I present some advantages of large classes and anecdotes that demonstrate how to exploit these advantages.

Applications of machine learning can be viewed as teacherstudent interactions in which the teacher provides training examples and the student learns a generalization of the training examples. One such application of great interest to the IUI community is adaptive user interfaces. In the traditional learning interface, the scope of teacher-student interactions consists solely of the teacher/user providing some number of training examples to the student/learner and testing the learned model on new examples. Active learning approaches go one step beyond the traditional interaction model and allow the student to propose new training examples that are then solved by the teacher. In this paper, we propose that interfaces for machine learning should even more closely resemble human teacher-student relationships. A teacher's time and attention are precious resources. An intelligent studentmust proactively contribute to the learning process, by reasoning about the quality of its knowledge, collaborating with the teacher, and suggesting new examples for her to solve. The paper describes a varietyof richinteraction modes that enhance the learning process and presents a decision-theoretic framework, called DIAManD, for choosing the best interaction. We apply the framework to the SMARTedit programming by demonstration system and describe experimental validation and preliminary user feedback.