Affordable, secure, and scalable assessment delivery is an essential component of large university courses, whether online or in-person. The switch to Computer-Based Testing (CBT) can have a surprising, and almost transformational effect on pedagogy. Modern CBT systems provide almost unlimited flexibility in the types of questions they can support, for both manual grading and autograding, and CBT has now been adopted at several universities and is under serious consideration at others. In this BoF, faculty interested in learning about CBT and how to implement it at their institution are invited to ask their questions. Faculty experienced with CBT are invited to share how CBT has changed their approach, pedagogy, and behavior and how to advocate for its adoption.

We propose a novel, inverted two-stage exam format that encourages anticipation of transfer problems. We report on its design, use, and initial assessment for low-stakes quizzes in an algorithms course. A typical two-stage exam, where the group stage comes after the individual stage, emphasizes retrospective learning: reflecting on already-solved problems. Our inverted two-stage format places the group stage first, and incentivizes prospective learning: preparing for transfer to novel problems that will appear on the individual stage and subsequent assessments. TAs reported the new format leads to reliable engagement. In surveys, most students preferred inverted two-stage quizzes to individual quizzes plus a TA walkthrough. Students who preferred this format cited the value of learning from peers, brainstorming in the problem domain, and working out ambiguities in the domain and problems.

While we celebrate the dramatic increase in women's undergraduate enrollment at computer science programs around the country, to see this surge translate into career-long outcomes, we cannot ignore ongoing gendered and racialized disparities in computing, particularly as they relate to a student's sense of belonging. Even in times of high enrollment, fostering a sense of belonging cannot occur just through ad-hoc methods, the goodwill of a few faculty, or a standalone mentoring program. Policies and structures must be put into place and enacted holistically. We report on a multi-phase, 10-year case study of undergraduate student experiences at the University of Illinois (2007, n=61; 2017, n=339). Our 2017 study explores the policies and structures enacted over a decade and their impact on departmental culture. We report on three areas: i) Inclusive classroom experiences; ii) Quality of mentorship opportunities; iii) Student sense of identity. While there have been significant departmental improvements, there are some cultural, policy, and structural issues to be addressed in order to foster a sense of belonging and success for all students.

We are excited to introduce CoMoTo -- the Collaboration Modeling Toolkit -- a new, web-based application that expands and enhances well-known software similarity detection systems. CoMoTo is an end-to-end data management, analysis, and visualization system whose purpose is to assist instructors of courses requiring programming exercises to monitor and investigate the extent of student collaboration, both allowed and illicit. We describe CoMoTo's interface, which was designed to facilitate scrutiny of collaboration data projected along student, course, assignment, etc. attributes, and to allow for interactive visualization of pairwise similarity measures via a dynamic graph. We also elaborate on the details of CoMoTo's implementation. Finally, we briefly discuss two use cases that foreshadow CoMoTo's broad utility in student code analysis, not only for plagiarism detection, but also for investigating early student coding styles, and for evaluating software similarity detection systems, themselves.

No abstract available.

A concept inventory is a standardized assessment tool intended to evaluate a student’s understanding of the core concepts of a topic. In order to create a concept inventory it is necessary to accurately identify these core concepts. A Delphi process is a structured multi-step process that uses a group of experts to achieve a consensus opinion. We present the results of three Delphi processes to identify topics that are important and difficult in each of three introductory computing subjects: discrete mathematics, programming fundamentals, and logic design. The topic rankings can not only be used to guide the coverage of concept inventories, but can also be used by instructors to identify what topics merit special attention.

A Delphi process is a structured multi-step process that uses a group of experts to achieve a consensus opinion. We present the results of three Delphi processes to identify topics that are important and difficult in each of three introductory computing subjects: discrete math, programming fundamentals, and logic design. The topic rankings can be used to guide both the coverage of standardized tests of student learning (i.e., concept inventories) and can be used by instructors to identify what topics merit emphasis.

We introduce the boasting problem, wherein useful trends in historical ordinal data (rankings) are discovered. Claims of the form our object was ranked r or better in x of the last t time units, are formalized, and maximal claims (boasts) of this form are defined under two natural partial orders. For the first partial order, we give an efficient and optimal algorithm for finding all such maximal claims. For the second, we apply a classical result from computational geometry to achieve an algorithm whose running time is significantly more efficient than that of a naive one. Finally, we connect this boasting problem to a novel variation of the problem of finding optimized confidence association rules as originally posed by Fukuda, et al. [2], and give an efficient algorithm for solving a simplification of the new problem.

The problem of finding optimized support association rules for a single numerical attribute, where the optimized region is a union of k disjoint intervals from the range of the attribute, is investigated. The first polynomial time algorithm for the problem of finding such a region maximizing support and meeting a minimum cumulative confidence threshold is given. Because the algorithm is not practical, an ostensibly easier, more constrained version of the problem is considered. Experiments demonstrate that the best extant algorithm for the constrained version has significant performance degradation on both a synthetic model of patterned data and on real world data sets. Running the algorithm on a small random sample is proposed as a means of obtaining near optimal results with high probability. Theoretical bounds on sufficient sample size to achieve a given performance level are proved, and rapid convergence on synthetic and real-world data is validated experimentally.

We consider the index selection problem. Given either a fixed query workload or an unknown probability distribution on possible future queries, and a bound B on how much space is available to build indices, we seek to build a collection of indices for which the average query response time is minimized. We give strong negative and positive peformance bounds.Let m be the number of queries in the workload. We show how to obtain with high probability a collection of indices using space O(B ln m) for which the average query cost is optB, the optimal performance possible for indices using at most B total space. Moreover, this space relaxation is necessary: unless NP ⊆ nO(log log n), no polynomial time algorithm can guarantee average query cost less than M1--ε optB using space αB, for any constant α, where M is the size of the dataset. We quantify the error in performance introduced by running the algorithm on a sample drawn from a query distribution.