Code plagiarism is an unfortunate problem which often plagues introductory level programming courses. In this poster, we study the relationship between plagiarism and student engagement using Submitty, our open-source course management, homework submission, and autograding system. We measure student participation through responses to live-lecture polls, posts in the course discussion forum, and daily and weekly office hours attendance patterns. While our assignments have strict deadlines, students are empowered to adapt deadlines by using penalty-free "late day" deadline extensions. Students who regularly participate in lecture polls are rewarded with additional late days. Submitty includes an open-source plagiarism detection module, Lichen, that detects software similarity that suggests academic dishonesty. Our implementation can be configured to examine every version of the assignment submitted by students and search for similarities between any version for any other student. We leverage the advantages of having a single, unified platform that combines submission timestamp data, course participation metrics, and code similarity data. Having many types of data stored in the same place allows us to effectively identify at-risk students earlier. We will present an exploration and analysis of data from an introductory level computer science course.

We present a system for the automated testing and grading of computer graphics applications. Our system runs, provides input to, and captures image and video output from graphical programming assignments. Instructors use a simple set of commands to script automated keyboard and mouse interactions with student programs at fixed times during execution. The resultant output, including plaintext standard output and mid-execution screenshots and GIFs, are displayed to the student to aid in debugging and ensure compliance with assignment specifications. Student output is automatically evaluated by basic text and image difference operations, or via an instructor-written validation method.

Fuzzing, testing a codebase against a set of randomly generated inputs, has become a promising model of testing across the industry due to its ability to reveal difficult to detect bugs. Separately, the use of randomized inputs when testing student code submissions removes the potential for student hard-coding behavior. Motivated by these factors, we present a solution for the automated generation of testcase inputs and expected outputs within Submitty, an open source automated grading system from Rensselaer Polytechnic Institute. We detail an enhanced workflow that allows instructors to provide our testing system with an assignment-specific input generation script and an assignment solution. The input generation script is run at student test-time, providing students with either entirely generated inputs or a combination of generated and hand-crafted testcases. The instructor solution is run against the same inputs to produce expected results. This model of testcase specification carries the benefit of simple regeneration of expected output files if an assignment's specification changes after submissions open or between semesters. We present preliminary results of the use of random input generation in our large introductory programming courses, and evaluate the ability of random inputs to curb student hardcoding behavior as it relates to an early submission incentive system, which grants students an extension for achieving a target assignment score early in the week an assignment is due. We examine random input generation's ability to reveal bugs in student submissions from previous semesters.

Grading web-based assignments poses many unique challenges when compared with other types of programming assignments. For introductory courses, grading involves not just validation of source code, but also performing some level of functional testing through a browser environment, where one clicks on content, and validates the browser state. In upper level courses, assignments increasingly use several different services, such as a web server and database, running concurrently, each potentially exposing ports for user access. Finally, for some assignments where students are encouraged to be creative, an instructor must then be able to view and interact with the running code, which has historically meant downloading, setting it up on their local machine, and running it -- which can prove burdensome. In this work, we present a system that can perform the task of automated grading, create a web-accessible environment on the server for the instructor to manually grade the assignment, and that can scale to handle many concurrent instances of both. To evaluate the effectiveness of our system, we will demonstrate its usage within the context of several assignments ranging over several different levels and courses of our Information Technology curriculum.

Submitty is an open-source course management platform for assignment submission with automated testing and grading, immediate feedback with an option for resubmission, and manual grading from course instructors and TAs. We have added an integrated discussion forum to the Submitty environment, which enables the students to communicate with their peers and have a public dialogue with the teaching staff outside of the classroom. Compared to a collection of closed-source external applications, our single-system login for distribution of course materials, assignment submission, and discussion forum participation is more convenient and accessible for students and simplifies administrative tasks. Students use the forum to ask questions about course logistics, homework help, exam review problems, and Submitty-related questions. The forum supports image and code segment posts, which facilitates targeted debugging help and sharing of additional reference material. An instructor-moderated forum is especially helpful in larger classes where students can feel it is otherwise difficult to get their questions answered. Every thread is tagged with course-specific categories and we provide full-text search of the forum. The forum becomes a curated collection of relevant frequently asked questions, which reduces the number of duplicate questions that must be answered by the staff. The forum includes customizable system and email notifications for important events such as: instructor announcement, assignment posted, team invitation, forum question answered, grades released, etc. Our current work focuses on private channels to facilitate team assignments (which can be monitored for participation by staff) and a grading interface for discussion-based assignments within the forum.

We present a container-based system to automatically run and evaluate networked applications that implement distributed algorithms. Our implementation of this design leverages lightweight, networked Docker containers to provide students with fast, accurate, and helpful feedback about the correctness of their submitted code. We provide a simple, easy-to-use interface for instructors to specify networks, deploy and run instances of student and instructor code, and to log and collect statistics concerning node connection types and message content. Instructors further have the ability to control network features such as message delay, drop, and reorder. Running student programs can be interfaced with via stream-controlled standard input or through additional containers running custom instructor software. Student program behavior can be automatically evaluated by analyzing console or file output and instructor-specified rules regarding network communications. Program behavior, including logs of all messages passed within the system, can optionally be displayed to the student to aid in development and debugging. We evaluate the utility of this design and implementation for managing the submission and robust and secure testing of programming projects in a large enrollment theory of distributed systems course. This research has been implemented as an extension to Submitty, an open source, language-agnostic course management platform with automated testing and automated grading of student programming assignments. Submitty supports all levels of courses, from introductory to advanced special topics, and includes features for manual grading by TAs, version control, team submission, discussion forums, and plagiarism detection.

Prior education research, including Computer Science, has established that students will attempt to cheat and violate academic integrity, with one of the more common forms being code plagiarism. The majority of existing tools for software plagiarism are closed source, requiring instructors to use them in a prescribed configuration and sending student code to a third-party server for analysis. At the core of this analysis is the need to perform a language-specific tokenization of the input program and then to use "digital fingerprinting" on the code to identify significant markers. This has required developers to write their own parser for each supported language, which is time-consuming to create and keep up-to-date, and thus a barrier to creation of these tools. Instead we bootstrap new languages into our plagiarism system by leveraging the "Language Server Protocol", an initiative to create open-source parsers and tokenizers for many languages (principally to be used within a range of popular IDEs). In this poster, we present our work on Lichen, the open source plagiarism detection tool that is integrated into the Submitty course management platform we use at Rensselaer Polytechnic Institute. This tool is a pipeline of modules for the specific tasks of tokenizing, fingerprinting, and then comparing the fingerprints for any number of files. Through this, a similarity score is generated for pairs of files, and these are used to help instructors determine to what extent code plagiarism has occurred.

With the continued growth of enrollment within computer science courses, it has become an increasing necessity to utilize autograding systems. These systems have historically graded assignments through either a jailed sandbox environment or within a virtual machine (VM). For a VM, each submission is given its own instantiation of a guest operating system and virtual hardware that runs atop the host system, preventing anything that runs within the VM communicating with any other VM or the host. However, using these VMs are costly in terms of system resources, making it less than ideal for running student submissions given reasonable, limited resources. Jailed sandboxes, on the other hand, run on the host itself, thus taking up minimal resources, and utilize a security model that restricts the process to specified directories on the system. However, due to running on the host machine, the approach suffers as new courses utilize autograding and bring their own set of potentially conflicting requirements for programming languages and system packages. Over the past several years, \em containers have seen growing popularity in usage within the software engineering industry as well as within autograding systems. Containers provide similar benefits of isolation as a VM while maintaining similar resource cost to running within a jailed sandbox environment. We present the implementation of both a jailed sandbox and container-based autograder, compare the running time and memory usage of the two implementations, and discuss the overall resource usage.

A flexible late submission policy can be used to help address breaches of academic integrity and student stress as they relate to programming assignments. To this end, we examine the use of late days -- earned, penalty free deadline extensions -- during our large format, two-semester introductory programming sequence. In particular, we examine correlations between patterns of student assignment submission and student plagiarism. Lexical-analysis-based techniques are used to identify probable cases of plagiarism across student submissions. Furthermore, we present metrics for estimating student stress at submission time. Our metric utilizes the automated grading score of the student"s current and previous submission attempts, the imminence of the assignment"s deadline, and the availability of late days. We corroborate these patterns with self-reported student surveys conducted at the end of the academic year.

Submitty is an open source programming assignment submission system from the Rensselaer Center for Open Source Software (RCOS) at Rensselaer Polytechnic Institute (RPI) accessed via an online interface. Submitty allows students to submit their code through file upload or version control, such as an internal Git/SVN server or Github, where it is then tested with a highly configurable and customizable automated grader. For each assignment, instructors can specify whether or not students can work in teams. For team assignments, the instructor can either assign teammates or allow the students to choose. In addition to the auto-grading for submissions, Submitty supports human grading. The human graded rubric is developed by the graders as they work, allowing reuse of common feedback messages and partial credit points. The rubric can be searched and modified during and after grading is complete for consistency. By default, grading is handled by instructors and TAs who are assigned to sections of students, which can be rotated through the semester. However, an instructor can choose to incorporate peer grading, which will allow students to anonymously view and submit grades for each other, receiving multiple peer grades per assignment. Submitty has been used at RPI for several years for a variety of courses, serving over 1500 students and 50 instructors and TAs each semester, and has recently been used by several other universities. We will present "case studies" of assignment configurations for autograding and manual grading and demonstrate the grading interface in support of team submissions and peer grading.

With surging enrollment in Computer Science courses at both the introductory and advanced level, it is critical to leverage automated testing and grading to ensure consistent assessment of student learning. Program analysis tools allow us to streamline the grading process so instructors and TAs can spend more time teaching, one-on-one tutoring, and mentoring students. We present complex use cases of automated assignment testing and grading within the open-source homework submission system, Submitty. Students receive immediate and detailed feedback from the automated grader, and can resubmit to correct errors. Submitty uses custom-built grading tools, including difference checking of plaintext program output, instructor authored assignment-specific custom graders, and static analysis tools that reason about program structure. In addition, it employs a variety of external tools, including version control (Git and SVN), unit testing frameworks (JUnit), memory debugging tools (Valgrind and DrMemory), and code coverage tools (Emma). In this poster we describe our experience with memory debugging and code coverage tools, and outline plans to include immutability inference and verification.

Traditionally, automated testing and grading of student programming assignments has been done in either a jailed sandbox environment or within a virtual machine (VM). For a VM, each submission is given its own instantiation of a guest operating system (OS) running atop the host OS, with no ability for a given submission to affect anything outside the VM. However, using a VM is expensive in terms of system resource usages, especially for RAM and memory, making it less than ideal for solutions without unlimited resources. Jailed sandboxes on the other hand allow student submissions to run directly on the server. Sufficient security measures must be implemented to ensure that students cannot access each other's submissions or the server at large, and must prevent runaway programs, over-utilization of system resources. Jailed sandboxes have a larger attack vector than VMs. Within the past several years, container systems have been gaining popularity and usage within the computer science industry, primarily through solutions such as Docker. These containers give similar security protections as a VM, but with better performance due to being able to utilize of resources installed within the host OS and other containers. However, containers do not have the full isolation of a VM, and thus implementing Docker for autograding ends up facing its own set of security concerns, as well as with the increased system resource usage. In this poster, we will analyze how well containers work, measuring system resources and throughput of submissions of containers against the traditional jailed environment.

Many computer science programs have capstone experiences or project courses that allow students to integrate knowledge from the full breadth of their major. Such capstone projects may be student-designed, instructor-designed, designed in conjunction with outside companies, or integrated with ongoing free and open source (FOSS) projects. The literature shows that the FOSS approach has attracted a great deal of interest, in particular when implemented with projects that have humanitarian goals (HFOSS). In this article, we describe five unique models from five distinct types of institutions for incorporating sustained FOSS or HFOSS (alternatively H/FOSS) project work into capstone experiences or courses. The goal is to provide instructors wishing to integrate open source experiences into their curriculum with additional perspectives and resources to help in adapting this approach to the specific needs and goals of their institution and students. All of the models presented are based on sustained engagement with H/FOSS projects that last at least one semester and often more. Each model is described in terms of its characteristics and how it fits the needs of the institution using the model. Assessment of each model is also presented. We then discuss the themes that are common across the models, such as project selection, team formation, mentoring, and student assessment. We examine the choices made by each model, as well as the challenges faced. We end with a discussion how the models have leveraged institutional initiatives and collaborations with outside organizations to address some of the challenges associated with these projects.

Student experience in introductory computer science classes can be enhanced by applying static analysis techniques to automatically grade assignments. At Rensselaer Polytechnic Institute (RPI), introductory computer science classes (using Python) exceed 650 students in size. As resources are limited, it is infeasible to have teaching staff individually examine each student's answer for small in-lecture exercises; however, qualitative data regarding student code independent from execution is still valuable (and in some cases required) to assess progress. When static analysis utilities were made available to instructors and integrated with automatic assignment testing, instructors were able to judge student performance and provide feedback at a scale that would otherwise be infeasible.

Submitty (http://submitty.org) is an open source programming assignment submission system from the Rensselaer Center for Open Source Software (RCOS) at Rensselaer Polytechnic Institute (RPI). Students can submit their code via a web interface in a variety of ways, where it is then tested with a highly configurable and customizable automated grader. Students receive immediate feedback from the grader, and can resubmit to correct errors as needed. Through an online interface, TAs can access detailed grading results and supplement the automated scores with manual grading (numeric and written feedback) of overall program structure, good use of comments, reasonable error checking, etc. and any non-programming components of the assignment. The instructor can also configure the system to allow for a configurable late day policy on a per assignment and per student basis. We currently use Submitty in eight different courses (spanning from introductory through advanced topics) serving over 1500 students and 35+ instructors and TAs each week. We will present a range of "case study" assignment configurations in a hands-on demo, going from simple through complex, using a variety of different automated grading methods including per-character and per-line output difference checkers, external unit testing frameworks (such as JUnit), memory debugging tools (Valgrind and DrMemory), code coverage (e.g., Emma), static analysis tools, and custom graders. Submitty can be customized per test case as appropriate to apply resource limits (running time, number of processes, output file size, etc.) and to display or hide from students the program output, autograding results, and testing logs.

We present a non-grade-penalty late day policy used in many of the large lecture, required courses in our computer science department. We study the effectiveness of this late day policy in reducing student stress, distributing demand for teaching assistant resources in peak hours before the homework deadline, and in maintaining or improving student understanding and homework grades. A complex late day policy can be efficiently implemented and managed within our open-source homework submission system that utilizes automated testing and grading, allowing students to submit and resubmit homeworks as they make progress on the assignment.

The Rensselaer Polytechnic Institute (RPI) Homework Submission Server is an ongoing open source project used to collect, compile, and automatically grade programming homeworks for students in our introductory and sophomore computer science classes. It allows viewing of homework, lab, test, and overall grades and late submissions and excused absences on homework. Our first hypothesis is that an electronic submission server is the preferred way for students to submit their coding homeworks because it provides immediate feedback about the correctness of their code and ensures fair, consistent grading since their code is compiled and run with the same test cases, on the same computer. Our second hypothesis is that students appreciate courses with a flexible policy for late submission of homeworks, allowing them to use a specific number of "late days" throughout the semester without penalty. We recently conducted a survey to test these hypotheses and will incorporate specific feedback from the students as we continue development and expand the server to more courses at RPI and other universities.

We present a study of a tangible user interface (TUI) for architectural design and daylighting analysis. This tool provides an intuitive way for architects and future building occupants to quickly construct physical models and then view a simulation of daylighting in the model at interactive rates. We conducted a user study of both formally-trained architects and non-architects in a set of analysis and design exercises. This study investigates the effectiveness of this interface as an educational tool, the precision and accuracy of the constructed physical models, and the overall effectiveness of the tangible interface. The four part study investigates users' intuitions about daylighting and their interaction with the tool for analysis of an existing space, for proposing renovations to the space, and for designing a totally new space with the same architectural program that better addresses the occupants' needs. These exercises revealed misconceptions in many of the participants' intuitions about day-lighting and overall the participants expressed interest in this simulation tool for daylighting analysis in architectural design.

Abstract The unintentional scattering of light between neighboring surfaces in complex projection environments increases the brightness and decreases the contrast, disrupting the appearance of the desired imagery. To achieve satisfactory projection results, the inverse problem of global illumination must be solved to cancel this secondary scattering. In this paper, we propose a global illumination cancellation method that minimizes the perceptual difference between the desired imagery and the actual total illumination in the resulting physical environment. Using Gauss‐Newton and active set methods, we design a fast solver for the bound constrained nonlinear least squares problem raised by the perceptual error metrics. Our solver is further accelerated with a CUDA implementation and multi‐resolution method to achieve 1–2 fps for problems with approximately 3000 variables. We demonstrate the global illumination cancellation algorithm with our multi‐projector system. Results show that our method preserves the color fidelity of the desired imagery significantly better than previous methods.

We present an application of interactive global illumination and spatially augmented reality to architectural daylight modeling that allows designers to explore alternative designs and new technologies for improving the sustainability of their buildings. Images of a model in the real world, captured by a camera above the scene, are processed to construct a virtual 3D model. To achieve interactive rendering rates, we use a hybrid rendering technique, leveraging radiosity to simulate the interreflectance between diffuse patches and shadow volumes to generate per-pixel direct illumination. The rendered images are then projected on the real model by four calibrated projectors to help users study the daylighting illumination. The virtual heliodon is a physical design environment in which multiple designers, a designer and a client, or a teacher and students can gather to experience animated visualizations of the natural illumination within a proposed design by controlling the time of day, season, and climate. Furthermore, participants may interactively redesign the geometry and materials of the space by manipulating physical design elements and see the updated lighting simulation.

Levee overtopping can lead to failure and cause catastrophic damage, as was the case during Hurricane Katrina. We present a computer simulation of erosion to study the development of the rills and gullies that form along an earthen embankment during overtopping. We have coupled 3D Smoothed Particle Hydrodynamics with an erodibility model to produce our simulation. Through comparison between simulations and between simulation and analogous laboratory experiments, we provide quantitative and qualitative results, evaluating the accuracy of our simulation.

Abstract When projectors are used to display images on complex, non‐planar surface geometry, indirect illumination between the surfaces will disrupt the final appearance of this imagery, generally increasing brightness, decreasing contrast, and washing out colors. In this paper we predict through global illumination simulation this unintentional indirect component and solve for the optimal compensated projection imagery that will minimize the difference between the desired imagery and the actual total illumination in the resulting physical scene. Our method makes use of quadratic programming to minimize this error within the constraints of the physical system, namely, that negative light is physically impossible. We demonstrate our compensation optimization in both computer simulation and physical validation within a table‐top spatially augmented reality system. We present an application of these results for visualization of interior architectural illumination. To facilitate interactive modifications to the scene geometry and desired appearance, our system is accelerated with a CUDA implementation of the QP optimization method.

This paper presents robust 3-D algorithms to segment vasculature that is imaged by labeling laminae, rather than the lumenal volume. The signal is weak, sparse, noisy, nonuniform, low-contrast, and exhibits gaps and spectral artifacts, so adaptive thresholding and Hessian filtering based methods are not effective. The structure deviates from a tubular geometry, so tracing algorithms are not effective. We propose a four step approach. The first step detects candidate voxels using a robust hypothesis test based on a model that assumes Poisson noise and locally planar geometry. The second step performs an adaptive region growth to extract weakly labeled and fine vessels while rejecting spectral artifacts. To enable interactive visualization and estimation of features such as statistical confidence, local curvature, local thickness, and local normal, we perform the third step. In the third step, we construct an accurate mesh representation using marching tetrahedra, volume-preserving smoothing, and adaptive decimation algorithms. To enable topological analysis and efficient validation, we describe a method to estimate vessel centerlines using a ray casting and vote accumulation algorithm which forms the final step of our algorithm. Our algorithm lends itself to parallel processing, and yielded an 8 x speedup on a graphics processor (GPU). On synthetic data, our meshes had average error per face (EPF) values of (0.1-1.6) voxels per mesh face for peak signal-to-noise ratios from (110-28 dB). Separately, the error from decimating the mesh to less than 1% of its original size, the EPF was less than 1 voxel/face. When validated on real datasets, the average recall and precision values were found to be 94.66% and 94.84%, respectively.

We present an application of interactive global illumination and spatially augmented reality to architectural daylight modeling that allows designers to explore alternative designs and new technologies for improving the sustainability of their buildings. Images of a model in the real world, captured by a camera above the scene, are processed to construct a virtual 3D model. To achieve interactive rendering rates, we use a hybrid rendering technique, leveraging radiosity to simulate the inter-reflectance between diffuse patches and shadow volumes to generate per-pixel direct illumination. The rendered images are then projected on the real model by four calibrated projectors to help users study the daylighting illumination. The virtual heliodon is a physical design environment in which multiple designers, a designer and a client, or a teacher and students can gather to experience animated visualizations of the natural illumination within a proposed design by controlling the time of day, season, and climate. Furthermore, participants may interactively redesign the geometry and materials of the space by manipulating physical design elements and see the updated lighting simulation.

We introduce a parallel approximation of an Over-determined Laplacian Partial Differential Equation solver (ODETLAP) applied to the compression and restoration of terrain data used for Geographical Information Systems (GIS). ODETLAP can be used to reconstruct a compressed elevation map, or to generate a dense regular grid from airborne Light Detection and Ranging (LIDAR) point cloud data. With previous methods, the time to execute ODETLAP does not scale well with the size of the input elevation map, resulting in running times that are prohibitively long for large data sets. Our algorithm divides the data set into patches, runs ODETLAP on each patch, and then merges the patches together. This method gives two distinct speed improvements. First, we provide scalability by reducing the complexity such that the execution time grows almost linearly with the size of the input, even when run on a single processor. Second, we are able to calculate ODETLAP on the patches concurrently in a parallel or distributed environment. Our new patch-based implementation takes 2 seconds to run ODETLAP on an 800 x 800 elevation map using 128 processors, while the original version of ODETLAP takes nearly 10 minutes on a single processor (271 times longer). We demonstrate the effectiveness of the new algorithm by running it on data sets as large as 16000 x 16000 on a cluster of computers. We also discuss our preliminary results from running on an IBM Blue Gene/L system with 32,768 processors.

We present a better algorithm for path planning on complex terrain in the presence of observers and define several metrics related to path planning to evaluate the quality of various terrain compression strategies.

We present the GeoStar project at RPI, which researches various terrain (i.e., elevation) representations and operations thereon. This work is motivated by the large amounts of hi-res data now available. The purpose of each representation is to lossily compress terrain while maintaining important properties. Our ODETLAP representation generalizes a Laplacian partial differential equation by using two inconsistent equations for each known point in the grid, as well as one equation for each unknown point. The surface is reconstructed from a carefully-chosen small set of known points. Our second representation segments the terrain into a set of regions, each of which is simply described. Our third representation has the most long term potential: scooping, which forms the terrain by emulating surface water erosion.Siting hundreds of observers, such as border guards, so that their viewsheds jointly cover the maximum terrain is our first operation. This process allows both observer and target to be above the local terrain, and the observer to have a finite radius of interest. Planning a path so that a smuggler may get from point A to point B while maximally avoiding the border guards is our second operation. The path metric includes path length, distance traveled uphill, and amount of time visible to a guard.The quality of our representations is determined, not only by their RMS elevation error, but by how accurately they support these operations.

Material limitations and fabrication costs generally run at odds with the creativity of architectural design, producing a wealth of challenging computational geometry problems. We have developed an algorithm for solving an important class of fabrication constraints: those associated with planar construction materials such as glass or plywood.Starting with a complex curved input shape, defined as a NURBS or subdivision surface, we use an iterative clustering method to remesh the surface into planar panels following a cost function that is adjusted by the designer. We solved several challenging connectivity issues to ensure that the topology of the resulting mesh matches that of the input surface.The algorithm described in this paper has been implemented and developed in conjunction with an architectural design seminar. How the participants incorporated this tool into their design process was considered. Their important feedback led to key algorithmic and implementation insights as well as many exciting ideas for future exploration. This prototype tool has potential to impact not only architectural design, but also the engineering for general fabrication problems.

We present a procedural approach to authoring layered, solid models. Using a simple scripting language, we define the internal structure of a volume from one or more input meshes. Sculpting and simulation operators are applied within the context of the language to shape and modify the model. Our framework treats simulation as a modeling operator rather than simply as a tool for animation, thereby suggesting a new paradigm for modeling as well as a new level of abstraction for interacting with simulation environments.Capturing real-world effects with standard modeling techniques is extremely challenging. Our key contribution is a concise procedural approach for seamlessly building and modifying complex solid geometry. We present an implementation of our language using a flexible tetrahedral representation. We show a variety of complex objects modeled in our system using tools that interface with finite element method and particle system simulations.