This work-in-progress research paper analyzes student activity as captured by the sequence of programming submissions they make. Although students who complete assignments in a computing course are often assessed in a summative way through the submission of a completed program, their activities during development provide additional insights into their work and problem-solving processes that cannot otherwise be captured. In the context of a data structures & algorithms course taught at the sophomore level, we first examined the number of attempts that students made while completing assignments throughout the semester. We then examined the cumulative portion of the class that had submitted an assignment relative to the due date, and the time periods after assignment release that students are active. Our work helps to illustrate how a programming activity trace provides a unique source of information on how a submission evolves. It was found that even with only feedback as motivation, students made many submissions and that number increased during the semester. Our initial work indicates that a substantial amount of data is available and helps suggest what aspects might provide information useful for further analytics using traditional statistics or machine-learning approaches.

Students taking computing courses develop skills by applying programming to various problems. In the past decade, courses have started to move from manually graded assessments to those which can be automated. A typical motivation for the use of automated assessment is to enable scaling, which is of particular importance for courses taught with high enrollment. However, automated assessment provides additional advantages such as reproducibility and rapid feedback. We have developed an automated assessment platform that performs a combination of static and dynamic analysis to evaluate student work. Our focus has been not on scaling but rather on serving student educational outcomes, both at the student level (e.g., providing feedback according to teaching best practices) and program level (e.g., ensuring that students across semesters meet the same standard). In this paper, we report on the design, development, and introduction of an automated assessment tool to improve instruction. The use of this tool has been aligned with a course on data structures & algorithms taught at the sophomore level. We discuss the development of the tool, the techniques that it uses, and evaluate its impact on grade accuracy.

Introducing software engineering students to data science provides an opportunity to reinforce foundational topics while also introducing students to emerging technologies. This paper discusses the experience of developing an undergraduate course that introduces data science. In addition to coverage of techniques in data management, data exploration, and machine learning, the course has a focus on scientific thinking along with the application of tools from software engineering. In contrast to the more common machine-learning first approach of applying algorithms and justifying them in terms of a numerical accuracy measure, we emphasize understanding data and drawing conclusions in an explainable way. In this work, we show how several foundational topics as defined by the Software Engineering Body of Knowledge (SWEBOK) map to topics in our data science course. We also discuss the design of a semester-long project that is used to elicit various data science skills.

This Innovative Practice Full Paper presents an automated reasoning approach to the assessment of a student's programming process as captured by an autograder. When students are assessed in computing courses, they are typically assessed on their completed work, such as a programming assignment. Although such an assessment measures a student's general ability, it provides only an indirect measure of the effectiveness of their programming (or problem-solving) process. Without the ability to examine the process by which a student completes an assignment, an instructor may have difficulty teaching problem-solving skills since they cannot measure how the student's skill develops. Some computing courses make use of an autograder which automatically assesses student homework. The trace captured by an autograder is a unique source of information about a student's problem-solving ability, which can be analyzed to measure properties of that learner's process. This provides a way to assess the process-based aspect of student learning, enabling instructors to not only teach problem-solving skills but to determine how well students acquire these skills. This enables teachers to evaluate the methodology they employ and to select instructional material based on student need. Automatically analyzing the trace captured by an autograder has two major challenges: the scale of the data makes traditional approaches computationally expensive, and the definition of a problem-solving process is often expressed in a commonsense way based on an expert's intuition. These challenges can be addressed by using an Answer Set Programming (ASP) approach. ASP is a declarative language that is intended for solving computationally hard problems, and through various extensions, supports human-style commonsense reasoning. Assessing a student is computationally hard due to the combinatorial nature of the different ways an assignment may be solved. In this paper, we discuss the design of an ASP-based system for automatically assessing a student's problem-solving process using the trace captured by an autograder, and present a preliminary evaluation of the system applied within a university-level course on data structures & algorithms.

This Innovative Practice Full Paper presents an analysis of autograder usage on software engineering design outcomes. Autograders have found increasing use in software engineering and computer science courses due to a variety of reasons. They are often introduced to support large class sizes, increase the reproducibility of grades, and provide formative feedback to students. Many of the classes that apply autograders are introductory in nature, such as CS1 and CS2 courses, or an introduction to algorithms course. These courses typically focus on assessing student programming ability on specific aspects of a programming language (e.g., Java inheritance) or algorithms (e.g., sorting, searching). One common drawback to automated assessment with programming is the need to impose additional structure on assignments. For example, students asked to implement a game might be given a detailed list of functions to implement, skipping analysis and design to determine what functions and algorithms are relevant. In this work, we discuss the experience of introducing an autograder into a second-year course that teaches software engineering topics such as personal software process, object-oriented design, and UML. Although we have the traditional motivations to use an autograder, it is challenging in a software engineering course since the additional structure required by an autograder is potentially detrimental to the engineering design outcomes. To address this, we developed an autograder in such a way as to minimize the design choices that it imposed on the students. The partial structure that is imposed together with the formative feedback produced by the autograder helps to scaffold student learning as they transition to more complex programs. The autograder was used during a spring 2023 offering of our software engineering course. We analyzed the data gathered in this course to evaluate the course change by asking: how does introducing an autograder impact course design outcomes? And, how does introducing an autograder impact personal process data that is collected? Answering these questions helps to inform the software engineering community about the suitability of introducing automated assessment tools into advanced computing and software engineering courses.

This Innovative Practice Full Paper presents the experience of using formative data produced by an autograder tool to understand how students complete assignments in a second year computing course. Autograding has increasingly been used in computing courses to evaluate student work. Using autograders supports the scalability of classes and provides other advantages such as accurate assessment, reproducibility over semesters, and rapid feedback. As a consequence of using an autograder, students produce a trace of their problem-solving process while completing assignments. In contrast to the summative assessment performed on a final submission, this trace captures the formative steps that were involved in the development of the final submission. Analyzing this trace can provide insights into student problem-solving methodology as well as the structure of the problems being assigned, and the impact of classroom interventions during an assignment. To produce this view into student learning, we have developed an autograder with the initial goal of improving student instruction. It uses a combination of static and dynamic analysis techniques to support accurate assessment and the ability to check for more complex requirements than can be captured by approaches that compare program output with a ground truth. The tool has been used in a sophomore level course on data structures & algorithms. This course uses the Java programming language and introduces topics such as lists, searching, sorting, binary search trees, priority queues, hash tables, and graphs. Our current analysis work focuses on analyzing traces produced by students to identify difficult parts of an assignment (as represented by grading regressions), and the path through possible solution states that students take as they complete an assignment. In this paper, we discuss the design of this tool, what insights the data captures provide into student learning, and give ideas for future directions supported by gathering formative assessment data.

We propose a new practice for the instruction of problem-solving skills: the Analysis-Design-Justification (ADJ) framework. The ADJ framework consists of learning outcomes which represent problem-solving skills, a type of problems to assess those outcomes, and a three-step process for problem-solving. Problems are open-ended and ill-defined, like those in the real world. Although there are several approaches to instructing problem-solving, ADJ supports a course environment with four specific requirements: established syllabi (ADJ may be added to a class already covering many topics), scale (ADJ can handle large classes), modality (ADJ can be performed synchronously and asynchronously), and heterogeneity (ADJ helps to provide scaffolding for diverse student populations). This framework takes inspiration from Polya's problem-solving process and uses the theoretical foundations underlying Problem-based Learning (PBL). The ADJ framework was enacted in the fall 2019 section of a Software Engineering (SE) class on operating-systems at Arizona State University (ASU). In the class, students were introduced to the ADJ framework through a series of group activities, and then asked to solve three ADJ problem sets. Students responded positively to use of the ADJ framework as both an instructional tool for class material, and to mature their problem-solving skills. However, some students had issues carrying out the ADJ process due to its focus on justification as opposed to constructing designs.

This Innovative Practice Work in Progress presents using the concept of a simulated internship to transition students to problems (and environments) which resemble those found in industry or research. New engineering graduates struggle not just with the lack of technical skills specific to a company but have issues in generalizing their existing skill set. They also lack procedural knowledge that is contextual to industry (e.g., dealing with stakeholders, communicating solutions). This requires an educational setting that is more conducive to practical problems than a typical course - students already acquire the needed technical skills but instead need to practice applying and integrating them. This can be supported with the creation of a simulated internship (SimInt) environment that aims to enable development and assessment of outcomes that more accurately capture solving real world problems than current technical skill outcomes. Although students who can complete an internship are already able to mature their skills, some students, for many reasons, are unable to pursue such opportunities. This can be addressed with SimInt. We propose a course to mimic the environment of an internship: students would be assigned a large-scale problem that is open-ended and ill-defined and solve the problem in the context of a simulated company. The focus of this experience would be on maturing problem-solving skills rather than practicing specific technical skills. This would involve integrating aspects of process (e.g., applying with a resume, interviewing, etc.), the environment of working with different stakeholders, and the development of solutions. In this work, we contribute a model for a real-world work environment that captures aspects not found in a typical classroom environment. We describe the differences between typical content-based course (e.g., lecture, flipped) and the new model. The model is described in terms of system elements and interactions between them that are conducive to student learning. SimInt is intended to offer an experience similar to a capstone while supporting scalability and being more appropriate as a formative experience.

Problem solving is a necessary skill for computing students to succeed in industry. Students commonly find it difficult to apply their technical skills in solving real-world problems, which can be addressed by the inclusion and scaffolding of open-ended and ill-defined problems in formative computing courses. An Analysis, Design, and Justification (ADJ) based framework was recently developed to enable this goal. The ADJ framework consists of learning outcomes, a type of problem, and a three-step problem-solving process. ADJ supports instruction in environments with constraints (such as large classes, and heterogeneous student ability) that are difficult to support with other approaches (such as Problem-Based Learning). In this paper, we report the experience and impact of leveraging the ADJ framework to develop real-world problem-solving skills in the context of a formative computing course. The ADJ framework was enacted in a class on operating-systems taught at the junior level. In the class, students were taught the concepts of the ADJ framework using a series of discussion activities, and then asked to apply it to generate solutions to three problem sets. The problem sets contained problems on topic areas covered by other assessments in the course but used open-ended and ill-defined problems. Assessments showed that students had different performance on existing (well-defined) problems as opposed to the real-world style problems of ADJ. The ADJ assessments provided a step towards better evaluation of problem-solving ability.

Knowledge of structure is critical to map legacy workflows to environments suitable to run on the cloud. We present a method which characterizes a workflow structure with the execution trace produced by instrumented logging functionality. The method generates the structure of workflows to support their reuse by permitting their transformation into modern execution environments. The method presented in the paper is implemented for Python workflows and demonstrated in the context of several legacy scientific workflows.

Scientific discovery relies on an experimental framework that corroborates hypotheses with experiments that are complex reproducible processes generating and transforming large datasets. The methods, implicit in the process, capture the semantics of the data, thus they are responsible for the generation of scientific information and discovery of scientific knowledge. Scientific workflows provide the semantics needed to wrap scientific data from their capture, analysis, publication, and archival. By annotating data with the processes that produce them, the scientist no longer manages data but information and allows their meaningful interpretation and integration. Any change to a scientific workflow may impact significantly the quality of the data produced, their semantics, their future analysis, use, integration, and distribution, as well as the performance of the execution. Yet, scientific workflows are typically transformed over time, updated with new versions of the tools that compose them, extended to new functionality, and composed. In this paper we discuss the various impacts of workflow transformation and illustrate them with a case study on the Structural Prediction for pRotein fOlding UTility System (SPROUTS) Workflow.