It is known that timely and personalized feedback is vital to the learning process, and because of increasing enrollment, instructors can find it harder to provide that feedback. Learning analytics presents a solution to this problem. The growth in popularity of online education systems better enables learning analytics by providing additional educational data. This work focuses on the analysis of students’ incorrect short answers and their pathways to correct solutions. By considering student submissions as sequences, this work uses a dimension called “distance” which can be used to predict how far off a student’s incorrect answer is from a correct one. This distance metric can be used for recognizing students who may need help, understanding which concepts students struggle with, evaluating assessment questions, and improving multiple-choice answers.This paper discusses the methods, relevant learning scenarios, and applications of the learning analytics system. It features the results and analysis of a usability test conducted on 56 faculty members.

Understanding the thought processes of students as they progress from initial (incorrect) answers toward correct answers is a challenge for instructors, both in this pandemic and beyond. This paper presents a general network visualization learning analytics system that helps instructors to view a sequence of answers input by students in a way that makes student learning progressions apparent. The system allows instructors to study individual and group learning at various levels of granularity. The paper illustrates how the visualization system is employed to analyze student responses collected through an intervention. The intervention is BeginToReason, an online tool that helps students learn and use symbolic reasoning-reasoning about code behavior through abstract values instead of concrete inputs. The specific focus is analysis of tool-collected student responses as they perform reasoning activities on code involving conditional statements. Student learning is analyzed using the visualization system and a post-test. Visual analytics highlights include instances where students producing one set of incorrect answers initially perform better than a different set and instances where student thought processes do not cluster well. Post-test data analysis provides a measure of student ability to apply what they have learned and their holistic understanding.

To develop code that meets its specification and is verifiably correct, such as in a software engineering course, students must be able to understand formal contracts and annotate their code with assertions such as loop invariants. To assist in developing suitable instructor and automated tool interventions, this research aims to go beyond simple pre- and post-conditions and gain insight into student learning of loop invariants involving objects. As students develop suitable loop invariants for given code with the aid of an online system backed by a verification engine, each student attempt, either correct or incorrect, was collected and analyzed automatically, and catalogued using an iterative process to capture common difficulties. Students were also asked to explain their thought process in arriving at their answer for each submission. The collected explanations were analyzed manually and found to be useful to assess their level of understanding as well as to extract actionable information for instructors and automated tutoring systems. Qualitative conclusions include the impact of the medium.

We present results in automating the analysis of student verbalizations in online learning environments, using an existing online tool designed to teach students to reason analytically about code as an example. The new extension captures "think-aloud'' data as students work through code reasoning activities. The data is recorded and transcribed automatically and used as input to a natural language processing / machine learning system designed to identify specific student attitudes (e.g., uncertain), behaviors (e.g., guessing), and difficulties (e.g., concept misunderstandings). We present the design and implementation of the tool, an analysis of its transcription accuracy, and an evaluation of its utility in identifying characteristics of student learning.

Several research studies have shown the benefits of code tracing to promote student understanding of program behavior. While code tracing on specific input values is a useful starting point, students ultimately need to be able to reason rigorously and logically about the correctness of their code on all (i.e., arbitrary) inputs. Otherwise, they may make false generalizations and may achieve only a shallow understanding. Results of a multi-semester experiment to answer the following research questions: (1) With or without steps, can students learn the basics of tracing code on symbolic input values using an online tool? And how important is classroom instruction? (2) What is the impact of motivation on student attitudes in learning to reason with such a tool? Data was obtained from 297 subjects who used the online reasoning tool in a second-year software development course for CS majors. Analysis indicates that students can do symbolic reasoning to trace code and that instruction and motivation have significant impact.

Using freely available online automated reasoning tools, we will demonstrate a sequence of engaging reasoning activities that are suitable to introduce beginning programmers and software engineering students to reason logically and symbolically about code. The automated tools have an underlying verification engine that makes it possible for the tool to offer activities and directed logical feedback not possible with typical development environments. The tools have been used in undergraduate classrooms for multiple years by well over a thousand students. The imperative language used by the tool is integrated with the underlying verification engine, and because it closely resembles many commercial languages, it presents little barrier to student usage. A comprehensive activity-based "Reason with Components" tool takes 5-10 minutes of instructor introduction and allows student exploration of contracts, objects, loops, recursion, and reusable concepts. Multiple versions of "Begin to Reason" tools are designed to help students learn the basics of code tracing in intro CS courses "on their own". Students and instructors can create new activities and can fine-tune the existing activities to their specific needs.

CS students can typically reason about what a piece of code does on specific inputs. While this is a useful starting point, graduates must also be able to logically analyze, comprehend, and predict the behavior of their code in more general terms, no matter what the inputs are. Results of data collection and analysis from an online educational system show it can help to pinpoint the difficulties in doing this for individual students and groups, and to partition the groups in terms of their difficulties so that instructional interventions may be better targeted. Unlike traditional debugging, this online system helps reveal difficulties in reasoning in more general terms because it is equipped with a verification engine.

Analytical reasoning is central to code correctness, and every computer science curriculum aims to teach students how to achieve this objective in one form or another. With the acceptance of object-based computing and component-based software engineering, the need for analytical reasoning that is based on formal contracts to establish correctness of software across module boundaries has become ever more obvious. Yet there are few institutions that have integrated modular, analytical reasoning principles into their undergraduate curriculum. Among many reasons for this shortcoming are: the effort it takes overloaded faculty to integrate new ideas of any kind in their courses, the challenge of institutionalizing ideas within a specific context, and constraints of a particular college. This paper presents our experiences over nearly two decades at five different institutions with the hope that they will serve as useful curriculum examples for like-minded educators at other institutions.

Computing students often have difficulty understanding the relevance of the math we teach, though educators appreciate the significance. This BoF will discuss ways to connect this math with what computing students think they should be doing: programming. This BoF will focus on the benefits (and perils) of connecting math to the development of correct programs with the goal of motivating the relevance of the math-related portion of the ACM/IEEE Computer Society CS2013 curriculum. The discussion will continue the spirit and essence held by the math-thinking working group, a distributed working group of approximately 170 people who have been promoting and clarifying the importance of mathematics in computer science education.

Undergraduate computer science students need to learn analytical reasoning skills to develop high-quality software and to understand why the software they develop works as specified. To accomplish this central educational objective, this article describes a systematic process of introducing reasoning skills into the curriculum and assessing how well students have learned those skills. To facilitate assessment, a comprehensive inventory of principles for reasoning about correctness that captures the finer details of basic skills that students need to learn has been defined and used. The principles can be taught at various levels of depth across the curriculum in a variety of courses. The use of a particular instructional process is illustrated to inculcate reasoning principles across several iterations of a sophomore-level development foundations course and a junior-level software engineering course. The article summarizes how learning outcomes motivated by the inventory of reasoning principles lead to questions that in turn form the basis for a careful analysis of student understanding and for fine-tuning teaching interventions that together facilitate continuous improvements to instruction.

SIGCSE has for a long time nourished an audience excited about teaching mathematical reasoning principles across the curriculum through the Math Thinking Birds-of-a-Feather session and panels on mathematical reasoning. While these forums are useful for discussing reasoning topics, they do not provide a consistent venue for sharing math-reasoning activities to be used in the classroom. Therefore, SIGCSE attendees interested in math thinking have routinely wished for a place for discussing engaging math reasoning examples and assignments. Providing such a forum is the purpose of this session. The exercises and assignments will help faculty find ways to incorporate mathematical reasoning in CS1, CS2, data structures and algorithms, discrete math, and software engineering courses.

A key goal of our research is to introduce an approach that involves at the outset using analytical reasoning as a method for developing high quality software. This paper summarizes our experiences in introducing mathematical reasoning and formal specification-based development using a web-integrated environment in an undergraduate software engineering course at two institutions at different levels, with the goal that they will serve as models for other educators. At Alabama, the reasoning topics are introduced over a two-week period and are followed by a project. At Clemson, the topics are covered in more depth over a five-week period and are followed by specification-based software development and reasoning assignments. The courses and project assignments have been offered for multiple semesters. Evaluation of student performance indicates that the learning goals were met.

Is it possible to excite students about learning the mathematical principles that underlie high-quality software? Can they use a development environment for "hands-on" experimentation with reasoning? Is this possible without displacing existing content? The answer is a resounding yes "from the experiences of professors at several institutions" but it takes the right set of pedagogical principles, reasoning tools, and hands-on exercises. This laboratory will help educators transfer the excitement of learning how to apply mathematical reasoning in building high quality software, by adopting one reasoning concept at a time.

No abstract available.

The need for undergraduate CS students to create and understand mathematical abstractions is clear, yet these skills are rarely taught in a systematic manner, if they are taught at all. This paper presents a systematic approach to teaching abstraction using rigorous mathematical models and a web-based reasoning environment. It contains a series of representative examples with varying levels of sophistication to make it possible to teach the ideas in a variety of courses, such as introductory programming, data structures, and software engineering. We also present results from our experimentation with these ideas over a 3-year period at our institution in a required course that introduces object-based software development, following CS2.

Is it possible to excite students about learning the mathematical principles that underlie high-quality software? Can we teach them to apply these principles using modern software tools? Can this be accomplished without displacing existing content? In each case, the answer is a resounding yes - but it takes the right set of pedagogical principles, teaching tools, and classroom exercises. This hands-on laboratory will introduce a set of principles, tools, and exercises that have proven to work. By adopting one content module at a time, educators will better prepare students to reason rigorously about the software they develop and maintain.

No abstract available.

This paper is intended for those instructors asked to teach an undergraduate introductory class on SQL query writing, which is usually difficult for students to learn. After reading Bruer's Schools for Thought [1], we realized that we need to modify our approach so that it uses informed instruction (instead of traditional instruction) and since have noticed improvements in student performance in SQL query writing.

A model and a taxonomy to characterize pointer manipulations are introduced, along with an instrumentation technology that leverages them to provide students with immediate reports of pointer errors in C++ programs. Data collected from CS2 student assignments show that the vast majority of student pointer errors either would not have been noticed at all, or would have been detected only much later in execution, if this infrastructure were not used. Possible applications of the underlying technology--both to conduct long-term educational research into students' understanding of pointers, and to improve pedagogy directly--are discussed.

Two kinds of interface contract violations can occur in component-based software: A client component can fail to satisfy a requirement of a component it is using, or a component implementation can fail to fulfill its obligations to the client. The traditional approach to detecting and reporting such violations is to embed assertion checks into component source code, with compile-time control over whether they are enabled. This works well for the original component developers, but it fails to meet the needs of component clients who do not have access to source code for such components. A wrapper-based approach, in which contract checking is not hard-coded into the underlying component but is "layered" on top of it, offers several relative advantages. It is practical and effective for C++ classes. Checking code can be distributed in binary form along with the underlying component, it can be installed or removed without requiring recompilation of either the underlying component or the client code, it can be selectively enabled or disabled by the component client on a per-component basis, and it does not require the client to have access to any special tools (which might have been used by the component developer) to support wrapper installation and control. Experimental evidence indicates that wrappers in C++ impose-modest additional overhead compared to inlining assertion checks.

Software components have existed in one form or another for a number of years. Work in this area can be classified into two broad categories. On the one hand, a number of researchers have approached the concept of software components from a first principles perspective, advancing ideas regarding what constitutes the ideal component paradigm from perspectives of efficiency, verifiability and reusability. On the other hand, recent commercial advances in a number of popular technologies have elevated the software component concept into widespread use within the software practitioner community. Such technologies include a number of technologies made popular by Microsoft (such as Active-X, COM, DCOM and Visual Basic), as well as CORBA and Java Beans.Neither of these perspectives on software components has become a standard cornerstone of software development pedagogy. Yet both perspectives may have an important role in preparing software developers to build high-quality software in the context of modern software development technologies. In particular, teaching students how to design and construct software components from first principles provides students with important guidance as to the "right way" to structure correct and efficient software systems (i.e., with emphasis on "what" component-based systems should contain). On the other hand, teaching students about current commercial component technologies exposes students to the important dimension of best commercial practice (i.e., with emphasis on "how" component-based systems could be built).The participants of this panel are all actively involved in the development of courses and curricula that provide various perspectives on component-based systems. They represent both the first principles and commercial perspectives discussed above. Position statements for each of the panelists appear below.

Pointer errors are stumbling blocks for student and veteran programmers alike. Although languages such as Java use references to protect programmers from pointer pitfalls, the use of garbage collection dictates that languages like C++ will still be used for real-time mission-critical applications. Pointers will stay in the classroom as long as they're used in industry, so as educators, we must find better ways to teach them. This paper presents checked pointers, a simple wrapper for C++ pointers that prevents pointer arithmetic and other common sources of pointer errors, and detects all dereferencing and deallocation errors, including memory leaks. The syntax of checked pointers is highly faithful to raw C++ pointers, but provides run-time error detection and debugging information. After debugging, changing one #include is all that is required to substitute a non-checking implementation that is as fast as raw C++.

Academic research sometimes suffers from the “ivory tower” problem: ideas that sound good in theory do not necessarily work well in practice. An example of research that potentially could impact practice over the next few years is a novel set of component-based software engineering design principles, known as the RESOLVE discipline. This discipline has been taught to students for several years [23], and previous papers (e.g., [24]) have reported on student-sized software projects constructed using it. Here, we report on a substantial commercial product family that was engineered using the same principles — an application that we designed, built, and continue to maintain for profit, not as part of a research project. We discuss the impact of adhering to a very prescriptive set of design principles and explain our experience with the resulting applications. Lessons learned should benefit others who might be considering adopting such a component-based software engineering discipline in the future.

Two kinds of interface contract violations can occur in component based software: a client component may fail to satisfy a requirement of a component it is using, or a component implementation may fail to fulfil its obligations to the client. The paper proposes a systematic approach for detecting both kinds of violations, so that violation detection is not hard coded into base level components, but is "layered" on top of them, and so that it can be turned "on" or "off" selectively for one or more components, with practically no change to executable code (limiting changes to a few declarations). Among the salient features of this approach are its use of formal specifications, the ability to handle parameterized (i.e., generic, or template) components, and the automatic generation of routine aspects of violation detection. We have designed, built, and experimented with a generator of checking components for C++ templates.

First-year computer science students need to see clearly that computer science as a discipline has an important intellectual role to play and that it offers deep philosophical questions, much like the other hard sciences and mathematics; that CS is not "just programming". An appropriate intellectual focus for CS1/CS2 can be built on the foundations of systems thinking and mathematical modeling, as these principles are manifested in a component-based software paradigm. We outline some of the main technical features of this approach to CS1/CS2 and report preliminary observations from our experience with it.

Reverse engineering of large legacy software systems generally cannot meet its objectives because it cannot be cost-effective.There are two main reasons for this.First, it is very costly to "understand" legacy code sufficiently well to permit changes to be made safely, because reverse engineering of legacy code is intractable in the usual computational complexity sense.Second, even if legacy code could be cost-effectively reverse engineered, the ultimate objective -re-engineering code to create a system that will not need to be reverse engineered again in the future -is presently unattainable.Not just crusty old systems, but even ones engineered today, from scratch, cannot escape the clutches of intractability until software engineers learn to design systems that support modular reasoning about their behavior.We hope these observations serve as a wake-up call to those who dream of developing high-quality software systems by transforming them from defective raw materials.

Software engineers often espouse the importance of using abstraction and encapsulation in developing software components. They advocate the "layering" of new components on top of existing components, using only information about the functionality and interfaces provided by the existing components. This layering approach is in contrast to a "direct implementation" of new components, utilizing unencapsulated access to the representation data structures and code present in the existing components. By increasing the reuse of existing components, the layering approach intuitively should result in reduced development costs, and in increased quality for the new components. However, there is no empirical evidence that indicates whether the layering approach improves developer productivity or component quality. We discuss three controlled experiments designed to gather such empirical evidence. The results support the contention that layering significantly reduces the effort required to build new components. Furthermore, the quality of the components, in terms of the number of defects introduced during their development, is at least as good using the layered approach. Experiments such as these illustrate a number of interesting and important issues in statistical analysis. We discuss these issues because, in our experience, they are not well known to software engineers.>