In this Innovative Practice Full Paper, we note that the way software is developed has changed significantly in the past 50 years. Software developers today cannot just be good at writing code; they must also possess non-technical skills in order to work successfully within diverse teams and have an appreciation for the tools and processes needed to build and maintain complex systems. In this work, we describe a novel first-year Software Engineering-First (SE-First) curriculum that introduces students to the broader picture of software development while students learn fundamental computing concepts. To assess the effectiveness of our novel first-year curriculum, we compare students who completed the first-year software engineering curriculum with students who completed our traditional computer science curriculum. We assess student knowledge of computing concepts, and their self-efficacy. Initial results show that students who complete the first-year software engineering courses perform as well or better on the computing concepts test, they are more confident in their computing abilities and in the application of computing skills to their field, and they have a higher success rate in their first-year computing courses (i.e., fewer students drop the course and fewer students receive a D or F course grade) compared with students who complete the traditional first-year computing program.

Software analysis tools and techniques often leverage structural code coverage information to reason about the dynamic behavior of software. Existing techniques instrument the code with the required structural obligations and then monitor the execution of the compiled code to report coverage. Instrumentation based approaches often incur considerable runtime overhead for complex structural coverage metrics such as Modified Condition/Decision (MC/DC). Code instrumentation, in general, has to be approached with great care to ensure it does not modify the behavior of the original code. Furthermore, instrumented code cannot be used in conjunction with other analyses that reason about the structure and semantics of the code under test. In this work, we introduce a non-intrusive preprocessing approach for computing structural coverage information. It uses a static partial evaluation of the decisions in the source code and a source-to-bytecode mapping to generate the information necessary to efficiently track structural coverage metrics during execution. Our technique is flexible; the results of the preprocessing can be used by a variety of coverage-driven software analysis tasks, including automated analyses that are not possible for instrumented code. Experimental results in the context of symbolic execution show the efficiency and flexibility of our non- intrusive approach for computing code coverage information.

Tests are increasingly specified as programs. Expressing tests as code is advantageous in that developers are comfortable writing and running code, and tests can be automated and reused as the software evolves. Tests expressed as code, however, can also contain faults. Some test faults are similar to those found in application code, while others are more subtle, caused by incorrect implementation of testing concepts and processes. These faults may cause a test to fail when it should not, or allow program faults to go undetected. In this work we explore whether lightweight static analyses can be cost-effective in pinpointing patterns associated with faults tests. Our exploration includes a categorization and explanation of test patterns, and their application to 12 open source projects that include over 40K tests. We found that several patterns, detectable through simple and efficient static analyses of just the test code, can detect faults with a low false positive rate, while other patterns would require a more sophisticated and extensive code analysis to be useful.

This paper introduces iProperty, a novel approach that facilitates incremental checking of programs based on a property differencing technique. Specifically, iProperty aims to reduce the cost of checking properties as they are initially developed and as they co-evolve with the program. The key novelty of iProperty is to compute the differences between the new and old versions of expected properties to reduce the number and size of the properties that need to be checked during the initial development of the properties. Furthermore, property differencing is used in synergy with program behavior differencing techniques to optimize common regression scenarios, such as detecting regression errors or checking feature additions for conformance to new expected properties. Experimental results in the context of symbolic execution of Java programs annotated with properties written as assertions show the effectiveness of iProperty in utilizing change information to enable more efficient checking.

Program invariants can help software developers identify program properties that must be preserved as the software evolves, however, formulating correct invariants can be challenging. In this work, we introduce iDiscovery, a technique which leverages symbolic execution to improve the quality of dynamically discovered invariants computed by Daikon. Candidate invariants generated by Daikon are synthesized into assertions and instrumented onto the program. The instrumented code is executed symbolically to generate new test cases that are fed back to Daikon to help further refine the set of candidate invariants. This feedback loop is executed until a fix-point is reached. To mitigate the cost of symbolic execution, we present optimizations to prune the symbolic state space and to reduce the complexity of the generated path conditions. We also leverage recent advances in constraint solution reuse techniques to avoid computing results for the same constraints across iterations. Experimental results show that iDiscovery converges to a set of higher quality invariants compared to the initial set of candidate invariants in a small number of iterations.

The last few years have seen a resurgence of interest in the use of symbolic execution—a program analysis technique developed more than three decades ago to analyze program execution paths. Scaling symbolic execution to real systems remains challenging despite recent algorithmic and technological advances. An effective approach to address scalability is to reduce the scope of the analysis. For example, in regression analysis, differences between two related program versions are used to guide the analysis. While such an approach is intuitive, finding efficient and precise ways to identify program differences, and characterize their impact on how the program executes has proved challenging in practice. In this article, we present Directed Incremental Symbolic Execution (DiSE), a novel technique for detecting and characterizing the impact of program changes to scale symbolic execution. The novelty of DiSE is to combine the efficiencies of static analysis techniques to compute program difference information with the precision of symbolic execution to explore program execution paths and generate path conditions affected by the differences. DiSE complements other reduction and bounding techniques for improving symbolic execution. Furthermore, DiSE does not require analysis results to be carried forward as the software evolves—only the source code for two related program versions is required. An experimental evaluation using our implementation of DiSE illustrates its effectiveness at detecting and characterizing the effects of program changes.

Abstract—Adding similar features and bug fixes often requires porting program patches from reference implementations and adapting them to target implementations. Porting errors may result from faulty adaptations or inconsistent updates. This paper investigates (1) the types of porting errors found in practice, and (2) how to detect and characterize potential porting errors. Analyzing version histories, we define five categories of porting errors, including incorrect control- and data-flow, code redun-dancy, inconsistent identifier renamings, etc. Leveraging this categorization, we design a static control- and data-dependence analysis technique, SPA, to detect and characterize porting inconsistencies. Our evaluation on code from four open-source projects shows that SPA can detect porting inconsistencies with 65 % to 73 % precision and 90 % recall, and identify inconsistency

Change impact analysis techniques estimate the potential effects of changes made to software. Directed Incremental Symbolic Execution (DiSE) is an intraprocedural technique for characterizing the impact of software changes on program behaviors. DiSE first estimates the impact of the changes on the source code using program slicing techniques, and then uses the impact sets to guide symbolic execution to generate path conditions that characterize impacted program behaviors. DiSE, however, cannot reason about the flow of impact between methods and will fail to generate path conditions for certain impacted program behaviors. In this work, we present iDiSE, an extension to DiSE that performs an interprocedural analysis. iDiSE combines static and dynamic calling context information to efficiently generate impacted program behaviors across calling contexts. Information about impacted program behaviors is useful for testing, verification, and debugging of evolving programs. We present a case-study of our implementation of the iDiSE algorithm to demonstrate its efficiency at computing impacted program behaviors. Traditional notions of coverage are insufficient for characterizing the testing efforts used to validate evolving program behaviors because they do not take into account the impact of changes to the code. In this work we present novel definitions of impacted coverage metrics that are useful for evaluating the testing effort required to test evolving programs. We then describe how the notions of impacted coverage can be used to configure techniques such as DiSE and iDiSE in order to support regression testing related tasks. We also discuss how DiSE and iDiSE can be configured for debugging; finding the root cause of errors introduced by changes made to the code. In our empirical evaluation we demonstrate that the configurations of DiSE and iDiSE can be used to support various software maintenance tasks.

We describe an approach to testing complex safety critical software that combines unit-level symbolic execution and system-level concrete execution for generating test cases that satisfy user-specified testing criteria. We have developed Symbolic Java PathFinder, a symbolic execution framework that implements a non-standard bytecode interpreter on top of the Java PathFinder model checking tool. The framework propagates the symbolic information via attributes associated with the program data. Furthermore, we use two techniques that leverage system-level concrete program executions to gather information about a unit's input to improve the precision of the unit-level test case generation. We applied our approach to testing a prototype NASA flight software component. Our analysis helped discover a serious bug that resulted in design changes to the software. Although we give our presentation in the context of a NASA project, we believe that our work is relevant for other critical systems that require thorough testing.

Detecting and characterizing the effects of software changes is a fundamental component of software maintenance. Version differencing information can be used to perform version merging, infer change characteristics, produce program documentation, and guide program re-validation. Existing techniques for characterizing code changes, however, are imprecise leading to unnecessary maintenance efforts.

Model checkers search the space of possible program behaviors to detect errors and to demonstrate their absence. Despite major advances in reduction and optimization techniques, state-space search can still become cost-prohibitive as program size and complexity increase. In this paper, we present a technique for dramatically improving the cost- effectiveness of state-space search techniques for error detection using parallelism. Our approach can be composed with all of the reduction and optimization techniques we are aware of to amplify their benefits. It was developed based on insights gained from performing a large empirical study of the cost-effectiveness of randomization techniques in state-space analysis. We explain those insights and our technique, and then show through a focused empirical study that our technique speeds up analysis by factors ranging from 2 to over 1000 as compared to traditional modes of state-space search, and does so with relatively small numbers of parallel processors.

Software testing efforts account for a large part of software development costs. However, as educators, we struggle to properly prepare students to perform software testing activities. This struggle is caused by multiple factors: (1) it is challenging to effectively incorporate software testing into an already over-packed curriculum, (2) ad-hoc efforts to teach testing generally happen too late in the students' career, after bad habits have already been developed, and (3) these efforts lack the necessary institutional consistency and support to be effective. To address these challenges we created Bug Hunt, a web-based tutorial to engage students in learning software testing strategies. In this paper we describe the most interesting aspects of the tutorial including the lessons and feedback mechanisms, and the facilities for instructors to configure the tutorial and obtain automatic student assessment. We also present the lessons learned after two years of deployment.

Recent advances in static program analysis have made it possible to detect errors in applications that have been thoroughly tested and are in wide-spread use. The ability to find errors that have eluded traditional validation methods is due to the development and combination of sophisticated algorithmic techniques that are embedded in the implementations of analysis tools. Evaluating new analysis techniques is typically performed by running an analysis tool on a collection of subject programs, perhaps enabling and disabling a given technique in different runs. While seemingly sensible, this approach runs the risk of attributing improvements in the cost-effectiveness of the analysis to the technique under consideration, when those improvements may actually be due to details of analysis tool implementations that are uncontrolled during evaluation.In this paper, we focus on the specific class of path-sensitive error detection techniques and identify several factors that can significantly influence the cost of analysis. We show, through careful empirical studies, that the influence of these factors is sufficiently large that, if left uncontrolled, they may lead researchers to improperly attribute improvements in analysis cost and effectiveness. We make several recommendations as to how the influence of these factors can be mitigated when evaluating techniques.

A learning object is a structured, standalone media resource that encapsulates high quality information to facilitate learning and pedagogy. In this paper, we describe our approach to design, develop, and validate learning objects for CS1. In particular, we focus on one learning object that teaches students about classes and objects. SCORM (Shareable Content Object Reference Model) standards and ACM/IEEE-CS Computing Curriculum 2001 form the basis of our design. Each learning object is self-contained and by design, the length of the content section is kept short to retain student interest. The learning object has a glossary providing definitions to key terms and a help menu. Each learning object covers a core Computer Science topic addressed by four components: (1) A brief tutorial or explanation including definitions, rules, and principles, (2) A set of real-world examples illustrates key concepts and includes worked examples and problems, models, and sample code, (3) A set of practice exercises provides important active experiences to the student, with constructive feedback to student responses, (4) A set of problems graded by the computer provides a final assessment. Our instructional design also incorporates theories of multimedia learning, providing guidance on the effective combination of text, graphics audio, and Flash animation. We also report on a pilot evaluation where students rated the learning object highly in terms of its design, usefulness, and appropriateness. We present student achievement results, comparing achievement of students participating in traditional face-to-face laboratory activities versus students using the Web-based learning object. A between-group post-test only research design showed no significant achievement difference between the two groups. Results confirm our belief that the use of modular, Web-based learning objects can be used successfully for independent learning and are a viable option for distance delivery of course components. Encouraged by these results, our project and research is continuing Fall 2004, with the development of additional learning objects and instrumentation mechanisms tracking real-time dynamic activity-based data.The "Practice Exercises" section of our "Simple Class" learning object, for example, has four exercise modules: (1) class identification, where students are asked to identify whether an item is an appropriate candidate as a class (Abraham Lincoln vs. President, for example), (2) data members and methods, where students interact with an animation (with sound) to identify the appropriate data members for a dog class, (3) dissect a class definition, where students are given code with highlighted segments and are asked to label each segment into either "class", "method name", "data member", or "method body", and (4) building a class, where students are given a heterogeneous set of data members and methods, and must pick the appropriate ones to build a class; if the selection is correct, the Java-based class will be expanded accordingly with specific Java code. For each exercise, we provide extensive real-time feedback for each response. Figure 1 shows a screen shot of one of the exercises on data members and methods.

Closed laboratories are becoming an increasingly popular approach to teaching introductory computer science courses. However, as observed in [1], “Considering the prevalence of closed labs and the fact that they have been in place in CS curricula for more than a decade, there is little published evidence assessing their effectiveness. ” In this paper, we report on how students’ performance in closed laboratories relates to their performances on a placement exam, homework assignments, course exams, and how it relates to their self-reported attitudes towards our CS1 course. This analysis provides insights to help us improve the design of our laboratories as well as other components of CS1.

An introductory CS1 course presents problems for educators and students due to students' diverse background in programming knowledge and exposure. Students who enroll in CS1 also have different expectations and motivations. Prompted by the curricular guidelines for undergraduate programs in computer science released in 2001 by the ACM/IEEE, and driven by a departmental project to reinvent the undergraduate computer science and computer engineering curricula at the University of Nebraska-Lincoln, we are currently implementing a series of changes which will improve our introductory courses. One key component of our project is an online placement examination tied to the cognitive domain that assesses student knowledge and intellectual skills. Our placement test is also integrated into a comprehensive educational research design containing a pre- and post-test framework for assessing student learning. In this paper, we focus on the design and implementation of our placement exam and present an analysis of the data collected to date.

In this paper, we discuss an integrated framework of case-based learning (CBL) in an agent that intelligently delivers learning materials to students. The agent customizes its delivery strategy for each student based on the student's background profile and his or her interactions with the graphic user interface (GUI) to our system, and based on the usage history of the learning materials. The agent's decision-making process is powered by case-based reasoning (CBR). To improve its reasoning process, our agent learns the differences between good cases (cases with a good solution for its problem space) and bad cases (cases with a bad solution for its problem space). It also meta-learns adaptation heuristics, the significance of input features of the cases, and the weights of a content graph for symbolic feature values. We have also built a simulation to comprehensively test the learning behavior of our agent.