Abstract To explain specific phenomena, scientists perform a sequence of tasks, e.g., to gather, analyze and interpret data, forming a scientific workflow. Depending on the complexity of the workflow, scientists require access to various kinds of tools, applications and infrastructures for individual tasks. Current approaches are often limited to managing these resources at design time, requiring the scientist to preemptively set up applications essential for their workflow. Therefore, a dynamic provisioning and configuration of computing resources are required that fulfills these needs at runtime. In this paper, we present a dynamic runtime model that couples workflow tasks with their individual applications and infrastructure requirements. This runtime model is used as a knowledge base by a model-driven workflow execution engine orchestrating the sequence of tasks and their infrastructure. We exhibit that the simplicity of the runtime model supports the creation of highly tailored infrastructures, the integration of self-developed applications, as well as a human-in-the-loop allowing scientists to monitor and interact with the workflow at runtime. To tackle the heterogeneity of cloud provider interfaces, we implement the workflow runtime model by extending the Open Cloud Computing Interface cloud standard, which provides an extensible data model as well as a uniform interface to manage cloud resources. We demonstrate the applicability of our approach using three case studies and discuss the benefits of the runtime model from a user and system perspective.

Abstract The competent programmer hypothesis is one of the fundamental assumptions of mutation testing, which claims that most programmers are competent enough to create correct or almost correct source code. This implies that faults should usually manifest through small variations of the correct code. Consequently, researchers assumed that the synthetic faults injected in source code through the mutation operators closely resemble the real faults. Unfortunately, it is still unclear whether the competent programmer hypothesis holds, as past research presents contradictory claims. Within this article, we provide a new perspective on the competent programmer hypothesis and its relation to mutation testing. We try to re‐create real‐world faults through chains of mutations to understand if there is a direct link between mutation testing and faults. The lengths of these paths help us to understand if the source code is really almost correct, or if large variations are required. Our experiments used a state‐of‐the‐art benchmark database of real faults named Defects4J 2.0.0. It contains 835 reproducible real‐world faults in 17 open‐source projects that comprise a total of 1044 bug‐fix pairs of files. Our results indicate that while the competent programmer hypothesis seems to be true, mutation testing is missing important operators to generate representative real‐world faults.

Abstract Many software metrics are designed to measure aspects that are believed to be related to software quality. Static software metrics, e.g., size, complexity and coupling are used in defect prediction research as well as software quality models to evaluate software quality. Static analysis tools also include boundary values for complexity and size that generate warnings for developers. While this indicates a relationship between quality and software metrics, the extent of it is not well understood. Moreover, recent studies found that complexity metrics may be unreliable indicators for understandability of the source code. To explore this relationship, we leverage the intent of developers about what constitutes a quality improvement in their own code base. We manually classify a randomized sample of 2,533 commits from 54 Java open source projects as quality improving depending on the intent of the developer by inspecting the commit message. We distinguish between perfective and corrective maintenance via predefined guidelines and use this data as ground truth for the fine-tuning of a state-of-the art deep learning model for natural language processing. The benchmark we provide with our ground truth indicates that the deep learning model can be confidently used for commit intent classification. We use the model to increase our data set to 125,482 commits. Based on the resulting data set, we investigate the differences in size and 14 static source code metrics between changes that increase quality, as indicated by the developer, and changes unrelated to quality. In addition, we investigate which files are targets of quality improvements. We find that quality improving commits are smaller than non-quality improving commits. Perfective changes have a positive impact on static source code metrics while corrective changes do tend to add complexity. Furthermore, we find that files which are the target of perfective maintenance already have a lower median complexity than files which are the target of non-pervective changes. Our study results provide empirical evidence for which static source code metrics capture quality improvement from the developers point of view. This has implications for program understanding as well as code smell detection and recommender systems.

Abstract Automated Static Analysis Tools (ASATs) are part of software development best practices. ASATs are able to warn developers about potential problems in the code. On the one hand, ASATs are based on best practices so there should be a noticeable effect on software quality. On the other hand, ASATs suffer from false positive warnings, which developers have to inspect and then ignore or mark as invalid. In this article, we ask whether ASATs have a measurable impact on external software quality, using the example of PMD for Java. We investigate the relationship between ASAT warnings emitted by PMD on defects per change and per file. Our case study includes data for the history of each file as well as the differences between changed files and the project in which they are contained. We investigate whether files that induce a defect have more static analysis warnings than the rest of the project. Moreover, we investigate the impact of two different sets of ASAT rules. We find that, bug inducing files contain less static analysis warnings than other files of the project at that point in time. However, this can be explained by the overall decreasing warning density. When compared with all other changes, we find a statistically significant difference in one metric for all rules and two metrics for a subset of rules. However, the effect size is negligible in all cases, showing that the actual difference in warning density between bug inducing changes and other changes is small at best.

Abstract To guide software development, the estimation of the impact of decision making on the development process can be helpful in planning. For this estimation, often prediction models are used which can be learned from project data. In this paper, an approach for the usage of agent-based simulation for the prediction of software evolution trends is presented. The specialty of the proposed approach lies in the automated parameter estimation for the instantiation of project-specific simulation models. We want to assess how well a baseline model using average (commit) behavior of the agents (i.e., the developers) performs compared to models where different amount of project-specific data is fed into the simulation model. The approach involves the interplay between the mining framework and simulation framework. Parameters to be estimated include, e.g., file change probabilities of developers and the team constellation reflecting different developer roles. The structural evolution of software projects is observed using change coupling graphs based on common file changes. For the validation of simulation results, we compare empirical with simulated results. Our results showed that an average simulation model can mimic general project growth trends like the number of commits and files well and thus, can help project managers in, e.g., controlling the onboarding of developers. Besides, the simulated co-change evolution could be improved significantly using project-specific data.

Software quality evolution and predictive models to support decisions about resource distribution in software quality assurance tasks are an important part of software engineering research. Recently, a fine-grained just-in-time defect prediction approach was proposed which has the ability to find bug-inducing files within changes instead of only complete changes. In this work, we utilize this approach and improve it in multiple places: data collection, labeling and features. We include manually validated issue types, an improved SZZ algorithm which discards comments, whitespaces and refactorings. Additionally, we include static source code metrics as well as static analysis warnings and warning density derived metrics as features. To assess whether we can save cost we incorporate a specialized defect prediction cost model. To evaluate our proposed improvements of the fine-grained just-in-time defect prediction approach we conduct a case study that encompasses 38 Java projects, 492,241 file changes in 73,598 commits and spans 15 years. We find that static source code metrics and static analysis warnings are correlated with bugs and that they can improve the quality and cost saving potential of just-in-time defect prediction models.

Knowledge about students' programming errors is a valuable source to get insights into students deficiencies and misconceptions. In this paper, we use data from an introductory C programming course to identify which errors are often made by students. Previous studies often focused only on syntactic and semantic errors as they can be easily identified by compilers. Studies focusing on logic errors were often restricted to a limited set of concepts or performed for a small set of data. We manually inspect 12371 submission by 280 students and have no restrictions regarding the error types we are looking for. We classify our found errors into six categories: syntactic, conceptual, strategic, sloppiness, misinterpretation, and domain knowledge. Our results show that a big portion of errors made by students is simply caused by sloppiness. But putting sloppiness aside, students seem to have most problems with strategic knowledge, i.e., the problem solving ability. We compare our results to previous studies and provide some implications of our results for future teaching practice.

Abstract Automated static analysis tools (ASATs) have become a major part of the software development workflow. Acting on the generated warnings, i.e., changing the code indicated in the warning, should be part of, at latest, the code review phase. Despite this being a best practice in software development, there is still a lack of empirical research regarding the usage of ASATs in the wild. In this work, we want to study ASAT warning trends in software via the example of PMD as an ASAT and its usage in open source projects. We analyzed the commit history of 54 projects (with 112,266 commits in total), taking into account 193 PMD rules and 61 PMD releases. We investigate trends of ASAT warnings over up to 17 years for the selected study subjects regarding changes of warning types, short and long term impact of ASAT use, and changes in warning severities. We found that large global changes in ASAT warnings are mostly due to coding style changes regarding braces and naming conventions. We also found that, surprisingly, the influence of the presence of PMD in the build process of the project on warning removal trends for the number of warnings per lines of code is small and not statistically significant. Regardless, if we consider defect density as a proxy for external quality, we see a positive effect if PMD is present in the build configuration of our study subjects.

Unfortunately, the article “A Comparative Study to Benchmark Cross-project Defect Prediction Approaches” has a problem in the statistical analysis which was pointed out almost immediately after the pre-print of the article appeared online. While the problem does not negate the contribution of the the article and all key findings remain the same, it does alter some rankings of approaches used in the study. Within this correction, we will explain the problem, how we resolved it, and present the updated results.

To tackle the cloud-provider lock-in, the Open Grid Forum (OGF) is developing the Open Cloud Computing Interface (OCCI), a standardized interface for managing any kind of cloud resources. Besides the OCCI Core model, which defines the basic modeling elements for cloud resources, the OGF also defines extensions that reflect the requirements of different cloud service levels, such as IaaS and PaaS. However, so far the OCCI PaaS extension is very coarse grained and lacks of supporting use cases and implementations. Especially, it does not define how the components of the application itself can be managed. In this paper, we present a model-driven framework that extends the OCCI PaaS extension and is able to use different configuration management tools to manage the whole lifecycle of cloud applications. We demonstrate the feasibility of the approach by presenting four different use cases and prototypical implementations for three different configuration management tools.

Cross-Project Defect Prediction (CPDP) as a means to focus quality assurance of software projects was under heavy investigation in recent years. However, within the current state-of-the-art it is unclear which of the many proposals performs best due to a lack of replication of results and diverse experiment setups that utilize different performance metrics and are based on different underlying data. Within this article [2, 3], we provide a benchmark for CPDP. Our benchmark replicates 24 CPDP approaches proposed by researchers between 2008 and 2015. Through our benchmark, we answer the following research questions:

Cross-Project Defect Prediction (CPDP) as a means to focus quality assurance of software projects was under heavy investigation in recent years. However, within the current state-of-the-art it is unclear which of the many proposals performs best due to a lack of replication of results and diverse experiment setups that utilize different performance metrics and are based on different underlying data. Within this article, we provide a benchmark for CPDP. We replicate 24 approaches proposed by researchers between 2008 and 2015 and evaluate their performance on software products from five different data sets. Based on our benchmark, we determined that an approach proposed by Camargo Cruz and Ochimizu (2009) based on data standardization performs best and is always ranked among the statistically significant best results for all metrics and data sets. Approaches proposed by Turhan et al. (2009), Menzies et al. (2011), and Watanabe et al. (2008) are also nearly always among the best results. Moreover, we determined that predictions only seldom achieve a high performance of 0.75 recall, precision, and accuracy. Thus, CPDP still has not reached a point where the performance of the results is sufficient for the application in practice.

Reliability is one of the key concerns of both cloud providers and consumers, who require accurate reliability evaluation methods to develop, deploy, and maintain cloud applications. However, few works assess the reliability of cloud applications considering deep dependencies in the deployment stack. To explore the impact of deep dependencies to the reliability assessment, in this paper, we propose a layered dependency graph-based reliability assessment method for cloud applications. By introducing the inner reliability of cloud components, and combining the deep dependencies between services and physical servers, the method can assess the reliability of all components as well as the application. We implement a framework for the method and compare the assessment results of our approach against existing methods. The results show that deep dependencies have few impacts on the accuracy while can improve the precision of reliability assessment methods because the failure rate of physical servers is much lower than software components.

Unit testing is an essential practice in Extreme Programming (XP) and Test-driven Development (TDD) and used in many software lifecycle models. Additionally, a lot of literature deals with this topic. Therefore, it can be expected that it is widely used among developers. Despite its importance, there is no empirical study which investigates, whether unit tests are used by developers in real life projects at all. This paper presents such a study, where we collected and analyzed data from over 70K revisions of 10 different Python projects. Based on two different definitions of unit testing, we calculated the actual number of unit tests and compared it with the expected number (as inferred from the intentions of the developers), had a look at the mocking behavior of developers, and at the evolution of the number of unit tests. Our main findings show, (i) that developers believe that they are developing more unit tests than they actually do, (ii) most projects have a very small amount of unit tests, (iii) developers make use of mocks, but these do not have a significant influence on the number of unit tests, (iv) four different patterns for the evolution of the number of unit tests could be detected, and (v) the used unit test definition has an influence on the results.

Fault prediction on high quality industry grade software often suffers from strong imbalanced class distribution due to a low bug rate. Previous work reports on low predictive performance, thus tuning parameters is required. As the State of the Art recommends sampling methods for imbalanced learning, we analyse effects when under- and oversampling the training data evaluated on seven different classification algorithms. Our results demonstrate settings to achieve higher performance values but the various classifiers are influenced in different ways. Furthermore, not all performance reports can be tuned at the same time.

Due to an increasing number of students, more and more learning institutions tend to use computer-supported learning tools like online learning platforms or intelligent tutoring systems. This has opened up the opportunity to collect a huge amount of students' data. Educational Data Mining (EDM) uses mining techniques to derive information from these data about students' knowledge, behavior and experience to improve education. In this paper, we present a framework for mining programming errors of computer science students by analyzing the students' solutions to a programming assignment. The framework serves as both, a computer aided assessment tool as well as an immediate feedback tool about the learning progress of the students for the educator.

The evolution of software projects is driven by developers who are in control of the developed artifacts and the quality of software projects depends on the work of participating developers. Thus, a simulation tool requires a suitable model of the commit behavior of different developer types. In this paper, we present an agent-based model for software processes containing the commit behavior for different developer types. The description of these types results from mining software repositories. Since relationships between software entities, e.g., files, classes, modules, axe represented as dependency graphs, simulation results can be assessed automatically by Conditional Random Fields (CRFs). By adjusting simulation parameters for one project we are able to give a quality trend of other projects similar in size and duration only by changing the effort and the size of other projects to simulate.

Research in software repository mining has grown considerably the last decade. Due to the data-driven nature of this venue of investigation, we identified several problems within the current state-of-the-art that pose a threat to the external validity of results. The heavy re-use of data sets in many studies may invalidate the results in case problems with the data itself are identified. Moreover, for many studies data and/or the implementations are not available, which hinders a replication of the results and, thereby, decreases the comparability between studies. Even if all information about the studies is available, the diversity of the used tooling can make their replication even then very hard. Within this paper, we discuss a potential solution to these problems through a cloud-based platform that integrates data collection and analytics. We created the prototype SmartSHARK that implements our approach. Using SmartSHARK, we collected data from several projects and created different analytic examples. Within this article, we present SmartSHARK and discuss our experiences regarding the use of SmartSHARK and the mentioned problems.

The engineering of cyber physical systems requires holistic simulation perspectives. To cope with the complexity of these systems, we aim to provide a simulation methodology that is efficient regarding model complexity. The required holistic perspective is reached on a coarse level, which is co-simulated with multiple detailed models of some areas of the system that are of particular interest to the investigated phenomena. Which areas are thus “zoomed in” is dynamic during a simulation run. To reflect this, the resulting Multi-Level-Simulation is deployed in a dynamic cloud environment, using the provided hardware resources in a cost-efficient manner.

While Service Oriented Architectures (SOAs) are for many parts deployed online, and today often in a cloud, the testing of the systems still happens mostly locally. In this paper, we want to present the MIDAS Testing as a Service (TaaS), a cloud platform for the testing of SOAs. We focus on the testing of whole SOA orchestrations, a complex task due to the number of potential service interactions and the increasing complexity with each service that joins an orchestration. Since traditional testing does not scale well with such a complex setup, we employ a Model-based Testing (MBT) approach based on the Unified Modeling Language (UML) and the UML Testing Profile (UTP) within MIDAS. Through this, we provide methods for functional testing, security testing, and usage-based testing of service orchestrations. Through harnessing the computational power of the cloud, MIDAS is able to generate and execute complex test scenarios which would be infeasible to run in a local environment.

Posting and answering questions on Stack Overflow (SO) is everyday business for many developers. We asked a group of developers what they expect to be true about questions and answers on SO. Most of their expectations were related to the likelihood of getting an answer or to voting behavior. From their comments, we formulated nine myths that they think are true about the platform. Then, we proceeded to use rather simple methods from statistics to check if these myths are supported by the data in the SO dump provided. Through our analysis, we determined that there is an effect for eight of the nine myths the developers believed in. However, for only four of the myths the effect size is large enough to actually make a difference. Hence, we could bust five myths the developers believed in.

The integration of cloud and grid infrastructures is still of current interest, because it provides a way for the scientific area to ensure sustainability of well engineered grid applications. The integration of well established grid infrastructures with cloud systems also fosters their complementary usage, simplified migration of applications, as well as efficient resource utilization. In this paper, we compare the layered conceptual grid model to the service model of clouds. Based on this comparison, we describe pragmatic possibilities to integrate cloud and grid systems. We analyze the connectivity options on the infrastructure level to gain access to both infrastructures using a unified client. In two case studies, we show the successful integration of the Amazon Web Services cloud with UNICORE~6 and the open source cloud Eucalyptus with Globus Toolkit~4. Based on these implementations, we discuss lessons learned.

In this article, we present a novel algorithmic method for the calculation of thresholds for a metric set. To this aim, machine learning and data mining techniques are utilized. We define a data-driven methodology that can be used for efficiency optimization of existing metric sets, for the simplification of complex classification models, and for the calculation of thresholds for a metric set in an environment where no metric set yet exists. The methodology is independent of the metric set and therefore also independent of any language, paradigm or abstraction level. In four case studies performed on large-scale open-source software metric sets for C functions, C+ +, C# methods and Java classes are optimized and the methodology is validated.

Manual reviews and inspections of software artefacts are time consuming and thus, automated analysis tools have been developed to support the quality assurance of software artefacts. Usually, software analysis tools are implemented for analysing only one specific language as target and for performing only one class of analyses. Furthermore, most software analysis tools support only common programming languages, but not those domain-specific languages that are used in a test process. As a solution, a framework for software analysis is presented that is based on a flexible, yet high-level facade layer that mediates between analysis rules and the underlying target software artefact; the analysis rules are specified using high-level XQuery expressions. Hence, further rules can be quickly added and new types of software artefacts can be analysed without needing to adapt the existing analysis rules. The applicability of this approach is demonstrated by examples from using this framework to calculate metrics and detect bad smells in Java source code, in UML models, and in test specifications written using the Testing and Test Control Notations (TTCN-3).

Carrier-grade networks of the future are currently being standardized and designed under the umbrella name of Next Generation Network (NGN). The goal of NGN is to provide a more flexible network infrastructure that supports not just data and voice traffic routing, but also higher level services and interfaces for third-party enhancements. Within this paper, opportunities to integrate grid and cloud computing strategies and standards into NGN are considered. The importance of standardized interfaces and interoperability testing demanded by carrier-grade networks are discussed. Finally, a proposal how the testing methods developed at the European Telecommunications Standards Institute (ETSI) can be applied to improve the quality of standards and implementations is presented.

The collective and coordinated usage of distributed resources for problem solution within dynamic virtual organizations can be realized with the grid computing technology. For distributing and solving a task, a grid application involves a complex workflow of dividing a task into smaller sub-tasks, scheduling and submitting jobs for solving those sub-tasks, and eventually collecting and combining the results of the sub-tasks into a final result. The quality assurance of grid applications is a challenge due to the highly distributed nature of the grid environment in which the grid application is deployed. This paper investigates the applicability of the testing and test control notation (TTCN-3) for testing the workflows of distributed grid applications. To this aim, a case study has been created that consists of a distributed grid application which includes a typical grid application workflow; as the main contribution, this case study contains a corresponding distributed TTCN-3 test suite that tests the correct execution of the grid application workflow. To demonstrate the adaptation of the abstract TTCN-3 test suite to a specific grid environment, corresponding reusable test adapters have been implemented for the grid middleware Globus Toolkit 4 (GT4). The realized test system demonstrates that TTCN-3 is applicable for testing the workflow of distributed grid applications.

Experience with the development and maintenance of large test suites specified using the Testing and Test Control Notation (TTCN-3) has shown that it is difficult to construct tests that are concise with respect to quality aspects such as maintainability or usability. The ISO/IEC standard 9126 defines a general software quality model that substantiates the term “quality” with characteristics and subcharacteristics. The domain of test specifications, however, requires an adaption of this general model. To apply it to specific languages such as TTCN-3, it needs to be instantiated. In this paper, we present an instantiation of this model as well as an approach to assess and improve test specifications. The assessment is based on metrics and the identification of code smells. The quality improvement is based on refactoring. Example measurements using our TTCN-3 tool TRex demonstrate how this procedure is applied in practise.

Abstract Comprehensive testing of modern communication systems often requires large and complex test suites, which have to be maintained throughout the system life cycle. Industrial experience, with those written using the standardized Testing and Test Control Notation (TTCN‐3), has shown that this maintenance is a non‐trivial task and its burden can be reduced by means of appropriate concepts and tool support. To this aim, Motorola has collaborated with the University of Göttingen to develop TRex, an open‐source TTCN‐3 development environment, which notably provides suitable metrics and refactorings to enable the assessment and automatic restructuring of test suites. This article presents concepts like metrics and refactoring for the quality assurance of TTCN‐3 test suites and their implementation provided by the TRex tool. These means make it far easier to construct and maintain TTCN‐3 tests that are concise and optimally balanced with respect to maintainability quality characteristics. Copyright © 2008 John Wiley & Sons, Ltd.