Large Language Model-Generated Code (LLMgCode) has become increasingly common in software development. So far LLMgCode has more quality issues than Human-Authored Code (HaCode). It is common for LLMgCode to mix with HaCode in a code change, while the change is signed by only human developers, without being carefully examined. Many automated methods have been proposed to detect LLMgCode from HaCode, in which the perplexity-based method ( Perplexity for short) is the state-of-the-art method. However, the efficacy evaluation of Perplexity has focused on detection accuracy. Yet it is unclear whether Perplexity is good enough in a wider range of realistic evaluation settings. To this end, we carry out a family of experiments to compare Perplexity against feature- and pre-training-based methods from three perspectives: detection accuracy , detection speed , and generalization capability . The experimental results show that Perplexity has the best generalization capability while having limited detection accuracy and detection speed. Based on that, we discuss the strengths and limitations of Perplexity , e.g., Perplexity is unsuitable for high-level programming languages. Finally, we provide recommendations to improve Perplexity and apply it in practice. As the first large-scale investigation on detecting LLMgCode from HaCode, this article provides a wide range of findings for future improvement.

Small and focused commits are highly valued in modern software development. However, developers sometimes submit a commit with more than one concern, represented by several lines of code changes for a specific purpose, e.g., adding new features or fixing bugs. Such composite commits confuse developers during code reviews as well as other software activities, resulting in various issues. Existing studies predominantly leverage code structure to untangle composite commits, but without considering code semantics that have been demonstrated to be important in many related studies. In this article, we propose UNTCC, a new approach that uses structural and semantic information forUNTanglingCompositeCommits. To achieve structural information, we propose the code change graph, a fine-grained, text-attributed graph representation of a commit, incorporating before-change and after-change code dependencies; and UNTCC employs the graph autoencoder to learn its structural representation. To achieve semantic information, UNTCC leverages a large language model (Llama-3.2-3B) to learn joint embeddings of the raw commit and its aligned graph representation, which guide the division of different concerns within a composite commit. The experimental evaluation using 27,853 composite commits from 9 C# and 10 Java projects shows that in terms of Accuracya/Accuracyc, UNTCC achieves 94%/74% in C# and 77%/54% in Java, outperforming state-of-the-art approaches by 2%—623%/32%—573% in C# and 22%—285%/35%—286% in Java. The results indicate that UNTCC can effectively untangle composite commits.

Enhancing the modular structure of existing systems has attracted substantial research interest, focusing on two main methods: (1) software modularization and (2) identifying design issues (e.g., smells) as refactoring opportunities. However, remodularization solutions often require extensive modifications to the original modules, and the design issues identified are generally too coarse to guide refactoring strategies. Combining the above two methods, this paper introduces a novel concept, PairSmell, which exploits modularization to pinpoint design issues necessitating refactoring. We concentrate on a granular but fundamental aspect of modularity principles-modular relation (MR), i.e., whether a pair of entities are separated or collocated. The main assumption is that, if the actual MR of a pair violates its ‘apt MR’, i.e., an MR agreed on by multiple modularization tools (as raters), it can be deemed likely a flawed architectural decision that necessitates further examination. To quantify and evaluate PairSmell, we conduct an empirical study on 20 C/C++ and Java projects, using 4 established modularization tools to identify two forms of PairSmell: inapt separated pairs$InSep$and inapt collocated pairs$InCol$. Our study on 260,003 instances reveals that their architectural impacts are substantial: (1) on average, 14.60 % and 20.44 % of software entities are involved in$InSep$and$InCol$MRs respectively; (2)$InSep$pairs are associated with 190 % more co-changes than properly separated pairs, while$InCol$pairs are associated with 35% fewer co-changes than properly collocated pairs, both indicating a successful identification of modular structures detrimental to software quality; and (3) both forms of PairSmell persist across software evolution. This evidence strongly suggests that PairSmell can provide meaningful insights for inspecting modular structure, with the identified issues being both granular and fundamental, making the enhancement of modular design more efficient.

Commit messages are valuable resources for describing why code changes are committed to repositories in version control systems (e.g., Git).They effectively help developers understand code changes and better perform software maintenance tasks.Unfortunately, developers often neglect to write high-quality commit messages in practice.Therefore, a growing body of work is proposed to generate commit messages automatically.These works all demonstrated that how to organize and represent code changes is vital in generating good commit messages, including the use of fine-grained graphs or embeddings to better represent code changes.In this study, we choose an alternative way to condense code changes before generation, i.e., proposing brief yet concise text templates consisting of the following three parts: (1) summarized code changes, (2) elicited comments, and (3) emphasized code identifiers.Specifically, we first condense code changes by using our proposed templates with the help of a heuristic-based tool named ChangeScribe, and then fine-tune CodeLlama-7B on the pairs of our proposed templates and corresponding commit messages.Our proposed templates better utilize pre-trained language models, while being naturally brief and readable to complement generated commit messages for developers.

Log statements have become an integral part of modern software systems.Prior research efforts have focused on supporting the decisions of placing log statements, such as where/what to log.With the increasing adoption of Large Language Models (LLMs) for coderelated tasks such as code completion or generation, automated approaches for generating log statements have gained much momentum.However, the performance of these approaches still has a long way to go.This paper explores enhancing the performance of LLM-based solutions for automated log statement generation by post-training LLMs with a purpose-built dataset.Thus the primary contribution is a novel approach called AUCAD, which automatically constructs such a dataset with information extracting from log-related issues.Researchers have long noticed that a significant portion of the issues in the open-source community are related to log statements.However, distilling this portion of data requires manual efforts, which is labor-intensive and costly, rendering it impractical.Utilizing our approach, we automatically extract logrelated issues from 1,537 entries of log data across 88 projects and identify 808 code snippets (i.e., methods) with retrievable source code both before and after modification of each issue (including log statements) to construct a dataset.Each entry in the dataset consists of a data pair representing high-quality and problematic log statements, respectively.With this dataset, we proceed to post-train multiple LLMs (primarily from the Llama series) for automated * Corresponding author.

Many build systems, such as Make, rely on build scripts that are written by users to specify dependencies. As a serious dependency error in Makefiles, Missing Dependencies (MDs) can result in compiling and linking outdated artifacts in incremental builds, preventing software project updates from being applied correctly. Many studies have explored the detection of MDs. Automatically fixing those missing build dependency errors has become an apparent but challenging task. The challenges mainly result from Makefiles having complex semantics and project maintainers declaring dependencies in a variety of ways. To address these challenges, we propose a new approach to fixing MDs called MDfixer. The core idea of MDfixer is to identify the dependency declaration style in a Makefile and generate patches for the same declaration style based on declaration graphs and automatic prompt generation. Specifically, MDfixer locates dependency declarations for targets that have errors in the Makefile based on error reports, and then builds a declaration graph for each build target with errors and identifies the target’s declaration style based on a distance metric between the target and the dependencies. Based on the declaration graph and automatic prompt generation, MDfixer generates patches with the same style for the dependencies that need to be added. We evaluated the effectiveness and efficiency of MDfixer with 35 well-known projects. The evaluation results show that MDfixer can fix all MDs. We submitted fixes for 2,786 individual dependency issues across 17 projects, with 11 of them merging our pull requests, resulting in a total of 2,099 errors being fixed. MDfixer consumes an average time of 3.31 min for fixing a project, with a median of 62.999s. It can assist practitioners in the effective and efficient fixing of MDs.

Modern software development reuses third-party libraries to cut costs but may introduce vulnerabilities. A critical practice is to verify the security of third-party libraries against public vulnerability reports. Many automated methods have been proposed to identify vulnerable libraries from vulnerability reports. Existing methods are designed for the generic identification of vulnerable libraries, considering the security of all software libraries. Generic identification is inherently challenging, resulting in limited accuracy. However, organizations only consider the security of libraries they trust and use, by self-managing a library whitelist. Therefore, we propose LibGuard, a framework to adapt existing methods to help organizations secure the libraries they use. LibGuard supplies a library whitelist for existing methods and filters the results according to a threshold, facilitating the discovery of risks overlooked by organizations while controlling false alarms. LibGuard is implemented in two ways. The first attaches the whitelist after existing methods. The second integrates the whitelist into existing methods. We evaluated LibGuard using 5,107 vulnerability reports and the library whitelist built from 79 Google projects and 29 Huawei projects. The results show that the two implementations of LibGuard increase the average F1 score by 10.25% and 11.77%, respectively. Moreover, LibGuard performs stably during the extension of whitelists. To our knowledge, this paper is the first study dedicated to securing self-managed third-party libraries, offering insights into adapting generic software security management to self-managed contexts.

Abstract Software engineers play a centric role throughout the software development lifecycle. Their activities directly impact the quality, performance, and successful delivery of software products, in particular for enterprises with an emphasis on high levels of quality assurance and timely delivery. Proper incentives that motivate software engineers are vital to secure and continuously improve development productivity and software quality. However, most existing research ignores the positive incentives for software engineers, especially industry‐oriented research. In addition, existing research largely relies on peer assessment and lacks objectivity and transparency. To this end, this study investigates the process of contribution measurement for software engineers in a global Information and Communications Technology (ICT) enterprise, to explore the practical experiences and significance of contribution measurement. We investigated the practices of contribution measurement through multiple methods, including archival analysis, interviews, and survey. A total of 22 software engineers were interviewed to understand the practical implementation process of measuring contributions and its impact on software processes as well as engineers. In addition, 74 responses to our questionnaire were collected and used for a comprehensive impact analysis on software engineers. The analysis results reveal five benefits for software development processes and four benefits for practitioners of contribution measurement in the studied enterprise. In addition, this study reports on the best practices of contribution measurement, such as team‐specific measurements, and provides a practical reference for researchers and organizations interested in studying or performing contribution measurement.

As code review is a tedious and costly software quality practice, researchers have proposed several machine learning-based methods to automate the process. The primary focus has been on accuracy, that is, how accurately the algorithms are able to detect issues in the code under review. However, human intervention still remains inevitable since results produced by automated code review are not 100% correct. To assist human reviewers in making their final decisions on automatically generated review comments, the comprehensibility of the comments underpinned by accurate localization and relevant explanations for the detected issues with repair suggestions is paramount. However, this has largely been neglected in the existing research. Large language models (LLMs) have the potential to generate code review comments that are more readable and comprehensible by humans, thanks to their remarkable processing and reasoning capabilities. However, even mainstream LLMs perform poorly in detecting the presence of code issues because they have not been specifically trained for this binary classification task required in code review. In this article, we contribute Comprehensibility of Automated Code Review using Large Language Models ( Carllm ), a novel fine-tuned LLM that has the ability to improve not only the accuracy but, more importantly, the comprehensibility of automated code review, as compared to state-of-the-art pre-trained models and general LLMs.

Incompletely declared build dependencies in MAKE-based build scripts can result in incorrect or inefficient incremental builds and parallel builds for C/C++ projects. In this sense, developing MAKE-based build scripts (e.g., Makefile) is a nontrivial task, since practitioners need to manually enumerate the dependencies between the parts involved in one build, which may result in serious dependency errors such as missing dependencies or redundant dependencies. To tackle this challenge, the software engineering community has invested considerable effort in dependency error detection. However, due to issues such as incomplete or even missing static dependencies (i.e., dependencies by users declared in Makefile), existing solutions either miss certain critical dependency errors or consume significant time when parsing build dependencies, posing a major challenge to ensure both detection effectiveness and efficiency. We propose a novel approach called BuildChecker to detect the above two critical types of dependency errors in MAKE dependencies that leverages a dynamically generated build execution-declaration model to improve error detection performance and reduce detection time. We evaluate BuildChecker with state-of-the-art tools (Mkcheck, Buildfs, VeriBuild, and VirtualBuild) on 30 projects. The experimental results show that BuildChecker is able to detect a total of 13,579 dependency errors with only 29 false positives, fewer than all the state-of-the-art tools. In terms of detection efficiency, BuildChecker outperforms Buildfs by 1.38 times and Mkcheck by 66.24 times. All dependency errors had been submitted to the practitioners and maintainers of these projects. At the time of writing this article, we received responses from the maintainers of four projects, who confirmed our error reports and fixes. BuildChecker demonstrates a great potential to support practitioners effectively detect build dependency errors.

Background:Along with the rapid development and widespread adoption of blockchain technology, many common practices have been summarized into blockchain-based design patterns for application development. However, the numerous and scattered patterns may cause confusion among practitioners. Therefore, adopting appropriate patterns to meet various requirements has become a major challenge, as it requires deep development experience and blockchain technology knowledge.Objective:To address this problem, this paper proposes a decision-support solution to assist with the selection of design patterns during the blockchain-based application development, including a layered taxonomy of design patterns, mappings of quality attributes with the patterns, and a decision model incorporating the taxonomy and mappings.Method:We collected 72 distinct and state-of-the-art design patterns via a Systematic Literature Review (SLR) to establish a layered taxonomy, and 18 unified quality attribute metrics were proposed for blockchain-based pattern assessment and mapping establishment. Based on the pattern taxonomy and quality attribute mappings, we developed a decision model that can provide intuitive guidance for pattern selection.Results:The proposed solution was evaluated through a case study in a seafood supply chain, in which we examined how well the decision model could help identify design flaws and provide reasonable solutions. Additionally, interviews and a questionnaire-based survey were conducted to measure the completeness, correctness, and usefulness of the proposed decision model. The evaluation results indicate that the proposed decision-support solution provides developers with comprehensive guidance, facilitates targeted decision making, and supports intuitive understanding.Conclusions:Our decision-support solution can improve the development efficiency of blockchain-based applications, especially in addressing potential design flaws, achieving targeted quality attributes, and reducing development costs.

Evolutionary designis a widely accepted practice for defining microservice boundaries. It is performed through a sequence of incremental refactoring tasks (we call it“microservice refactoring”), each restructuring only part of a microservice system (a.k.a., refactoring part) into well-defined services for improving the architecture in a controlled manner. Despite its popularity in practice, microservice refactoring suffers from insufficient methodological support. While there are numerous studies addressing similar software design tasks,i.e., software remodularization and microservitization, their approaches prove inadequate when applied to microservice refactoring. Our analysis reveals that their approaches may even degrade the entire architecture in microservice refactoring, as they only optimize the refactoring part in such applications, but neglect the relationships between the refactoring part and the remaining system. As the first response to the need,Micro2Microis proposed to re-partition the refactoring part while optimizing three quality objectives including the interdependence between the refactoring and non-refactoring parts. In addition, it allows architects to intervene in the decision-making process by interactively incorporating their knowledge into the iterative search for optimal refactoring solutions. An empirical study on 13 open-source projects of different sizes shows that the solutions fromMicro2Microperform well and exhibit quality improvement with an average up to 45% to the original architecture. Users ofMicro2Microfound the suggested solutions highly satisfactory. They acknowledge the advantages in terms of infusing human intelligence into decisions, providing immediate quality feedback, and quick exploration capability.

Traceability allows stakeholders to extract and comprehend the trace links among software artifacts introduced across the software life cycle, to provide significant support for software engineering tasks. Despite its proven benefits, software traceability is challenging to recover and maintain manually. Hence, plenty of approaches for automated traceability have been proposed. Most rely on textual similarities among software artifacts, such as those based on Information Retrieval (IR). However, artifacts in different abstraction levels usually have different textual descriptions, which can greatly hinder the performance of IR-based approaches (e.g., a requirement in natural language may have a small textual similarity to a Java class). In this work, we leverage the consensual biterms and transitive relationships (i.e., inner- and outer-transitive links) based on intermediate artifacts to improve IR-based traceability recovery. We first extract and filter biterms from all source, intermediate, and target artifacts. We then use the consensual biterms from the intermediate artifacts to enrich the texts of both source and target artifacts, and finally deduce outer and inner-transitive links to adjust text similarities between source and target artifacts. We conducted a comprehensive empirical evaluation based on five systems widely used in other literature to show that our approach can outperform four state-of-the-art approaches in Average Precision over 15% and Mean Average Precision over 10% on average.

Reliable effort estimation is of paramount importance to software planning and management, especially in industry that requires effective and on-time delivery. Although various estimation approaches have been proposed (e.g., planning poker and analogy), they may be manual and/or subjective, which are difficult to apply to other projects. In recent years, deep learning approaches for effort estimation that rely on learning expert features or semantic features respectively have been extensively studied and have been found to be promising. Semantic features and expert features describe software tasks from different perspectives, however, in the literature, the best combination of these two features has not been explored to enhance effort estimation. Additionally, there are a few studies that discuss which expert features are useful for estimating effort in the industry. To this end, we investigate the potential 13 expert features that can be used to estimate effort by interviewing 26 enterprise employees. Based on that, we propose a novel model, called Fine-SE, that leverages semantic features and expert features for effort estimation. To validate our model, a series of evaluations are conducted on more than 30,000 software tasks from 17 industrial projects of a global ICT enterprise and four open-source software (OSS) projects. The evaluation results indicate that Fine-SE provides higher performance than the baselines on evaluation measures (i.e., mean absolute error, mean magnitude of relative error, and performance indicator), particularly in industrial projects with large amounts of software tasks, which implies a significant improvement in effort estimation. In comparison with expert estimation, Fine-SE improves the performance of evaluation measures by 32.0%-45.2% in within-project estimation. In comparison with the state-of-the-art models, Deep-SE and GPT2SP, it also achieves an improvement of 8.9%-91.4% in industrial projects. The experimental results reveal the value of integrating expert features with semantic features in effort estimation.

Trustworthy Open Source Software (OSS) development processes are the basis that secures the long-term trustworthiness of software projects and products. With the aim to investigate the trustworthiness of the Pull Request (PR) process, the common model of collaborative development in OSS community, we exploit process mining to identify and analyze the normal and anomalous patterns of PR processes, and propose our approach to identifying anomalies from both control-flow and semantic aspects, and then to analyze and synthesize the root causes of the identified anomalies. We analyze 17531 PRs of 18 OSS projects on GitHub, extracting 26 root causes of control-flow anomalies and 19 root causes of semantic anomalies. We find that most PRs can hardly contain both semantic anomalies and control-flow anomalies, and the internal custom rules in projects may be the key causes for the identified anomalous PRs. We further discover and analyze the patterns of normal PR processes. We find that PRs in the non-fork model (42%) are far more likely than the fork model (5%) to bypass the review process, indicating a higher potential risk. Besides, we analyzed nine poisoned projects whose PR practices were indeed worse. Given the complex and diverse PR processes in OSS community, the proposed approach can help identify and understand not only anomalous PRs but also normal PRs, which offers early risk indications of suspicious incidents (such as poisoning) to OSS supply chain.

Hyperledger Fabric (Fabric for short), is a consortium blockchain platform that adopts the smart contract paradigm and provides complete operational functions. Although it has become the system with the highest throughput among open source blockchain systems, its performance cannot meet the needs of industrial-grade application scenarios. To further expand the application scenarios of blockchain, this paper proposes a Transaction Batch Processing Scheduling (TBPS) algorithm for multi-channel Fabric networks based on Lyapunov optimization theory. The algorithm maximizes the consensus efficiency of the system while ensuring the minimum transaction accumulation, and provides stability conditions and optimal performance for the system under transaction batch processing. Finally, we built a blockchain network of Fabric’s latest stable version v 2.0 via the cloud platform, providing an order of magnitude of algorithmic parameters by testing transaction processing rates. To simulate the distribution of performance indicators such as transaction delay, system transaction accumulation and average transaction processing rate under different impact factors, and verify the effectiveness of the proposed TBPS algorithm.

Software Process Model (SPM) is an abstraction of the software development process over time to assist in managing the process. SPM has attracted significant attention from researchers and practitioners in the past decades. Due to the complexity of SPM, building a practical process model often requires collaboration between academia and industry. Unfortunately, there are few empirical studies on SPM conducted in collaboration with enterprises. In this paper, we report on the challenges and solutions encountered while modeling software processes based on our collaboration with a global enterprise. These experiences are valuable to both researchers and practitioners. We presented the modeling process in detail and collected all the interview records during collaboration. As a result of building an SPM in the enterprise, we identify seven challenges and discussed solutions for each of them. The fundamental issue with SPM remains the quality and availability of data, even within industry settings. To enhance the value and applicability of models, we propose a checklist for building simulation models. The checklist can be used by modelers and practitioners to verify details that are easily overlooked during the modeling process. Our experience report provides a practical reference with researchers and practitioners who are interested in modeling software process.

Incremental and parallel builds performed by build tools such as Make are the heart of modern C/C++ software projects. Their correct and efficient execution depends on build scripts. However, build scripts are prone to errors. The most prevalent errors are missing dependencies (MDs) and redundant dependencies (RDs). The state-of-the-art methods for detecting these errors rely on clean builds (i.e., full builds of a subset of software configurations in a clean environment), which is costly and takes up to a few hours for large-scale projects. To address these challenges, we propose a novel approach called EChecker to detect build dependency errors in the context of incremental builds. The core idea of EChecker is to automatically update actual build dependencies by inferring them from C/C++ pre-processor directives and Makefile changes from new commits, which avoids clean builds when possible. EChecker achieves higher efficiency than the methods that rely on clean builds while maintaining effectiveness. We selected 12 representative projects, with their sizes ranging from small to large, with 240 commits (20 commits for each project), based on which we evaluated the effectiveness and efficiency of EChecker. We compared the evaluation results with a state-of-the-art build dependency error detection tool. The evaluation shows that the F-1 score of EChecker improved by 0.18 over the state-of-the-art method. EChecker increases the build dependency error detection efficiency by an average of 85.14 times (with a median of 16.30 times). The results demonstrate that EChecker can support practitioners in detecting build dependency errors efficiently.

Software engineers play an important role throughout the software development life-cycle, particularly in industry emphasizing quality assurance and timely delivery. Contribution measurement provides proper incentives to software engineers that motivate them to continuously improve the quality and efficiency of their work. However, existing research tends to ignore contribution measurement for software engineers in practice, relying heavily on peer review and lacking objectivity and transparency. Specifically, these studies still have two weaknesses. First, a few studies explore which metrics can be useful for contribution measurement in practice. Second, managers measure the contribution of software engineers based on their experience and lack of explainable automated tools to assist them.

Traceability plays a vital role in facilitating various software development activities by establishing the traces between different types of artifacts (e.g., issues and commits in software repositories). Among the explorations for automated traceability recovery, the IR (Information Retrieval)-based approaches leverage textual similarity to measure the likelihood of traces between artifacts and show advantages in many scenarios. However, the globalization of software development has introduced new challenges, such as the possible multilingualism on the same concept (e.g., "[SEE PDF]" vs. "attribute") in the artifact texts, thus significantly hampering the performance of IR-based approaches. Existing research has shown that machine translation can help address the term inconsistency in bilingual projects. However, the translation can also bring in synonymous terms that are not consistent with those in the bilingual projects (e.g., another translation of "[SEE PDF]" as "property"). Therefore, we propose an enhancement strategy called AVIATE that exploits translation variants from different translators by utilizing the word pairs that appear simultaneously across the translation variants from different kinds artifacts (a.k.a. consensual biterms). We use these biterms to first enrich the artifact texts, and then to enhance the calculated IR values for improving IR-based trace-ability recovery for bilingual software projects. The experiments on 17 bilingual projects (involving English and 4 other languages) demonstrate that AVIATE significantly outperformed the IR-based approach with machine translation (the state-of-the-art in this field) with an average increase of 16.67 in Average Precision (31.43%) and 8.38 (11.22%) in Mean Average Precision, indicating its effectiveness in addressing the challenges of multilingual traceability recovery.

Peer code review has become a must-have in modern software development. However, many code review requests (CRRs) could be a backlog for large-scale and active projects, blocking continuous integration and continuous delivery (CI/CD). Prioritizing CRRs to make the relevant ones to be reviewed first is a critical method for addressing this issue. Early studies have shown that many factors affect the review priority of a CRR, including its properties and relationships with other CRRs. However, the relationships, e.g., modifying the same files and sharing the same authors, are rarely considered when developing CRR prioritizers. In this paper, we propose a Graph-Powered Prioritizer (namely GPP) to make full use of the properties and relationships of CRRs. GPP uses the multi-graph structure to develop an initial representation of a collection of CRRs and uses the graph neural network algorithm to learn the prioritization-adapted representation, and eventually, outputs an ordered list of CRRs based on it. With experimental evaluation, we define relevant CRRs in the context of CI/CD as those that are likely to achieve three objectives, i.e., being merged while undergoing a few iterations in a short duration. We compare GPP against two rule-based and six learning-based prioritizers on 15 open-source software projects with more than 420K CRRs. The experimental results indicate that GPP outperforms the baselines on three basic ranking-aware evaluation metrics, including NDCG (82.94%), MRR (36.52%), and MAP (63.80%); while providing benefits in recommending the most relevant CRRs and balancing multiple objectives. Data&materials: https://figshare.com/s/133f23da558b7b254041

Abstract Software process simulation models (SPSMs) that are based on descriptive process models offer the executability that can demonstrate dynamic changes of software processes over time. Verification and validation (V&V) is critical in SPSMs for guaranteeing the quality and reliability of models. V&V of dynamic software process models is more complex and challenging than for static software process models. This work systematically summarizes and maps V&V studies in SPSM to provide guidelines for future research and practice. Specifically, this study aims at identifying the focus of research on V&V, the methods used for V&V, and how to implement V&V of SPSMs in software engineering research. We conducted a systematic mapping study on studies of SPSMs that report on their V&V activities. Under the guidance of a V&V meta‐model for SPSMs, we study four research questions about V&V process. We identified 107 primary studies from a pool of 313 papers on SPSMs until 2021. There are two main results of our study. The first one presents the relationship between quality aspects of SPSMs and the V&V methods to assure them. The second result reveals the relationships among the modeling process, three modeling steps, five quality aspects, and 10 V&V methods. Generally, researchers do not pay sufficient attention to V&V, as 65.8% ( ) failed to mention or elaborate on their V&V process. We systematically summarize and map the state‐of‐the‐art V&V research in software process modeling field to support modelers' practice and improve their V&V process.

Abstract Software process simulation (SPS) has become an effective tool for software process management and improvement. However, its adoption in industry is less than what the research community expected due to the burden of measurement cost and the high demand for domain knowledge. The difficulty of extracting appropriate metrics with real data from process enactment is one of the great challenges. We aim to provide evidence‐based support of the process metrics for software process (simulation) modeling. A systematic literature review was performed by extending our previous review series to draw a comprehensive understanding of the metrics for process modeling following our proposed ontology of metrics in SPS. We identify 131 process modeling studies that collectively involve 1975 raw metrics and classified them into 21 categories using the coding technique. We found product and process external metrics are not used frequently in SPS modeling while resource external metrics are widely used. We analyze the causal relationships between metrics. We find that the models exhibit significant diversity, as no pairwise relationship between metrics accounts for more than 10% SPS models. We identify 17 data issues may encounter in measurement and 10 coping strategies. The results of this study provide process modelers with an evidence‐based reference of the identification and the use of metrics in SPS modeling and further contribute to the development of the body of knowledge on software metrics in the context of process modeling. Furthermore, this study is not limited to process simulation but can be extended to software process modeling, in general. Taking simulation metrics as standards and references can further motivate and guide software developers to improve the collection, governance, and application of process data in practice.

Abstract As a predominant design method for microsservices architecture (MSA), domain‐driven design (DDD) utilizes a series of standard patterns in both models and implementations to effectively support the design of architectural elements. However, an implementation may deviate from its original domain model that uses certain patterns. The deviation between a domain model and its implementation is a type of architectural drift, which needs to be detected promptly. This paper proposes an approach, namely DOMICO, to check the conformance between the domain model and its implementation, by which the conformance is formalized by defining eight common structural patterns of domain modeling and their representations in both models and the corresponding source code. Based on the formalization, our approach can not only identify the discrepancies (e.g., divergence, absence, and modification) with respect to pattern elements, but also detect possible violations of 24 compliance rules imposed by the patterns. To validate DOMICO, we performed a case study to investigate its use in a supply chain project and its performance. The results show that DOMICO can accurately identify 100% inconsistency issues in the cases examined. As the first conformance checking approach for DDD, DOMICO can be integrated into the regular domain modeling process and help ensure the conformity of microservice implementations to models.

Code review is an important practice in software development. One of its main objectives is for the assurance of code quality. For this purpose, the efficacy of code review is subject to the credibility of reviewers, i.e., reviewers who have demonstrated strong evidence of previously making quality-enhancing comments are more credible than those who have not. Code reviewer recommendation (CRR) is designed to assist in recommending suitable reviewers for a specific objective and, in this context, assurance of code quality. Its performance is susceptible to the relevance of its training dataset to this objective, composed of all reviewers’ historical review comments, which, however, often contains a plethora of comments that are irrelevant to the enhancement of code quality. Furthermore, recommendation accuracy has been adopted as the sole metric to evaluate a recommender's performance, which is inadequate as it does not take reviewers’ relevant credibility into consideration. These two issues form the ground truth problem in CRR as they both originate from the relevance of dataset used to train and evaluate CRR algorithms. To tackle this problem, we first propose the concept of Quality-Enhancing Review Comments (QERC), which includes three types of comments - change-triggering inline comments, informative general comments, and approve-to-merge comments. We then devise a set of algorithms and procedures to obtain a distilled dataset by applyingQERCto the original dataset. We finally introduce a new metric – reviewer's credibility for quality enhancement (RCQE) – as a complementary metric to recommendation accuracy for evaluating the performance of recommenders. To validate the proposed QERC-based approach to CRR, we conduct empirical studies using real data from seven projects containing over 82K pull requests and 346K review comments. Results show that: (a)QERCcan effectively address the ground truth problem by distilling quality-enhancing comments from the dataset containing original code reviews, (b)QERCcan assist recommenders in finding highly credible reviewers at a slight cost of recommendation accuracy, and (c) even “wrong” recommendations using the distilled dataset are likely to be more credible than those using the original dataset.

MicroService Architecture (MSA), a predominant architectural style in recent years, still faces the arduous task of identifying the boundaries of microservices. Domain-Driven Design (DDD) is regarded as one of the major design methods for addressing this task in practice, which aims to iteratively build domain models using a series of patterns, principles, and practices. The adoption of DDD for MSA (DDD4Min short) can, however, present considerable challenges in terms of a sufficient understanding of the methodological requirements and the application domains. It is imperative to establish a systematic understanding about the various aspects of employing DDD4M and provide effective guidance. This study reports an empirical inquiry that integrates a systematic literature review and a confirmatory survey. By reviewing 34 scientific studies and consulting 63 practitioners, this study reveals several distinctive findings with regard to the state and challenges of as well as the possible solutions for DDD4M applications, from the5W1Hperspectives:When,Where,Why,Who,What, andHow. The analysis and synthesis of evidence show a wide variation in understanding of domain modeling artifacts. The status quo indicates the need for further methodological support in terms of application process, domain model design and implementation, and domain knowledge acquisition and management. To advance the state-of-the-practice, our findings were organized into a preliminary checklist that intends to assist practitioners by illuminating a DDD4M application process and the specific key considerations along the way.

Blockchain systems have received increased interest over the past few years, and several new fields of use, such as supply chain systems, are being investigated. Since blockchain is still a new technology, various papers have explored how to apply it to support use cases outside the limited scope of digital currencies. Systems require solid technological implementation and perceived trust among users to ensure their interests and successful usage in practice. This study aimed to understand what graphic user interface (GUI) elements of a blockchain-based system make users trust that their best interests, such as security and privacy, are maintained in the systems. As a case study, we developed a few blockchain-based supply chain GUI mockups with different elements that reflect the security and privacy features of the system. We then conducted 30 interviews in Norway and China to collect the users’ opinions on whether the information presented in the GUIs helps them trust the system. The results show that users want access to as much information and data as the system can provide. The users’ trust in the system increases if the GUI features give users the impression that the inner workings of the blockchain-based system are transparent. However, users prefer the information presented as more conceptual than technical in the first place. However, users appreciate the possibility of clicking on the conceptual explanation and getting more in-depth blockchain-related technical information if needed.

Logs record the behavioral data of running programs and are typically generated by executing log statements. Software developers generally carry out logging practices with clear intentions and associated concerns (I&Cs). However, I&Cs may not be properly fulfilled in source code as log placement - specifically determination of a log statement's context and content - is often susceptible to an individual's profile and experience. Some industrial studies have been conducted to discern developers' main logging I&Cs and the way I&Cs are fulfilled. However, the findings are only based on the developers from a single company in each individual study and hence have limited generalizability. More importantly, there lacks a comprehensive and deep understanding of the relationships between developers' profiles and experiences and their logging practices from a wider perspective. To fill this significant gap, we conducted an empirical study using mixed methods comprising questionnaire surveys, semi-structured interviews, and code analyses with practitioners from a wide range of companies across a variety of industrial domains. Results reveal that while developers share common logging I&Cs and conduct logging practices mainly in the coding stage, their profiles and experiences profoundly influence their logging I&Cs and the way the I&Cs are fulfilled. These findings pave the way to facilitate the acceptance of important logging I&Cs and the adoption of good logging practices by developers

The rapid growth of software industry has led to a significant increase in the production of a variety of data during software development process, highlighting the apparent need for improved data quality management. As an effective means of software process research and practice, Software Process Simulation Modeling (SPSM) requires large amount and high quality data that precisely depicts what happens during the development process. Accordingly, process simulation models can be used as a reference framework for assessing the issues in data management and data governance from a process perspective. The objective of the work reported in this paper is to provide insights into the data issues in real-world industrial settings and the corresponding coping strategies for software process modelers in particular in order to assist them in preparing and assessing data for their simulation models when conducting effective SPSM in the real-world settings. This paper reports on an empirical investigation that applies software process simulation practices to study the data issues and the data governance strategies based on an industrial case from one global ICT enterprise. As the outcome, a refined process for data preparation is presented, along with a taxonomy of the data issues and the corresponding coping strategies. This paper also explores traceability recovery approaches to mine more accurate process state information from software artifacts and analyzes the impact of the recovered data traceability information by evaluating the improved fidelity of the process simulation model.

The success of a software organization relies heavily on the quality of its products and services, which in turn are influenced by the knowledge, capability, and experience of the software engineers involved in development processes. It is popular to apply quantitative assessments of software engineers for quality assurance. However, the extent to which it benefits software organizations and how it can be effectively implemented in industrial settings remains unclear. One global Information and Communications Technology (ICT) enterprise has implemented a quantitative assessment practice of software engineer’s outputs to improve its engineering capability and product and service quality. To investigate the benefits and experiences of adopting this practice in industrial settings, we conducted an empirical study using a mixed-method approach (i.e., archive analysis, interviews, and surveys). The results indicate that this practice can benefit the ICT enterprise in terms of standardizing development processes, optimizing team structures, and offering suggestions for training and management, etc. Meanwhile, this paper reports on the best practices to tackle the challenges during the adoption of the practice in the ICT enterprise, e.g., customization for teams and synergy of quantitative and qualitative assessment. In addition, we discuss the implications and recommendations of institutionalizing quantitative engineer assessment in software organizations. For organizations intending to improve software quality from the human aspect, this study provides empirical references on how to implement quantitative engineer assessment meanwhile mitigate potential risks.

In code reviews, developers examine code changes authored by peers and provide feedback through comments. Despite the importance of these comments, no accepted approach currently exists for assessing their quality. Therefore, this study has two main objectives: (1) to devise a conceptual model for an explainable evaluation of review comment quality, and (2) to develop models for the automated evaluation of comments according to the conceptual model. To do so, we conduct mixed-method studies and propose a new approach: EvaCRC (Evaluating Code Review Comments). To achieve the first goal, we collect and synthesize quality attributes of review comments, by triangulating data from both authoritative documentation on code review standards and academic literature. We then validate these attributes using real-world instances. Finally, we establish mappings between quality attributes and grades by inquiring domain experts, thus defining our final explainable conceptual model. To achieve the second goal, EvaCRC leverages multi-label learning. To evaluate and refine EvaCRC, we conduct an industrial case study with a global ICT enterprise. The results indicate that EvaCRC can effectively evaluate review comments while offering reasons for the grades. Data and materials: https://doi.org/10.5281/zenodo.8297481

In large-scale, active software projects, one of the main challenges with code review is prioritizing the many Code Review Requests (CRRs) these projects receive. Prior studies have developed many Learning-to-Rank (LtR) models in support of prioritizing CRRs and adopted rich evaluation metrics to compare their performances. However, the evaluation was performed before observing the complex interactions between CRRs and reviewers, activities and activities in real-world code reviews. Such a pre-review evaluation provides few indications about how effective LtR models contribute to code reviews. This study aims to perform a post-review evaluation on LtR models for prioritizing CRRs. To establish the evaluation environment, we employ Discrete-Event Simulation (DES) paradigm-based Software Process Simulation Modeling (SPSM) to simulate real-world code review processes, together with three customized evaluation metrics. We develop seven LtR models and use the historical review orders of CRRs as baselines for evaluation. The results indicate that employing LtR can effectively help to accelerate the completion of reviewing CRRs and the delivery of qualified code changes. Among the seven LtR models, LambdaMART and AdaRank are particularly beneficial for accelerating completion and delivery, respectively. This study empirically demonstrates the effectiveness of using DES-based SPSM for simulating code review processes, the benefits of using LtR for prioritizing CRRs, and the specific advantages of several LtR models. This study provides new ideas for software organizations that seek to evaluate LtR models and other artificial intelligence-powered software techniques.Data&materials: https://figshare.com/s/a033e99cd2a61e64c8bc.

Abstract By adopting agile and lean practices, DevOps aims to achieve rapid value delivery by speeding up development and deployment cycles, which however lead to more security concerns that cannot be fully addressed by an isolated security role only in the final stage of development. DevSecOps promotes security as a shared responsibility integrated into the DevOps process that seamlessly intertwines development, operations, and security from the start throughout to the end of cycles. While some companies have already begun to embrace this new strategy, both industry and academia are still seeking a common understanding of the DevSecOps movement. The goal of this study is to report the state‐of‐the‐practice of DevSecOps, including the impact of DevOps on security, practitioners' understanding of DevSecOps, and the practices associated with DevSecOps as well as the challenges of implementing DevSecOps. The authors used a mixed‐methods approach for this research. The authors carried out a grey literature review on DevSecOps, and surveyed the practitioners of DevSecOps in industry of China. The status quo of DevSecOps in industry is summarized. Three major software security risks are identified with DevOps, where the establishment of DevOps pipeline provides opportunities for security‐related activities. The authors classify the interpretations of DevSecOps into three core aspects of DevSecOps capabilities, cultural enablers, and technological enablers. To materialise the interpretations into daily software production activities, the recommended DevSecOps practices from three perspectives—people, process, and technology. Although a preliminary consensus is that DevSecOps is regarded as an extension of DevOps, there is a debate on whether DevSecOps is a superfluous term. While DevSecOps is attracting an increasing attention by industry, it is still in its infancy and more effort needs to be invested to promote it in both research and industry communities.

In software quality management , the selection strategy for proper metrics varies depending on the application scenarios and measurement objectives. MicroService Architecture (MSA), despite being commonly employed nowadays, still cannot be reliably measured and compared if the microservices in a system are independent. Software managers and architects need to understand whether their microservices are “decoupled enough”, if not, which ones are over-coupled, and by how much. In this paper, we contribute a novel set of metrics – Microservice Coupling Index (MCI) – derived from the relative measurement theory. Instead of measuring coupling evidence with simple counts, we measure how dependent and coupled the microservices are relative to the possible couplings between them. We measured the MCI metrics for 15 open source projects that involve 113 distinct microservices. Empirical investigation confirmed that MCIs differ quite significantly from existing coupling measures and that they are more discriminative than existing ones for separating high and low degrees of microservice couplings and thus more useful in comparing design alternatives. A series of experimental studies were conducted, showing that the larger the MCIs, the less likely the bugs and changes can be localized and separated, and the less likely that the individual microservices in a system can be independently developed and evolved.

Continuity and steadiness are vital for services with massive users, which requires the anomalies of services should be detected and resolved in a timely manner. Our previous work proposed a tool, namelyImpAPTr (Impact Analysis based on Pruning Tree), to identify the combination of multiple dimensional attributes as the clues leading to the root cause of service anomalies. However,ImpAPTrapplies a threshold driven strategy, i.e., it needs to be triggered by a$\geq 0.05\%$drop of the success rate of the service calls (abbr.SRSC), which may face problems in an atypical yet pervasive situation in field application. For example, the combination of trivial anomalies (i.e., each causes a drop less than 0.05% toSRSC) can lead to a far more than 0.05% drop onSRSC. Besides, a suitable threshold is usually hard to be determined, etc. To address these problems, we propose a new method, namelyImpAPTr+in this paper to free the constraint of the 0.05% threshold. The basic idea is to involve time dimension and identify clues across multiple time intervals of data. We performed evaluation on three typical methods (i.e.,ImpAPTr+,R-AdtributorandSqueeze) with both production environment dataset and simulation dataset. The former dataset is directly retrieved from the service monitoring data inMeituan, one of the largest on-line service providers worldwide. The latter dataset is fabricated also using the monitoring data from the same company. The results indicate: (1)ImpAPTr+outperforms previous approaches to a large degree in terms of accuracy. (2) BothImpAPTr+andR-Adtributorare able to find proper clues within seconds. (3)ImpAPTr+tends to find proper clues with shorter time intervals (i.e., less data), which implies that the method is more suitable for near real-time monitoring scenarios.

The microservice architecture has been commonly adopted by large scale software systems exemplified by a wide range of online services. Service monitoring through anomaly detection and root cause analysis (RCA) is crucial for these microservice systems to provide stable and continued services. However, compared with monolithic systems, software systems based on the layered microservice architecture are inherently complex and commonly involve entities at different levels of granularity. Therefore, for effective service monitoring, these systems have a special requirement of multi-granular RCA. Furthermore, as a large proportion of anomalies in microservice systems pertain to problematic code, to timely troubleshoot these anomalies, these systems have another special requirement of RCA at the finest code-level. Microservice systems rely on telemetry data to perform service monitoring and RCA of service anomalies. The majority of existing RCA approaches are only based on a single type of telemetry data and as a result can only support uni-granular RCA at either application-level or service-level. Although there are attempts to combine metric and tracing data in RCA, their objective is to improve RCA's efficiency or accuracy rather than to support multi-granular RCA. In this article, we propose a new RCA solutionTrinityRCLthat is able to localize the root causes of anomalies at multiple levels of granularity including application-level, service-level, host-level, and metric-level, with the unique capability of code-level localization by harnessing all three types of telemetry data to construct a causal graph representing the intricate, dynamic, and nondeterministic relationships among the various entities related to the anomalies. By implementing and deployingTrinityRCLin a real production environment, we evaluateTrinityRCLagainst two baseline methods and the results show thatTrinityRCLhas a significant performance advantage in terms of accuracy at the same level of granularity with comparable efficiency and is particularly effective to support large-scale systems with massive telemetry data.

Background:Logging practices provide the ability to record valuable runtime information of software systems to support operations tasks such as service monitoring and troubleshooting. However, current logging practices face common challenges. On the one hand, although the importance of logging practices has been broadly recognized, most of them are still conducted in an arbitrary or ad-hoc manner, ending up with questionable or inadequate support to perform these tasks. On the other hand, considerable research effort has been carried out on logging practices, however, few of the proposed techniques or methods have been widely adopted in industry.Objective:This study aims to establish a comprehensive understanding of the research state of logging practices, with a focus on unveiling possible problems and gaps which further shed light on the potential future research directions.Method:We carried out a systematic mapping study on logging practices with 56 primary studies.Results:This study provides a holistic report of the existing research on logging practices by systematically synthesizing and analyzing the focus and inter-relationship of the existing research in terms of issues, research topics and solution approaches. Using3W1H—Why to log,Where to log,What to logandHow well is the logging—as the categorization standard, we find that: (1) the best known issues in logging practices have been repeatedly investigated; (2) the issues are often studied separately without considering their intricate relationships; (3) theWhere and Whatquestions have attracted the majority of research attention while little research effort has been made on theWhyandHow wellquestions; and (4) the relationships between issues, research topics, and approaches regarding logging practices appear many-to-many, which indicates a lack of profound understanding of the issues in practice and how they should be appropriately tackled.Conclusions:This study indicates a need to advance the state of research on logging practices. For example, more research effort should be invested onwhy to logto set the anchor of logging practices as well as onhow well is the loggingto close the loop. In addition, a holistic process perspective should be taken into account in both the research and the adoption related to logging practices.

Continuous Integration (CI) enables developers to detect defects early and thus reduce lead time. However, the high frequency and long duration of executing CI have a detrimental effect on this practice. Existing studies have focused on using CI outcome predictors to reduce frequency. Since there is no reported project using predictive CI, it is difficult to evaluate its economic impact. This research aims to investigate predictive CI from a process perspective, including why and when to adopt predictors, what predictors to be used, and how to practice predictive CI in real projects. We innovatively employ Software Process Simulation to simulate a predictive CI process with a Discrete-Event Simulation (DES) model and conduct simulation-based experiments. We develop the Rollback-based Identification of Defective Commits (RIDEC) method to account for the negative effects of false predictions in simulations. Experimental results show that: 1) using predictive CI generally improves the effectiveness of CI, reducing time costs by up to 36.8% and the average waiting time before executing CI by 90.5%; 2) the time-saving varies across projects, with higher commit frequency projects benefiting more; and 3) predictor performance does not strongly correlate with time savings, but the precision of both failed and passed predictions should be paid more attention. Simulation-based evaluation helps identify overlooked aspects in existing research. Predictive CI saves time and resources, but improved prediction performance has limited cost-saving benefits. The primary value of predictive CI lies in providing accurate and quick feedback to developers, aligning with the goal of CI.

Since 2009, the deep learning revolution, which was triggered by the introduction of ImageNet, has stimulated the synergy between Software Engineering (SE) and Machine Learning (ML)/Deep Learning (DL). Meanwhile, critical reviews have emerged that suggest that ML/DL should be used cautiously. To improve the applicability and generalizability of ML/DL-related SE studies, we conducted a 12-year Systematic Literature Review (SLR) on 1,428 ML/DL-related SE papers published between 2009 and 2020. Our trend analysis demonstrated the impacts that ML/DL brought to SE. We examined the complexity of applying ML/DL solutions to SE problems and how such complexity led to issues concerning the reproducibility and replicability of ML/DL studies in SE. Specifically, we investigated how ML and DL differ in data preprocessing, model training, and evaluation when applied to SE tasks, and what details need to be provided to ensure that a study can be reproduced or replicated. By categorizing the rationales behind the selection of ML/DL techniques into five themes, we analyzed how model performance, robustness, interpretability, complexity, and data simplicity affected the choices of ML/DL models.

Traceability systems are important for solving problems due to the increasing scale of the global supply chain, such as food safety crises and market disorder. Blockchain, as an immutable and decentralized ledger, is able to optimize the traditional traceability system by ensuring the transparency and reliability of the system data. However, the use of blockchain technology may lead to a rapid increase in the complexity of system design and development. It is challenging to address widespread and complicated business, changeable processes, and massive data in practice, which are the main factors restricting the wide application of a blockchain-based traceability system (BTS). Therefore, in this paper, we reviewed relevant studies and proposed a reference architecture for BTSs. The proposed reference architecture can improve the cohesiveness, maintainability, and extensibility of BTSs through domain-driven design (DDD) and microservices. Considering the efficiency reduction caused by massive data and complicated data structure, we further changed the traditional single blockchain framework into multiple subchain networks, which could improve development efficiency and system performance. With the guidance of the architecture trade-off analysis method (ATAM), we evaluated our reference architecture and implemented a prototype in the salmon supply chain scenario. The results show that our solution is effective and adaptable to meet the requirements of BTSs.

Hyperledger Fabric is another development of blockchain technology after Ethereum, which is more suitable as an operating platform for smart contracts. However, the testing technology of Hyperledger Fabric smart contracts (also known as chaincode) is not yet mature currently. Based on this, this paper studies the vulnerability detection of Golang chaincodes. Firstly, we summarize 17 kinds of Golang chaincode vulnerabilities by investigating existing research. Secondly, taking the high accuracy of dynamic detection and the high efficiency of static detection into consideration, we propose a chaincode vulnerability detection framework that combines the dynamic symbolic execution and the static abstract syntax tree analysis technology. We also implement a supporting-tool that can detect the above 15 types of vulnerabilities. Finally, we test the tool by 15 chaincodes collected from GitHub and unknown vulnerabilities were detected in 13 projects. The precision turned out to be 91% after manual inspection. In order to verify the recall rate, we manually inject 30 vulnerabilities into the collected chaincodes and all of them are detected. The evaluation results show the accuracy of the proposed vulnerability detection method for Hyperledger Fabric smart contracts.

Hyperledger Fabric has become one of the most widely used consortium blockchain frameworks with the ability to execute custom smart contracts. Performance modeling and network evaluation are necessary for performance estimation and optimization of the Fabric blockchain platform. The compatibility and effectiveness of existing performance modeling methods must be improved. For this reason, we proposed a compatible performance modeling method using queuing theory for Fabric considering the limited transaction pool. Taking the 2.0 version of Fabric as a case, we have established the model for the transaction process in the Fabric network. By analyzing the two-dimensional continuous-time Markov process of this model, we solved the system stationary equation and obtained the analytical expressions of performance indicators such as the system throughput, the system steady-state queue length, and the system’s average response time. We collected the required parameter values through the official test suite. An extensive analysis and simulation was performed to verify the accuracy and the effectiveness of the model and formula. We believe that this method can be extended to a wide range of scenarios in other blockchain systems.

In order to solve the interconnection and intercommunication problem of a large number of co-existing blockchains, such as public chains, private chains, and consortium chains, cross-chain technology has recently become a hot research topic among scholars years. Due to the limited processing speed of cross-chain transactions, too many cross-chain transactions in the short term may cause network congestion and negatively affect cross-chain performance. Therefore, it is necessary to evaluate and optimize the performance of the cross-blockchain transaction process. This paper takes a typical cross-blockchain model Cosmos as an example and proposes a queuing theoretical model based on limited space. The difference equation is established through the three-dimensional continuous-time Markov process, and performance metrics such as average queue length, transaction execution time, and transaction response time are obtained. Finally, we experimentally simulate the analytical solutions of the relevant performance metrics to verify the effectiveness of the proposed model. We believe this analytical approach can be generalized to other cross-blockchain systems.

Modern code review is a critical and indispensable practice in a pull-request development paradigm that prevails in Open Source Software (OSS) development. Finding a suitable reviewer in projects with massive participants thus becomes an increasingly challenging task. Many reviewer recommendation approaches (recommenders) have been developed to support this task which apply a similar strategy, i.e. modeling the review history first then followed by predicting/recommending a reviewer based on the model. Apparently, the better the model reflects the reality in review history, the higher recommender's performance we may expect. However, one typical scenario in a pull-request development paradigm, i.e. one Pull-Request (PR) (such as a revision or addition submitted by a contributor) may have multiple reviewers and they may impact each other through publicly posted comments, has not been modeled well in existing recommenders. We adopted the hypergraph technique to model this high-order relationship (i.e. one PR with multiple reviewers herein) and developed a new recommender, namely HGRec, which is evaluated by 12 OSS projects with more than 87K PRs, 680K comments in terms of accuracy and recommendation distribution. The results indicate that HGRec outperforms the state-of-the-art recommenders on recommendation accuracy. Besides, among the top three accurate recommenders, HGRec is more likely to recommend a diversity of reviewers, which can help to relieve the core reviewers' workload congestion issue. Moreover, since HGRec is based on hypergraph, which is a natural and interpretable representation to model review history, it is easy to accommodate more types of entities and realistic relationships in modern code review scenarios. As the first attempt, this study reveals the potentials of hypergraph on advancing the pragmatic solutions for code reviewer recommendation.

Software information sites (e.g., Jira, Stack Overflow) are now wide-ly used in software development. These online platforms for collaborative development preserve a large amount of Software Engineering (SE) texts. These texts enable researchers to detect developers’ attitudes toward their daily development by analyzing the sentiments expressed in the texts. Unfortunately, recent works reported that neither off-the-shelf tools nor SE-specified tools for sentiment analysis on SE texts can provide satisfying and reliable results. In this paper, we propose to incorporate pre-trained transformer models into the sentence-classification oriented deep learning framework named TextCNN to better capture the unique expression of sentiments in SE texts. Specifically, we introduce an optimized BERT model named RoBERTa as the word embedding layer of TextCNN, along with additional residual connections between RoBERTa and TextCNN for better cooperation in our training framework. An empirical evaluation based on four datasets from different software information sites shows that our training framework can achieve overall better accuracy and generalizability than the four baselines.

Traceability approves trace links among software artifacts based on whether two artifacts are related by system functionalities. The traces are valuable for software development, but are difficult to obtain manually. To cope with the costly and fallible manual recovery, automated approaches are proposed to recover traces through textual similarities among software artifacts, such as those based on Information Retrieval (IR). However, the low quality & quantity of artifact texts negatively impact the calculated IR values, thus greatly hindering the performance of IR-based approaches. In this study, we propose to extract co-occurred word pairs from the text structures of both requirements and code (i.e., consensual biterms) to improve IR-based traceability recovery. We first collect a set of biterms based on the part-of-speech of requirement texts, and then filter them through the code texts. We then use these consensual biterms to both enrich the input corpus for IR techniques and enhance the calculations of IR values. A nine-system-based evaluation shows that in general, when solely used to enhance IR techniques, our approach can outperform pure IR-based approaches and another baseline by 21.9% & 21.8% in AP, and 9.3% & 7.2% in MAP, respectively. Moreover, when used to collaborate with another enhancing strategy from different perspectives, it can outperform this baseline by 5.9% in AP and 4.8% in MAP.

The traceability of software artifacts has been recognized as an important factor to support various activities in software development processes. However, traceability can be difficult and time-consuming to create and maintain manually, thereby automated approaches have gained much attention. Unfortunately, existing automated approaches for traceability suffer from practical issues. This paper aims to gain an understanding of the potential challenges for the underperforming of the state-of-the-art, ML-based trace link classifiers applied in real-world projects. By investigating different industrial datasets, we found that two critical (and classic) challenges, i.e. data imbalance and sparse problems, lie in real-world projects’ traceability automation. To overcome these challenges, we developed a framework called SPLINT to incorporate hybrid textual similarity measures and semi-supervised learning strategies as enhancements to the learning-based traceability approaches. We carried out experiments with six open-source platforms and ten industry datasets. The results confirm that SPLINT is able to operate at higher performance on two communities’ datasets. Specifically, the industrial datasets, which significantly suffer from data imbalance and sparsity problems, show an increase in F2-score over 14% and AUC over 8% on average. The adjusted class-balancing and self-training policies used in SPLINT (CBST-Adjust) also work effectively for the selection of pseudo-labels on minor classes from unlabeled trace sets, demonstrating SPLINT’s practicability.

Abstract As a recently predominant architecture style, MicroService Architecture (MSA) is likely to suffer the issues of poor maintainability due to inappropriate microservice boundaries. Architectural Smell (AS), as a metaphor for potential architectural issues that may have negative impacts on software maintenance, can be used to pinpoint refactoring opportunity for evolving microservice boundary. However, existing studies mostly focus on AS detection with little further investigation on the possible impacts, causes, and solutions of AS, which does little help in addressing the bad smells in architecture. Our goal in this study is to bridge this gap by investigating the possible impacts, causes, and solutions of AS in MSA‐based systems. An industrial case study is carried out to collect repository data and practitioners' views on six typical ASes in a real MSA‐based telecommunication system. Statistical Analysis and Coding techniques are used in the analyses of quantitative and qualitative data respectively. The results show that AS influences the modularity, modifiability, analyzability, and testability of the MSA‐based system, which further induce extra cross‐team communication, change‐ and fault‐prone microservices. To explore the causes for AS, a five‐aspect conceptual classification with technology, project, organization, business, and professional is proposed, in which the business and organization aspects take the major roles. Both technical and non‐technical solutions are distilled to deal with ASes despite potential constraints. These results and their comparison to current literature are discussed, which provide practical implications in coping with AS in microservices.

Despite widespread agreement on the benefits of code review, its outcomes may not be as expected. The complications can undermine the purpose of the development process and even destroy the entire development cycle. Both academia and the industrial communities have invested a great deal of time and effort into code reviews. When a project team adheres to the best practices and creates a conducive environment, it is likely that code reviews could be conducted effectively and efficiently. By reviewing peer-reviewed scientific publications and gray literature on code review best practices, we summarized 57 practices as well as 19 code review pains that they address. Our review has shown that following best practices can ease the process of code review considerably. Multiple actionable practices are needed to support code review pains at the same time. To enable the adoption of best practices, OSS and industrial communities alike invest in integrating automatic techniques with code review tools. We hope that this review will provide researchers and practitioners with a comprehensive understanding of code review practices, aiding them in conducting code reviews more successfully.

Nowadays, software permeates almost every aspect of our lives. To produce complex and large-scale software products, a large number of software engineers are required. Accordingly, researchers and educators recognize the importance of Software Engineering Education (SEE), and many studies related to SEE have been published in recent years. To synthesize the large amount of research in SEE, some Systematic Literature Reviews (SLRs) focusing on different areas of SEE have been conducted and reported. However, due to their limited focuses, none of these SLRs is able to depict an overall state-of-the-art for SEE. To remedy this, we conducted a tertiary study on SEE, which identifies 26 relevant SLRs published between 2004 and 2019. By classifying and positioning these SLRs in two dimensions, i.e. the education methods/tools applied for SEE and the research topics related to SEE, we present a landscape of SEE, which locates the SLRs on SEE and their research dimensions. Further, we collected the issues studied in the published research and those that need to be addressed for instructors. This paper also discusses the challenges of the current SEE research landscape.

With its unique advantages such as decentralization and immutability, blockchain technology has been widely used in various fields in recent years. The smart contract running on the blockchain is also playing an increasingly important role in decentralized application scenarios. Therefore, the automatic detection of security vulnerabilities in smart contracts has become an urgent problem in the application of blockchain technology. Hyperledger Fabric is a smart contract platform based on enterprise-level licensed distributed ledger technology. However, the research on the vulnerability detection technology of Hyperledger Fabric smart contracts is still in its infancy. In this paper, we propose HFContractFuzzer, a method based on Fuzzing technology to detect Hyperledger Fabric smart contracts, which combines a Fuzzing tool for golang named go-fuzz and smart contracts written by golang. We use HFContractFuzzer to detect vulnerabilities in five contracts from typical sources and discover that four of them have security vulnerabilities, proving the effectiveness of the proposed method.

Sentiment analysis on software engineering (SE) texts has been widely used in the SE research, such as evaluating app reviews or analyzing developers' sentiments in commit messages. To better support the use of automated sentiment analysis for SE tasks, researchers built an SE-domain-specified sentiment dictionary to further improve the accuracy of the results. Unfortunately, recent work reported that current mainstream tools for sentiment analysis still cannot provide reliable results when analyzing the sentiments in SE texts. We suggest that the reason for this situation is because the way of expressing sentiments in SE texts is largely different from the way in social network or movie comments. In this paper, we propose to improve sentiment analysis in SE texts by using sentence structures, a different perspective from building a domain dictionary. Specifically, we use sentence structures to first identify whether the author is expressing her sentiment in a given clause of an SE text, and to further adjust the calculation of sentiments which are confirmed in the clause. An empirical evaluation based on four different datasets shows that our approach can outperform two dictionary-based baseline approaches, and is more generalizable compared to a learning-based baseline approach.

Software has become ubiquitous in every corner of modern societies. During the last five decades, software engineering has also changed significantly to advance the development of various types and scales of software products. In this context, Software Engineering Education plays an important role in keeping students updated with software technologies, processes, and practices that are popular in industries. We investigate from literature the extent Software Engineering Education addresses major Software Engineering Trends in the academic setting. We conducted a systematic mapping study about teaching major Software Engineering Trends in project courses. We classified 126 papers based on their investigated Software Engineering Trends, specifically Software Engineering processes and practices, teaching approaches, and the evolution of Software Engineering Trends over time. We reveal that Agile Software Development is the major trend. The other Trends, i.e., Software Implementation, Usability and Value, Global Software Engineering, and Lean Software Startup, are relatively small in the academic setting, but continuously growing in the last five years. System of Systems is the least investigated among all Trends. The study points out the possible gaps between Software Industry and Education, which implies actionable insights for researchers, educators, and practitioners.

Blockchain and smart contracts are being embraced by more and more industrial practitioners in multiple domains including agriculture, manufacturing, and healthcare. As a distributed, immutable, and partly public ledger, the consortium blockchain demonstrates its potential to enable trustworthy interoperability and collaboration between organizations. However, the mismatch between the unruled software engineering practices and the increased interest of the consortium blockchain technology may pose threats to the quality of systems implemented. To mitigate the possible threats, this study takes the angle of software engineering to systematically understand the challenges and possible solutions in terms of developing and operating a consortium blockchain-based system. For this purpose, we conducted a case study on a typical consortium blockchain-based system and exhaustively collected the data by two rounds in-depth interviews on practitioners of different roles in the case project. Based on the data analysis, eight pairs of challenges and potential solutions were identified, which cover the phases of the development and operation of consortium blockchains. Moreover, we also captured two implications after further analysis of the findings, which worth the special attention of researchers in the near future, i.e. DevOps and microservices for blockchain or smart contracts.

Background: Machine Learning (ML) has been widely used as a powerful tool to support Software Engineering (SE). The fundamental assumptions of data characteristics required for specific ML methods have to be carefully considered prior to their applications in SE. Within the context of Continuous Integration (CI) and Continuous Deployment (CD) practices, there are two vital characteristics of data prone to be violated in SE research. First, the logs generated during CI/CD for training are imbalanced data, which is contrary to the principles of common balanced classifiers; second, these logs are also time-series data, which violates the assumption of cross-validation. Objective: We aim to systematically study the two data characteristics and further provide a comprehensive evaluation for predictive CI/CD with the data from real projects. Method: We conduct an experimental study that evaluates 67 CI/CD predictive models using both cross-validation and time-series-validation. Results: Our evaluation shows that cross-validation makes the evaluation of the models optimistic in most cases, there are a few counter-examples as well. The performance of the top 10 imbalanced models are better than the balanced models in the predictions of failed builds, even for balanced data. The degree of data imbalance has a negative impact on prediction performance. Conclusion: In research and practice, the assumptions of the various ML methods should be seriously considered for the validity of research. Even if it is used to compare the relative performance of models, cross-validation may not be applicable to the problems with time-series features. The research community need to revisit the evaluation results reported in some existing research.

Context: Following on other scientific disciplines, such as health sciences, the use of Grey Literature (GL) has become widespread in Software Engineering (SE) research. Whilst the number of papers incorporating GL in SE is increasing, there is little empirically known about different aspects of the use of GL in SE research.

Background: Logs provide crucial information to understand the dynamic behavior of software systems in modern software development and maintenance. Usually, logs are produced by log statements which will be triggered and executed under certain conditions. However, current studies paid very limited attention to developers' Intentions and Concerns (I&C) on logging practice, leading uncertainty that whether the developers' I&C are properly reflected by log statements and questionable capability to capture the expected information of system behaviors in logs. Objective: This study aims to reveal the status of developers' I&C on logging practice and more importantly, how the I&C are properly reflected in software source code in real-world software development. Method: We collected evidence from two sources of a series of interviews and source code analysis which are conducted in a big-data company, followed by consolidation and analysis of the evidence. Results: Major gaps and inconsistencies have been identified between the developers' I&C and real log statements in source code. Many code snippets contained no log statements that the interviewees claimed to have inserted. Conclusion: Developers' original I&C towards logging practice are usually poorly realized, which inevitably impacted the motivation and purpose to conduct this practice.

Background: Software Process Simulation Modeling (SPSM) is of paramount importance to support quantitative management of software development process. Hybrid process simulation combines multiple simulation paradigms to reflect complex changes in realistic software processes, which brings inherent challenges to process management. Constructing a hybrid model requires more modeling expertise and experience than modeling by solo-paradigm. However, a few studies explicitly discuss the challenges they encountered as a topic, which may discourage practitioners. Objective: Our aim in this study is to present an industrial process modeling project as an exemplar to demonstrate and discuss the technical issues and challenges associated with hybrid process simulation in practice. Method: Based on the collaboration with a global software enterprise, we constructed a hybrid process simulation model that combines System Dynamics (SD) and Discrete Event Simulation (DES) to predict the project duration and release date for management. Results: Several challenges around hybrid process simulation of software development process are identified and discussed with the proposal of sets of solutions from different perspectives. The model is validated by comparing the simulation result with the actual enactment of the process in industry. In addition, the result confirms the rationality and efficacy of the suggested solutions to some extent. Conclusions: In the collaboration with the enterprise, five-step modeling procedure was adopted for constructing the hybrid process model. The experience reported about the detailed steps of hybrid modeling may offer reference value to the SPSM community.

For many on-line systems with massive users, to provide services continuously and steadily is vital for business, which requires the anomalies of services should be located and resolved in a timely manner. As a common IT infrastructure, various APM (Application Performance Management) systems/frameworks have been adopted to monitor each call request to a service. Nevertheless, the call request may contain multidimensional attributes (e.g., City, ISP, Platform, etc.), which may further contain multiple values (e.g., ISP could be T-Mobile, CMCC, etc.). As a result, an anomaly such as DSR (Declining Success Rate) to service typically occurs with a combination of such attribute values, which creates major challenges to locate the root cause of the anomaly due to potentially huge numbers of the combinations. In this paper, we propose a novel method, ImpAPTr (Impact Analysis based on Pruning Tree), to identify the combination of dimensional attributes as the clues leading to the root cause of anomalies regarding DSR timely. In the evaluation with the simulated dataset, ImpAPTr detects valid clues in milliseconds with an accuracy of 99.37% (within the top 10 candidate results), 97.72% (top 5), and 94.51% (top 3), respectively, which outperforms previous approaches to a large degree. A field test with a production environment dataset indicates that ImpAPTr is able to detect valid clues in a few seconds.

Context: Emerging from the agile culture, DevOps particularly emphasizes development and deployment speed to achieve rapid value delivery, which however brings some security risks to the software development process. DevSecOps is an extension of DevOps, which is considered as a means to intertwine development, operation and security. Some companies with security concerns begin to take DevSecOps into consideration when it comes to the application of DevOps. Objective: The goal of this study is to report the state-of-the-practice of DevSecOps as well as calling for academia to pay more attention to DevSecOps. Method: Using Google search engine to collect articles on DevSecOps, we conducted a Grey Literature Review (GLR) on the selected articles. Results: Whilst there exists three major software security risks in DevOps, the establishment of DevOps pipeline provides opportunities for software security activities. Based on the preliminary consensus that DevSecOps is an extension of DevOps, it is observed that the interpretations of DevSecOps can be classified into three core aspects, which are: DevSecOps capabilities, cultural enablers, and technological enablers. Furthermore, to materialize the interpretations into daily software production activities, the recommended DevSecOps practices we obtain from Grey Literature (GL) can be categorized in terms of process, infrastructure and collaboration. Conclusion: Although DevSecOps is getting increasing attention by industry, it is still in its infancy and needs to be promoted by both academia and industry.

Software development is a collective undertaking, and the team’s efficiency is critical in development. In order to reduce project management overheads and improve productivity, a global information and communication technology enterprise institutionalizes an organization wide small-team practice, called fireteams, to tackle the problems arising from human and social aspects, such as amicability, talent, skill, and communications. This paper reports a mixed-method research, which combines archive analysis, interviews and survey, to empirically investigate the characteristics and impacts of fireteam in this industrial setting. We identify three categories of fireteam in terms of its demonstrated characteristics: ordinary agile team with extensions, single-function team, and entire life-cycle team; elaborate four key activities of fireteam, i.e. team formation, maintenance, communication, and meeting. Less communication and management overheads, higher agility & concurrency, and improved personal ability are the three important contributors that increase the productivity of fireteams. Whereas management & leadership effort, divergent understanding of fireteam, and self-organized team are discovered as the three major problems associated with fireteams. Although the benefits of fireteam can be observed from its adoption, this practice does not achieve the enterprise’s anticipations very well. Some considerations and recommendations are also discussed to improve this small-team practice.

Summary The proliferation of DevOps enables significant acceleration and automation of the delivery and deployment of massive software products. Unfortunately, the development of supporting documents that is vital for large‐scale software systems in many cases does not keep pace with the rhythm of feature delivery using DevOps in practice, which becomes the bottleneck for many software organizations to deliver full value to the customers as claimed by the DevOps. This paper proposes, implements, and evaluates an integrated approach, DevDocOps, for continuous automated documentation, in particular for DevOps. With DevDocOps, supporting documents are created along with the development process simultaneously by various roles within a DevOps project, which largely guarantees the accuracy and integrity of documents as well as significantly increases their delivery speed. Within an established delivery chain, a set of templates are created to collect and transform the required information from its origin to the target documents for delivery. A real system, iDoc, is implemented to map, collect, and synthesize the information from document templates and automate the documentation process. DevDocOps has been successfully adopted in a top‐tier global telecommunication enterprise to support more than 5000 users with different roles related to documentation. The lag time between the releases of the product version and its supporting document has been shortened from 1 to 2 months on average to less than 2 days. DevDocOps extends the scope of DevOps and enhances the value delivery by supporting continuous documentation and bridges the gap between feature delivery and document delivery with automation.

Context: Data synthesis is one of the most significant tasks in Systematic Literature Review (SLR). Software Engineering (SE) researchers have adopted a variety of methods of synthesizing data that originated in other disciplines. One of the qualitative data synthesis methods is meta-ethnography, which is being used in SE SLRs. Objective: We aim at studying the adoption of meta-ethnography in SE SLRs in order to understand how this method has been used in SE. Method: We conducted a tertiary study of the use of meta-ethnography by reviewing sixteen SLRs. We carried out an empirical inquiry by integrating SLR and confirmatory email survey. Results: There is a general lack of knowledge, or even awareness, of different aspects of meta-ethnography and/or how to apply it. Conclusion: There is a need of investment in gaining in-depth knowledge and skills of correctly applying meta-ethnography in order to increase the quality and reliability of the findings generated from SE SLRs. Our study reveals that meta-ethnography is a suitable method to SE research. We discuss challenges and propose recommendations of adopting meta-ethnography in SE. Our effort also offers a preliminary checklist of the systematic considerations for doing meta-ethnography in SE and improving the quality of meta-ethnographic research in SE.

Background: Seeking an appropriate architecture for a software design is always a challenge in recent decades. Although microservices as a lightweight architecture style is claimed that can improve the current practices with several characteristics, many practices are based upon the different circumstances and reflect the variant effects. An empirical inquiry brings us a systematic insight into the industrial practices on microservices. Objective: This study is to investigate the gap between the ideal visions and real industrial practices on microservices and what benefits we can gain from the industrial experiences. Method: We carried out a series of industrial interviews with thirteen different types of companies. The collected data were then codified according to the defined qualitative methods. Results: We characterized the gaps between the typical characteristics accepted in the community and the industrial practices of microservices. Furthermore, the compromise between benefits and sufferings of microservices around these nine dimensions were also investigated. Conclusion: We confirmed the benefits of the microservices that can be obtained from practice as well as their possible pains that need to be addressed with extra expense from experiences. Besides, some outlined pains, e.g., organizational transformation, decomposition, distributed monitoring, and bug localization, may inspire researchers to conduct the further research.

Logs are the execution results of logging statements in software systems after being triggered by various events, which is able to capture the dynamic behavior of software systems during runtime and provide important information for software analysis, e.g., issue tracking, performance monitoring, etc. Obviously, to meet this purpose, the quality of the logs is critical, which requires appropriately placement of logging statements. Existing research on this topic reveals that where to log? and what to log? are two most concerns when conducting logging practice in software development, which mainly relies on developers' personal skills, expertise and preference, rendering several problems impacting the quality of the logs inevitably. One of the reasons leading to this phenomenon might be that several recognized best practices(strategies as well) are easily neglected by software developers. Especially in those software projects with relatively large number of participants. To address this issue, we designed and implemented a plug-in tool (i.e., JLLAR) based on the Intellij IDEA, which applied machine learning technology to identify and create a set of rules reflecting commonly recognized logging practices. Based on this rule set, JLLAR can be used to scan existing source code to identify issues regarding the placement of logging statements. Moreover, JLLAR also provides automatic code completion and semi code completion (i.e., to provide recommendations) regarding logging practice to support software developers during coding.

Software Engineering (SE) community has recently been investing significant amount of effort in qualitative research to study the human and social aspects of SE processes, practices, and technologies. Ethnography is one of the major qualitative research methods, which is based on constructivist paradigm that is different from the hypothetic-deductive research model usually used in SE. Hence, the adoption of ethnographic research method in SE can present significant challenges in terms of sufficient understanding of the methodological requirements and the logistics of its applications. It is important to systematically identify and understand various aspects of adopting ethnography in SE and provide effective guidance. We carried out an empirical inquiry by integrating a systematic literature review and a confirmatory survey. By reviewing the ethnographic studies reported in 111 identified papers and 26 doctoral theses and analyzing the authors' responses of 29 of those papers, we revealed several unique insights. These identified insights were then transformed into a preliminary checklist that helps improve the state-of-the-practice of using ethnography in SE. This study also identifies the areas where methodological improvements of ethnography are needed in SE.

Synthesizing data extracted from primary studies is an integral component of the methodologies in support of Evidence Based Software Engineering (EBSE) such as System Literature Review (SLR). Since a large and increasing number of studies in Software Engineering (SE) incorporate qualitative data, it is important to systematically review and understand different aspects of the Qualitative Research Synthesis (QRS) being used in SE. We have reviewed the use of QRS methods in 328 SLRs published between 2005 and 2015. We also inquired the authors of 274 SLRs to confirm whether or not any QRS methods were used in their respective reviews. 116 of them provided the responses, which were included in our analysis. We found eight QRS methods applied in SE research, two of which, narrative synthesis and thematic synthesis, have been predominantly adopted by SE researchers for synthesizing qualitative data. Our study determines that a significant amount of missing knowledge and incomplete understanding of the defined QRS methods in the community. Our effort also identifies an initial set factors that may influence the selection and use of appropriate QRS methods in SE.

As an emerging approach to support fast delivery of software features with reliable quality, DevOps attracts more and more practitioners and shows the potential to become one of the mainstream approach for software development and operation. Many universities begin to offer DevOps related courses to the students majored in software engineering and computer science. However, as a critical part of a DevOps course, the project practicing using DevOps might cast big challenges for teachers, compared to traditional project practicing. For example, the more frequent than ever delivery in DevOps practicing will inevitably increase the workload vastly for teachers to conduct effective evaluation. In this paper, we introduce a web based system (DevOpsEnvy) to support the management and monitoring of student teams practicing DevOps. By integrating several popular open source tools, this system provides students with features such as group management, project status monitoring and student performance data analysis, etc. Meanwhile, DevOpsEnvy system also provides teachers with sufficient evidence to perform evaluation. Our preliminary trial in Nanjing University revealed several advantages of DevOpsEnvy system.

Background: Capture-ReCapture (CRC), as a technique for post-inspection defect estimation, has been studied in Software Engineering (SE) community since 1990s. While most studies focused on the performance evaluation of various CRC models and estimators, few have been done on the assessment of the credibility of estimation results, rendering the difficulty of decision-making for quality management when applying CRC for defect estimation. Objective: This research aims to explore and investigate a reliable and practical approach to assess the credibility of CRC based defect estimation. Method: One fundamental assumption of applying CRC method is the statistical independence of samples that can be measured by 'Coefficient of CoVariation' (CCV). We applied CCV as an indicator of the statistical dependence between the observations (i.e., the defects detected by inspectors), and assessed the estimation results of CRC with the published datasets in SE literature by examining the correlation between Relative Error (RE) and CCV. Based on the observed correlation, we further propose CĈV, which replaces the unknown N (the actual number of defects) with the estimated number (N), to assess the credibility of CRC estimates. Results: We found that most datasets are with non-zero CCVs and the R2 (Coefficient of Determination) of non-linear curve-fitting for their CCVs and REs is higher than 0.8. Conclusions: Our study shows the evidence that the statistical dependence among inspectors is ubiquitous in the existing CRC-related studies. Besides, the significant correlation between CCV (by CĈV in practice) and RE may enable the possibility of the assessment of CRC-based estimation in support of quality management.

Verification and Validation (V&V) is a critical step in software process modelling to secure the model's quality and credibility. Software Process Simulation Models (SPSMs) that are based on descriptive process models offer the executability that is able to demonstrate the dynamic changes of software process over time. The V&V of process simulation models go beyond static process models and turn to be more complex and challenging to software modelers. This study aims to identify what aspects of process simulation models are verified and validated by using which V&V methods in what conditions in software engineering research. We conducted a systematic literature review (mapping study) on the studies of software process simulation that report of their V&V activities. We identified 72 relevant studies from a pool of 331 papers on SPSM until 2015. These studies can be mapped to ten V&V methods applied for five aspects of process models to be verified and validated, i.e., syntactic quality, semantic quality, pragmatic quality, performance, and value. A systematic map is presented to illustrate the relationships between the identified V&V methods and their supporting aspects of process models. This mapping will provide the community reference value when developing, verifying, and validating software process (simulation) models.

V-model and its variants have become the most common process models adopted in automotive industry guiding the development of systems on a variety of refinement levels. Along with the exponentially growing complexity of modern vehicle systems, however, the late verification and validation in the conventional V-model expand in uncontrollable ways that result in higher cost of development and higher risk of failure than ever. This paper describes an inc-V development process for automotive industry that improves the conventional V-model and variants by introducing and institutionalizing early and continuous integrated verification enabled by simulation-based development. We developed a continuous simulation model of the inc-V process, and the initial version is used to investigate the characteristics of the inc-V compared to V. The preliminary finding from the simulations of an example project is that the inc-V process is able to improve the traditional V process by saving effort, shortening duration, and increasing product quality. The finding also show how the advance of development technology impacts the systems engineering processes.

Context: The assessment of Threats to Validity (TTVs) is critical to secure the quality of empirical studies in Software Engineering (SE). In the recent decade, Systematic Literature Review (SLR) was becoming an increasingly important empirical research method in SE. One of the mechanisms of insuring the level of scientific value in the findings of an SLR is to rigorously assess its validity. Hence, it is necessary to realize the status quo and issues of TTVs of SLRs in SE. Objective: This study aims to investigate the-state-of-the-practice of TTVs of the SLRs published in SE, and further support SE researchers to improve the assessment and strategies against TTVs in order to increase the quality of SLRs in SE. Method: We conducted a tertiary study by reviewing the SLRs in SE that report the assessment of TTVs. Results: We identified 316 SLRs published from 2004 to the first half of 2015, in which TTVs are discussed. The issues associated to TTVs were also summarized and categorized. Conclusion: The common TTVs related to SLR research, such as internal validity and reliability, were thoroughly discussed in most SLRs. The threats to construct validity and external validity drew less attention. Moreover, there are few strategies and tactics being reported to cope with the various TTVs.

Background: The Capture-recapture (CRC) method has been adopted in software inspection post-inspection defect estimation. One outstanding advantage of the CRC method is that it is able to produce objective estimates without relying on historical data. However, a common impression about the CRC method is its poor performance regarding estimation accuracy with small inspection teams. Involving more inspectors seems to be helpful, yet no conclusive results exist on the reasonable team size in order to get acceptable CRC estimates.

Very recently, an increasing number of software companies adopted DevOps to adapt themselves to the ever-changing business environment. While it is important to mature adoption of the DevOps for these companies, no dedicated maturity models for DevOps exist. Meanwhile, maturity models such as CMMI models have demonstrated their effects in the traditional paradigm of software industry, however, it is not clear whether the CMMI models could guide the improvements with the context of DevOps. This paper reports a case study aiming at evaluating the feasibility to apply the CMMI models to guide process improvement for DevOps projects and identifying possible gaps. Using a structured method(i.e., SCAMPI C), we conducted a case study by interviewing four employees from one DevOps project. Based on evidence we collected in the case study, we managed to characterize the maturity/capability of the DevOps project, which implies the possibility to use the CMMI models to appraise the current processes in this DevOps project and guide future improvements. Meanwhile, several gaps also are identified between the CMMI models and the DevOps mode. In this sense, the CMMI models could be taken as a good foundation to design suitable maturity models so as to guide process improvement for projects adopting the DevOps.

There was a time when researching software processes meant just that – we were interested in making sure that the process for software development was effective. From a software process perspective, we did not really have to worry about competition, the business nor the domain in which our software was used – because these were not seen as affecting the software processes through which the software was developed. But, things have changed! Software has become more ubiquitous. Software is used in products that are governed by regulation. Software is being developed in organizations that heretofore did not consider themselves software companies – such as automotive and medical device companies. Software is fundamental to many businesses that would not continue to survive if their software is not running effectively. In addition, modern software systems consist of a complex mix of products and services, some are decades old, and some are merely emerging. They may easily suffer from symptoms of aging and need to be continuously adapted to cope with changing requirements and environments. The sources of such changes may be new customers, intense competition, changing organizational structures and regulatory frameworks, changing interacting systems, bug fixing, software degradation and erosion, emerging opportunities and risks in the business environment, as well as emerging software technologies and platforms. In order to overcome or avoid the negative effects of software aging, we have to place change and evolution at the center of the software development and maintenance processes. Evolving process drivers are not necessarily software driven. If we consider drivers in (i) these will be evolved through software engineering, either being developed in industry or by researchers. Drivers in (ii) may possibly evolve through software engineering, and the world of software engineering will have an input. But, often it is the business case that takes priority. And, drivers in (iii) are determined by the business in which we work 5. In these cases, we, as software engineers must consider the business domain to establish what software processes are of interest and will be required to use particular processes 6. However, developing software engineering processes on a one-off basis and then letting them be used ad infinitum without any further changes will not work. Change in the evolving process drivers cause software engineering processes to evolve. In addition, these evolving process drivers can be critical, particularly as the software engineering process increasingly contributes to the company's profitability, be that a direct or indirect contribution. The latest trends of developing and maintaining emerging and evolving software systems also leads to new opportunities and challenges regarding the development processes, including, but not limited to, process evolution, scalability, and process verification and validation. The research presented in the International Conference on Software and Systems Process contributes to this evolution. International Conference on Software and Systems Process is also playing a role in answering these questions. The three papers chosen for this special issue all have an industry focus. It is through publishing such papers that we can bridge the industry-research gap, ensuring that our research will have an impact, not only through our teaching and graduate programs but also by having a direct effect on the improvement of processes within industry. During the evolution history of this conference series, from 1984 to 1996, the International Software Process Workshops (SPW) attracted many academic researchers and industrial practitioners. Then came the International Conference on the Software Process, (ICSP, from 1991 until 1996), the International Workshop on Software Process Simulation and Modeling (from 1998 until 2006), and the SPW (in 2005 and 2006). International Workshop on Software Process Simulation and Modeling and SPW were held together in 2006 and merged in 2007 to form the new International Conference on Software Process. In 2009, the third ICSP conference was held in Vancouver, Canada, on May 16–17, and was co-located with ICSE 2009 (the International Conference on Software Engineering). The ICSSP (International Conference on Software and System Process) conferences continue the successful ICSP conference series, while broadening ICSP's scope of software development processes to system development and explicitly including processes of other domains such as health care, business, and manufacturing. By sharing process development theories and practices from such domains, ICSSP 2014 aimed at investigating novel solutions to today's software and systems process challenges. The theme of ICSSP 2014 was ‘Processes for Emerging and Evolving Software Systems’. There were 55 submissions (including 35 full papers and 20 short papers) to the conference that were received from 18 countries and regions including Australia, Austria, Brazil, Canada, China, Estonia, Denmark, France, Germany, Ireland, Italy, Japan, Netherlands, Portugal, Russian Federation, Sweden, United Kingdom, and United States of America. After a rigorous review process, 23 papers were accepted for the proceedings of the ICSSP 2014 including 12 full papers and 11 short papers. Following the conference, the editors ranked the accepted full papers based on their reviews and sent invitations to authors of five papers to expand and submit their papers for consideration in the special issue. Papers were submitted, peer reviewed, and revised. Based on the reviews, the Guest Editors selected four out of five papers to be included in this special issue of the Journal of Software Evolution and Process. We have also included a paper from keynote speaker, Prof K. Ryan from Lero. In an extension of his talk at ICSSP 2014, he discusses the importance of changing processes because of the changing world of software. He specifically discusses how specialization, industrialization, globalization, and Agile methods have disrupted software development and presents examples of how software process research has evolved new methods to cope with these changes. Software process lines provide a systematic approach to develop and manage software processes. It defines a reference process containing general process assets, whereas a well-defined customization approach allows process engineers to create new process variants. Variability operations are an instrument to realize flexibility by explicitly declaring required modifications, which are applied to create a procedurally generated company-specific process. However, little is known about which variability operations are suitable in practice. M. Kuhrmann, D. Méndez Fernández and T. Ternité, in their paper ‘On the Use of Variability Operations in the V-Modell XT Software Process Line’, study the feasibility of variability operations to support the development of software process lines in the context of the V-Modell XT. They provide an initial catalog of variability operations as an improvement proposal for other process models. The identified variability operations allow for systematically modifying the content of process model elements and the process documentation, and they allow for altering the structure of a process model and its description. High-maturity software development processes, such as the Team Software Process and the accompanying Personal Software Process, can generate significant amounts of data that can be periodically analyzed to identify performance problems, determine their root causes and devise improvement actions. However, there is a lack of tool support for automating that type of analysis and hence diminish the manual effort and expert knowledge required. M. Raza and J. P. Faria, in their paper ‘A Model for Analyzing Performance Problems and Root Causes in the Personal Software Process’, propose a comprehensive performance model, addressing time estimation accuracy, quality, and productivity, to enable the automated (tool based) analysis of performance data produced by Personal Software Process developers, namely, identify, and rank performance problems and their root causes. Software test processes are complex and costly. To reduce testing effort without compromising effectiveness and product quality, automation of test activities has been adopted as a popular approach in software industry. However, because test automation usually requires substantial upfront investments, automation is not always more cost-effective than manual testing. V. Garousi and D. Pfahl, in their paper ‘When to Automate Software Testing? A Decision-Support Approach Based on Process Simulation’, investigate how the simulation model using the System Dynamics modeling technique can help decision-makers decide whether and to what degree the company should automate their test processes. Workflow temporal verification guarantees on-time completion which is one of the most important quality of service dimensions for business processes running in the cloud. However, as today's business systems often need to handle a large number of concurrent customer requests, conventional response time based process monitoring strategies conducted in a one by one fashion cannot be applied efficiently to a large batch of parallel processes because of significant time overhead. To address this problem, X. Liu, D. Wang, D. Yuan, F. Wang, Y/ Yang, in their paper ‘Workflow Temporal Verification for Monitoring ParallelBusiness Processes’, proposes a quality of service-aware throughput based checkpoint selection strategy, which can dynamically select a small number of checkpoints along the system timeline to facilitate the temporal verification of throughput constraints and achieve the target on-time completion rate, based on a novel runtime throughput consistency model. In conclusion, we would like to thank all the authors and reviewers of this special issue. We also would like to thank the ICSSP Steering Committee: Barry Boehm, Ross Jeffrey, Mingshu Li, Leon Osterweil, and David Raffo for their support and help with the conference. We hope you enjoy reading the papers and that, within industry, this research can be useful.

Background: Code inspection has been accepted as an effective method to detect and remove defects and code reading is a critical step in code inspection. However, there are very limited empirical studies on the content and appropriate forms of the suitable software artifacts as the supporting materials, hence inspectors may not be well-supported with necessary knowledge to carry out code reading. Objective: This research aims to investigate the impact of different common supporting materials (i.e., comments vs. design documents) on code reading. Method: A relatively large-scale controlled experiment with 135 senior students was designed and executed to compare the impacts of different supporting materials on code reading. The subjects were randomly separated into three groups with different treatments, i.e, the comments, the design documents and the comments+design documents, respectively. Two metrics regarding the code reading performance (i.e., Effectiveness and Defect Detection Rate) were used to compare the different impacts derived from the two different types of supporting materials. Qualitative feedbacks were also collected using questionnaires for the final analysis. Results: The results indicate that students performed better when being provided with comments than comments+design documents. Also, the removal of design documents shows little impact on inspection effectiveness and may lead to an increase in defect detection rate. Conclusion: Comments may provide more help and value than design documents as supporting material in small to median sized code reading.

Context: Capture-recapture method has long been adopted in software engineering as a relatively objective way for defect estimation. While many relevant studies have been carried out to evaluate various capture-recapture models and estimators, there still lacks common understanding on the adoption status of the method in software engineering. It is necessary to systematically collect empirical evidence of Capture-recapture adoption hence form necessary understanding on the method.

Context: The quality of an Systematic Literature Review (SLR) is as good as the quality of the reviewed papers. Hence, it is vital to rigorously assess the papers included in an SLR. There has been no tertiary study aimed at reporting the state of the practice of quality assessment used in SLRs in Software Engineering (SE).

Software process simulation (SPS) has become an active research area for managing and improving software development processes since its introduction in the last two decades. Hybrid simulation, the combination of simulation paradigms to address a problem, is becoming more popular as the problems we are presented with become more complex. However, integrating multiple simulation paradigms faces the issues of compatibility, interoperatability and synchronization when executing simulation. The objective of this research is to present the state-of-the-art of this research area, the hybrid mechanism when integrating paradigms, and more importantly provide practical support for the effective adoption of hybrid simulation in SPS context. Based on an extended systematic literature review, this paper presents the preliminary results by answering the research questions. Depending upon the way these simulation paradigms represent different aspects and levels of software process and the context in which they can be modeled by SPS, two hybrid mechanisms: Hierarchical Mechanism and Interlinked Mechanism, have been frequently employed. The detailed discussions of integration strategies and recommendations when applying hybrid simulation may offer reference value to the SPS community.

Training and learning are one important purpose of Software Process Simulation (SPS). Some previous reviews showed a noticeable number of studies that combine SPS and Soft- ware Engineering Education (SEE). The objective of this research is to present the latest state-of-the-art of this area, and more importantly provide practical support for the effective adoption of SPS in educational context. We conducted an extended Systematic Literature Review (SLR) based on our previous reviews. The review identified 42 primary studies from 1992 to 2013. This paper presents the preliminary results by answering the research questions. The overall findings confirmed the positive impact of SPS on education. The detailed discussions and recommendations may offer reference value to the community.

In order to enhance the understanding of important concepts and strengthen the awareness of software process, we designed a special project-practicing course in Nanjing University as an attempt to solve typical issues in these courses (e.g., focusing on aspects of software process, participation, limited time in a regular semester, etc.). The course is composed of 6-hour lecture and 32-hour bidding game. Preliminary results indicated several advantages with this new education approach on process-specific practicing course, which we already reported on CSEE&T2013. Since this course has been delivered to students from school (less experiences) and industry (more experiences), we noticed students' different performances on this course. In this paper, we collected course results from six classes, based on a comprehensive analysis from 8 different aspects; we try to understand where “EXPERIENCE” impacts students' difference performance and benefit from the understanding to improve our education on software engineering.

Code inspection is believed to be an effective technique to remove defects and improve software quality. However, the adoption of code inspection in industry is far less than it should be, which may lead to many novice inspectors in industry. For these novice inspectors, a suitable reading technique should be of the first step to begin this quality journey. While reports indicated that Checklist-Based Reading (CBR) and Ad Hoc Reading (AHR) had been the most adopted inspection techniques in industry, we deem it is necessary to investigate these two techniques first. In this paper, we present a case study of the adoption of code reading techniques in one small-sized software company. In this study, five engineers used different techniques (i.e., CBR vs. AHR) to read source code in 20 modules. Both quantitative data and qualitative data are collected during the case study. Initial analysis of these data indicates that industrial novice inspectors using CBR tended to have a lower reading speed than those using AHR. Both techniques could help these novice inspectors to remove a certain portion of defects during code review, and compared to AHR approach, CBR may help them find larger percentage of defects. However, there still exist several issues, for example, missing large portion of review-removable defects could not be avoided for novice inspectors. What's more, CBR may limit reviewers' ability to find defects outside the checklist, and to establish effective checklist remains a big challenge for novice inspectors. Besides, both internal factors (e.g., faith in inspection to achieve high quality) as well as external factors (e.g., schedule pressure) may also impact novice inspectors to adopt code reading.

Context: Training and learning is one important purpose of Software Process Simulation (SPS). A noticeable number of studies employing SPS for Software Engineering Education (SEE) raised our interest. Objective: This research aims to reveal the-state-of-the-art and empirical effectiveness assessment of this meaningful subject area. Method: We conducted a Systematic Literature Review (SLR) to investigate this subject. Results: The review identified 34 primary studies and 10 simulators. Conclusions: The findings show that this is a continuous research area and SPS game appears more attractive to education than non-game SPS. In contrast, only one study on non-game SPS in education was reported after 2005. SimSE was identified as a landmark among the simulators for SEE in the past years.

ABSTRACT Software process simulation (SPS) has been evolving over the past two decades after being introduced to the software engineering community in the 1980s. At that time the SPS technology attracted a great deal of interest from both academics and practitioners in the software process community—even to the extent of being one of the recommended techniques for achieving multiple Key Process Areas of Level 4 of the Capability Maturity Model Integration. However, in recent years, the growth of SPS seems to have slowed along with the number of reported applications in industry. This article summarizes the special panel that was held during ICSSP 2012 whose goals were to assess whether this technology remains applicable to today's software engineering projects and challenges and to point out the most beneficial opportunities for future research and industry application. Copyright © 2014 John Wiley & Sons, Ltd.

Cloud Computing has been increasingly accepted as a promising computing paradigm in industry, with one of the most common delivery models being Infrastructure as a Service (IaaS). An increasing number of providers have started to supply public IaaS services with different terminologies, definitions, and goals. As such, understanding the full scope of performance evaluation of candidate services would be crucial and beneficial for both service customers (e.g., cost-benefit analysis) and providers (e.g., direction of improvement). Given the numerous and diverse IaaS service features to be evaluated, a natural strategy is to implement different types of evaluation experiments separately. Unfortunately, it could be hard to fairly distinguish between different experimental types due to different environments and techniques that may be adopted by different evaluators. To overcome such obstacles, we have first established a novel taxonomy to help profile and clarify the nature of IaaS services performance evaluation and then built a three-layer conceptual model to generalize the existing performance evaluation practices. Using relevant elements/classifiers in the taxonomy and conceptual model, evaluators can construct natural language-style descriptions and experimental design blueprints to outline the evaluation scope and also to guide new evaluation implementations. In essence, the generated descriptions and blueprints abstractly define and characterize the actual evaluation work. This enables relatively fair and rational comparisons between different performance evaluations according to their abstract characteristics.

Benchmark suites are significant for evaluating various aspects of Cloud services from a holistic view. However, there is still a gap between using benchmark suites and achieving holistic impression of the evaluated Cloud services. Most Cloud service evaluation work intended to report individual benchmarking results without delivering summary measures. As a result, it could be still hard for customers with such evaluation reports to understand an evaluated Cloud service from a global perspective. Inspired by the boosting approaches to machine learning, we proposed the concept Boosting Metrics to represent all the potential approaches that are able to integrate a suite of benchmarking results. This paper introduces two types of preliminary boosting metrics, and demonstrates how the boosting metrics can be used to supplement primary measures of individual Cloud service features. In particular, boosting metrics can play a summary Response role in applying experimental design to Cloud services evaluation. Although the concept Boosting Metrics was refined based on our work in the Cloud Computing domain, we believe it can be easily adapted to the evaluation work of other computing paradigms.

In order to enhance the understanding of important concepts and strengthen the awareness of software process, students need to learn from their experiences in process-specific project practices. However, it's often difficult to design and carry out such practices in tertiary education environment. Typical challenges may include: 1) the difficulty to separate process-specific project practices from other (e.g., technical) practices in a software project, which may result in students paying more attention on technical aspects than process-specific aspects. 2) The limitation of a habitual technical-alone perspective may neglect concerns of other project stakeholders (e.g., project owner). We designed a special project-practicing course in Nanjing University as an attempt to solve these issues. The course is composed of 6-hour lecture and 32-hour bidding game. We found several positive results with this new education approach on process-specific practicing course. For example, it was short and flexible, which is easy to be placed in a regular semester. Besides, students were also forced to pay close attention only to process-specific aspects of the practice project. What's more, students were able to think from different perspectives, e.g., the senior management and customers.

Complexity cohesion and coupling have been recognized as prominent indicators for software quality. One characterization of software complexity is the existence of dependency relationship. Moreover, degree of dependency reflects the cohesion and coupling between software elements. Dependencies on design and implementation phase have been proven as important predictors for software bugs. We empirically investigated how requirements dependencies correlate with and predict software integration bugs, which can provide early estimate regarding software quality, therefore facilitate decision making early in the software lifecycle. We conducted network analysis on requirements dependency networks of two commercial software projects. We then performed correlation analysis between network measures (e.g., degree, closeness) and number of bugs. Afterwards, bug prediction models were built using these network measures. Significant correlation is observed between most of our network measures and number of bugs. These network measures can predict the number of bugs with high accuracy and sensitivity. We further identified the significant predictors for bug prediction. Besides, the indication effect of network measures on bug number varies among different types of requirements dependency. These observations show that requirements dependency network can be used as an early indicator of software Integration bugs.

Given an increasing number of Cloud services available in the market, evaluating candidate Cloud services is crucial and beneficial for both service customers (e.g. cost benefit analysis) and providers (e.g. direction of improvement). When it comes to performing any evaluation, a suitable methodology is inevitably required to direct experimental implementations. Nevertheless, there is still a lack of a sound methodology to guide the evaluation of Cloud services. By borrowing the lessons from evaluation of traditional computing systems, referring to the guidelines for Design of Experiments (DOE), and summarizing the existing experiences of real experimental studies, we proposed a generic Cloud Evaluation Experiment Methodology (CEEM) for Cloud services evaluation. Furthermore, we have established a pre-experimental knowledge base and specified corresponding suggestions to make this methodology more practical in the Cloud Computing domain. Through evaluating the Google AppEngine Python runtime as a preliminary validation, we show that Cloud evaluators may achieve more rational and convincing experimental results and conclusions following such an evaluation methodology.

Cloud Computing, as one of the most promising computing paradigms, has become increasingly accepted in industry. Numerous commercial providers have started to supply public Cloud services, and corresponding performance evaluation is then inevitably required for Cloud provider selection or cost-benefit analysis. Unfortunately, inaccurate and confusing evaluation implementations can be often seen in the context of commercial Cloud Computing, which could severely interfere and spoil evaluation-related comprehension and communication. This paper introduces a taxonomy to help profile and standardize the details of performance evaluation of commercial Cloud services. Through a systematic literature review, we constructed the taxonomy along two dimensions by arranging the atomic elements of Cloud-related performance evaluation. As such, this proposed taxonomy can be employed both to analyze existing evaluation practices through decomposition into elements and to design new experiments through composing elements for evaluating performance of commercial Cloud services. Moreover, through smooth expansion, we can continually adapt this taxonomy to the more general area of evaluation of Cloud Computing.

Given the diversity of commercial Cloud services, performance evaluations of candidate services would be crucial and beneficial for both service customers (e.g. cost-benefit analysis) and providers (e.g. direction of service improvement). Before an evaluation implementation, the selection of suitable factors (also called parameters or variables) plays a prerequisite role in designing evaluation experiments. However, there seems a lack of systematic approaches to factor selection for Cloud services performance evaluation. In other words, evaluators randomly and intuitively concerned experimental factors in most of the existing evaluation studies. Based on our previous taxonomy and modeling work, this paper proposes a factor framework for experimental design for performance evaluation of commercial Cloud services. This framework capsules the state-of-the-practice of performance evaluation factors that people currently take into account in the Cloud Computing domain, and in turn can help facilitate designing new experiments for evaluating Cloud services.

Background- Risks associated with a software project have the potential to affect all stakeholders. Today much software makes use of off-the-shelf (OTS) components. A better understanding of OTS-derived software risks will help to define responsibilities for these risks, and also to avoid them. Aim- Our objective is to identify, classify and compare risks of OTS-based software projects from both a software development and a software acquisition perspective. Method- To identify and classify the risks, we performed a systematic mapping study. In order to compare risks of OTS-based software development and acquisition in the real world setting, we used the mapping study results to survey occurrences of 11 shared risks in OTS-based software, in 35 OTS-based software developments and 34 OT-Sbased software acquisitions of Indonesian background. The survey is a partial replication of a previous study. Results- We identified 133 risks associated with OTS-based software development and 36 risks associated with OTS-based software acquisition. These risks are grouped into 17 risk categories. Risks occurred more frequently in software acquisition than in software development. In addition, two risks, insufficient OTS component documents and lack of provider technical support and training, frequently occurred only in the software development. Conclusions- In OTS-based projects, most risks for acquisition and development are similar. Technical-related risks are found less often in acquisition and project management related risks are found less often in development. Shared risks are perceived differently by developers and acquirers. Better understanding of actual and perceived risk in OTS-based software projects will improve risk management. Further work to validate these results is ongoing.

Background: Change propagation analysis helps predict the parts of the software that may be affected if a change is made. Existing research on change propagation focuses on design and code level changes. However, as a software evolves, the requirements that drive these changes also have intricate dependencies. Understanding the effect of these requirement dependencies on change prorogation is useful but not trivial. More than twenty requirements dependency types have been identified in the literature, however there still lacks an evaluation of the applicability of these dependency types in requirements and change propagation analysis. Aim: We aim to investigate whether these dependency types are useful for change propagation analysis. Method: We conducted a case study in a real-world industry project. This case study evaluates two representative dependency models covering twenty five types of dependencies. Results: Our initial evaluation has found that five dependency types are particularly useful in change propagation analysis and practitioners with different backgrounds have various viewpoints on change propagation. Thus change impact analysis should involve a wide range of stakeholders including project managers, requirements engineers, designers and developers. Conclusions: Our case study provides insights into requirements dependencies and their effects on change propagation analysis for both research and practice.

Background: Software systems must evolve in order to adapt in a timely fashion to the rapid changes of stakeholder needs, technologies, business environment and society regulations. Numerous studies have shown that cost, schedule or defect density of a software project may escalate as the requirements evolve. Requirements evolution management has become one important topic in requirements engineering research. Aim: To depict a holistic state-of-the-art of requirement evolution management. Method: We undertook a systematic review on requirements evolution management. Results: 125 relevant studies were identified and reviewed. This paper reports the preliminary results from this review: (1) the terminology and definition of requirements evolution; (2) fourteen key activities in requirements evolution management; (3) twenty-eight metrics of requirements evolution for three measurement goals. Conclusions: Requirements evolution is a process of continuous change of requirements in a certain direction. Most existing studies focus on how to deal with evolution after it happens. In the future, more research attention on exploring the evolution laws and predicting evolution is encouraged.

has become an important software engineering research method but costs tremendous efforts. [Aim]: This paper proposes an approach to leverage on empirically evolved ontology to support automating key SLR activities. [Method]: First, we propose an ontology, SLRONT, built on SLR experiences and best practices as a groundwork to capture common terminologies and their relationships during SLR processes; second, we present an extended version of SLRONT, the COSONT and instantiate it with the knowledge and concepts extracted from structured abstracts. Case studies illustrate the details of applying it for supporting SLR steps. [Results]: Results show that through using COSONT, we acquire the same conclusion compared with sheer manual works, but the efforts involved is significantly reduced. [Conclusions]: The approach of using ontology could effectively and efficiently support the conducting of systematic literature review. Keywords-systematic literature review; ontology; structured abstract; software cost estimation I.

The L4.verified project was a rare success in large-scale, formal verification: it provided a formal, machine-checked, code-level proof of the full functional correctness of the seL4 microkernel. In this paper we report on the development process and management issues of this project, highlighting key success factors. We formulate a detailed descriptive model of its middle-out development process, and analyze the evolution and dependencies of code and proof artifacts. We compare our key findings on verification and re-verification with insights from other verification efforts in the literature. Our analysis of the project is based on complete access to project logs, meeting notes, and version control data over its entire history, including its long-term, ongoing maintenance phase. The aim of this work is to aid understanding of how to successfully run large-scale formal software verification projects.

Handling large-sized classes and maintaining students' involvement are two of the major challenges in Personal Software Process (PSP) education in universities. In order to tackle these two challenges, we adapted and incorporated some typical practices of Pair Programming (PP) into the PSP class at summer school in Software Institute of Nanjing University in 2010, and received positive results, such as higher students' involvement and conformity of process discipline, as well as (half) workload reduction in evaluating assignments. However, the experiment did not confirm the improved performance of the paired students as expected. Based on the experience and feedbacks, we improved this approach in our PSP course in 2011. Accordingly, by analyzing the previous experiment results, we redesigned the experiment with a number of improvements, such as lab environment, evaluation methods and student selection, to further investigate the effects of this approach in PSP education, in particular students' performance. We also introduced several new metrics to enable the comparison analysis of the data collected from both paired students and solo students. The new experiment confirms the value of pairing practices in PSP education. The results show that in PSP class, compared to solo students, paired students can achieve better performance in terms of program quality and exam scores.

Software Process Simulation (SPS) has been evolving in the last decades and gained researchers' and practitioners' interests in software process community. However, the growth of SPS seems slow down in recent years, and the evidence for its benefits to industrial practice is not commonly reported as expected. SPS still awaits a breakthrough success. This special panel aims to initiate a discussion on the challenges faced SPS and the solutions to tackling the problems, and to work out a research agenda for future advance of SPS.

Software processes evolve as software too. The evolving software processes are adapted to accommodate the rapid progress and changes in software engineering practice. Software process simulation has to evolve as well in order to effectively investigate the evolving software processes. This article illustrates the evolution of software process and the resulting challenges to process simulation with two examples from experiences. From these come a list of recommendations for improving the state-of-the-practice of software process simulation.

The L4.verified project successfully completed a large-scale machine-checked formal verification at the code level of the functional correctness of the seL4 operating system microkernel. The project applied a middle-out process, which is significantly different from conventional software development processes. This paper reports a simulation model of this process; it is the first simulation model of a formal verification process. The model aims to support further understanding and investigation of the dynamic characteristics of the process and to support planning and optimization of future process enactment. We based the simulation model on a descriptive process model and information from project logs, meeting notes, and version control data over the project's history. Simulation results from the initial version of the model show the impact of complex coupling among the activities and artifacts, and frequent parallel as well as iterative work during execution. We examine some possible improvements on the formal verification process in light of the simulation results.

SUMMARY Process Management Model (PMM) and Process Simulation Model (PSM) are the critical infrastructural components of the Trustworthy Process Management Framework (TPMF), which involves a large and heterogeneous group of stakeholders in process modeling and simulation to improve process trustworthiness. Process Modeling Stakeholders (PMS) have different levels of dependency on various process modeling and simulation techniques. They may also possess different perspectives or concerns in modeling. To support trustworthy process management, this paper integrates the stakeholder‐oriented approach and hybrid simulation technique into software process modeling at three levels of abstraction (i.e., activity, sub‐process and system). The hybrid process simulation combinesmicro‐leveldiscrete process models with themacro‐levelcontinuous process models to capture process dynamics. In particular, the stakeholder‐oriented approach addresses the various perspectives of PMS during process modeling and simulation. Finally, a case study with a realistic process model demonstrates that this approach incrementally integrates stakeholders' modeling concerns through hybrid simulation, which is difficult to achieve using discrete or continuous modeling/simulation techniques independently. Copyright © 2010 John Wiley & Sons, Ltd.

SUMMARY Software process modeling and simulation have become effective tools for support of software process management and improvement over the past two decades. They have recently been integrated into the Trustworthy Process Management Framework (TPMF) as the infrastructural components to facilitate the delivery of trustworthy software products. This paper proposes the concept of Trustworthy Software Process Models as inputs to TPMF and introduces transformable process modeling for supporting effective and productive development of trustworthy process models. Furthermore, this paper undertakes an exploratory study on process model transformation by investigating and comparing process modeling semantics between quantitative (e.g., System Dynamics, SD) and qualitative forms of modeling and simulation. By following the model transformation scheme, a quantitative continuous (SD) software evolution process model is successfully transformed into its qualitative form for simulation. The results present the different capabilities and performance between these two modeling paradigms, as well as the possible benefits and interesting perspectives of transformable process modeling. Copyright © 2010 John Wiley & Sons, Ltd.

Nowadays, many universities include Personal Software Process (PSP) into their software engineering curriculum. However, delivering PSP course in tertiary education environment always faces at least two challenges. Firstly, in a typical PSP course in education environment, one teacher may teach much more students than a typical PSP class in industry, hence it is extremely difficult to provide evaluation of students' assignments in time. Secondly, participation of students in university often has significantly different characteristics compared to those trainees who had industry experiences. Based on education practice in Software Institute of Nanjing University, this paper proposed an approach to teaching PSP in tertiary education environment with higher efficiency and effectiveness. In this approach, a complete PSP course is delivered and cooperative learning (in pair) is encouraged. Besides, an evaluation team is established to provide timely evaluation on students' submissions and to help students correct their development behaviors. To validate this teaching approach, we conducted an experiment which involved all the freshman students enrolled in software engineering. We compared some process data collected from the submissions of both groups (individual and pair) of students. The results of the experiment show that the load of students' submissions reduced by half while students' interest of learning increased.

Background: Systematic Literature Reviews (SLRs) have been gaining significant attention from Software En-gineering (SE) researchers since 2004. Several researches have also working on improving the scientific and techno-logical infrastructure available to support SLRs in SE. Objective: The study reported in this paper aims to vali-date the QGS-based search process for SLR, i.e. whether a more effective and/or productive search can be achieved by following such a systematic process. Method: We used a dual-case study, in which each case includes two observations of SE literature search for the same SLR but using and not using the QGS-based approach. Results: The overall sensitivity and precision of each observation were calculated for the search cases that im-plemented different search design. Conclusions: A systematic search process (QGS-based search in this paper) appears to gain higher sensitivity and precision in the two cases of SLR. Such a search process may help capture more relevant studies as well as save re-viewers ’ time spent in literature search activities. Our ob-servations also proof that an integrated search strategy is recommended for SLRs in SE to avoid the possible limita-tions of applying single manual or automated search. 1

Recognized as one of the powerful technologies in software process engineering, Software Process Modeling (SPM) has received significant attention over the last three decades. Although empirical research plays a critical role in software engineering, the state-of-the-practice of empirical research in SPM has not been systematically reviewed. This paper serves as a status report of the assessment of empirical research in SPM by analyzing all refereed studies that were published in relevant venues from 1987 to 2008 using systematic review methodology. The primary findings indicate that in current SPM-related empirical studies, (1) software process management and improvement (SPI) was not yet the most popular primary research objectives, (2) exploratory empirical research methods, e.g., case study and action research, were dominantly used, (3) there were common issues in empirical research reports in terms of following rigorous reporting guidelines. Based on the review results, we also suggest the future needs for empirical research in SPM, in terms of research topics, SPM techniques, the strengths of research methodology and the rigors of empirical studies.

Technologies and methods for the development of embedded system projects are highly constrained by predefined hardware and software platforms. In this sense, embedded system projects may have more goals (derived from constraints) to achieve than regular software projects. Without pragmatic support, engineers from different disciplines are likely to neglect some project goals in the real-world embedded system projects. As a consequence, the success of embedded system projects may be more difficult to achieve than regular software projects. In this paper we report experiences gained during applying a goal driven project management methodology on several embedded system projects in a software company. We evaluated the effectiveness and efficiency of our Goal-Driven Development (GDD) methodology in practice by both projects results and feedbacks from relevant stakeholders. The results of our study show that GDD enables embedded system project teams to systematically and effectively identify, understand, track, and ultimately realize the project goals to meet relevant stakeholders' expectations. Being supported by GDD, explicit linkages and assignments are established between goals and solutions with project team's commitments.

BACKGROUND: Systematic Literature Reviews (SLRs) have gained significant popularity among software engineering (SE) researchers since 2004. Several researchers have also been working on improving the scientific and technological support for SLRs in SE. We argue that there is also an essential need for evidence-based body of knowledge about different aspects of the adoption of SLRs in SE. OBJECTIVE: The main objective of this research is to empirically investigate the adoption and use of SLRs in SE research from various perspectives. METHOD: We used multi-method approach as it is based on a combination of complementary research methods which are expected to compensate each others' limitations. RESULTS: A large majority of the participants are convinced of the value of using a rigorous and systematic methodology for literature reviews. However, there are concerns about the required time and resources for SLRs. One of the most important motivators for performing SLRs is new findings and inception of innovative ideas for further research. The reported SLRs are more influential compared to the traditional literature reviews in terms of number of citations. One of the main challenges of conducting SLRs is drawing a balance between rigor and required effort. CONCLUSIONS: SLR has become a popular research methodology for conducting literature review and evidence aggregation in SE. There is an overall positive perception about this methodology. The findings provide interesting insights into different aspects of SLRs. We expect that the findings can provide valuable information to readers on what can be expected from conducting SLRs and the potential impact of such reviews.

Process simulation has become a powerful technology in support of software project management and process improvement over the past decades. This research, inspired by the Impact Project, intends to investigate the technology transfer of software process simulation to the use in industrial settings, and further identify the best practices to release its full potential in software practice. We collected the reported applications of process simulation in software industry, and identified its wide adoption in the organizations delivering various software intensive systems. This paper, as an initial report of the research, briefs a historical perspective of the impact upon practice based on the documented evidence, and also elaborates the research-practice transition by examining one detailed case study. It is shown that research has a significant impact on practice in this area. The analysis of impact trace also reveals that the success of software process simulation in practice highly relies on the association with other software process techniques or practices and the close collaboration between researchers and practitioners.

This paper presents GoPoMoSA, a Goal-oriented Process Modeling and Simulation Advisor that semi-automatically discovers suitable Software Process Modeling and Simulation (SPMS) techniques for (inexperienced) process modelers to achieve their process modeling goals. GoPoMoSA takes the goal-oriented modeling approach that captures the associations among Process Modeling Stakeholder goals and existing SPMS techniques via Relevant Process Elements modeled in the knowledge graphs. We evaluated the accuracy and feasibility of GoPoMoSA with data collected from 212 published SPMS literatures and a real-world process modeling and simulation case on requirements traceability. Our results show that GoPoMoSA (1) was able to find suitable SPMS techniques based on stakeholder goals with an average of 85.38% accuracy; (2) helped novice process modelers effectively and efficiently achieve their goals.

With the research and debates on software process, the mainstream software processes can be grouped into two categories, the plan-driven (disciplined) processes and the agile processes. In terms of the classification, personal software process (PSP) is a typical plan-driven process while SCRUM is an agile-style instance. Although they are distinct from each other per se, our research found that PSP and SCRUM may also complement each other when SCRUM provides an agile process management framework, and PSP provides the skills and disciplines that a qualified team member needs to estimate, plan and manage his/her job. This paper proposes an integrated process model, SCRUM-PSP, which combines the strengths of each. We also verified that this integrated process by adopting it into a real project environment where typical agile processes are favored, i.e. change-prone requirements, rapid development, fast delivery, etc. As a result, manageability and predictability which traditional plan-driven processes usually benefit can also be achieved. The work described in this paper is a worthy attempt to embrace both process agility and discipline.

Systematic Literature Reviews (SLRs) have been gaining significant attention from software engineering researchers since 2004. Several researchers have reported their experiences of and lessons learned from applying systematic reviews to different subject matters in software engineering. However, there has been no attempt at independently exploring experiences and perceptions of the practitioners of systematic reviews in order to gain an in-depth understanding of various aspects of systemic reviews as a new research methodology in software engineering. We assert that there is a need of evidence based body of knowledge about the application of systematic reviews in software engineering. To address this need, we have started an empirical research program that aims to contribute to the growing body of knowledge about systematic reviews in software engineering. This paper reports the design, logistics, and results of the first phase empirical study carried out in this program. The results provide interesting insights into different aspects of systematic reviews based on the analysis of the data gathered from 17 interviewees with varying levels of knowledge of and experiences in systematic reviews. The findings from this study are expected to contribute to the existing knowledge about using systematic reviews and help further improve the state-of-the-practice of this research methodology in software engineering.

Software process simulation modeling (SPSM) research has increased since the first ProSim workshop held in 1998 and Kellner, Madachy and Raffo (KMR) discussed the "why, what and how" of process simulation. This paper aims to assess how SPSM has evolved during the past 10 years in particular whether the reasons for SPSM, the simulation paradigms, tools, problem domains, and model scopes have changed. We performed a systematic literature review of software process simulation papers from the ProSim series publications in the last decade. We identified 96 studies from the sources and included them in this review. The papers were categorized into four major types and data needed to address each research question was extracted. We found a need for refining the reasons and the classification scheme for SPSM introduced by KMR. More emerging SPSM paradigms and model scopes were added to enhance KMR's discussion. Trends over time showed that interest in continuous modeling was decreasing and interest in micro-processes was increasing. Hybrid models were based primarily on system dynamics and discrete event simulation and were all implemented by vertical integration. We recommend SPSM research concentrate more on recent software processes and on making SPSM more reusable and thus easier to build.

Software Process Simulation (SPS) research has increased since 1998 when the first ProSim Workshop was held. This paper aims to reveal how SPS has evolved during the past 10 years based on the preliminary results from the systematic literature review of SPS publications from 1998 to 2007. Trends over the period showed that interest in continuous modelling was decreasing and interest in micro-processes was increasing. Hybrid models were based primarily on system dynamics and discrete event simulation and were all implemented by vertical integration.

Software process improvement has been a focus of industry for many years. To assist the procedure and implementation of process improvement we provide a software process recovery method based on mining project enactment data. The goal of the method is to uncover the actual process used in order to provide input to improve the quality of a defined software process. The recovered model (or patterns) is at the same level of abstraction as the predefined process model. This provides an easy and clear way to identify the gap between the planned process model and the real enactment. We investigate the enactment of a defined software process from the view of understanding the appropriateness and fitness for purpose of the process model from the viewpoint of the project managers in the context of a small software development organization. We collected data from organizations and applied our method to a pilot case study. The main contribution of our work is to provide a software process model recovery method which supports software process change and improvement.