Comments are widely used in source code. If a comment is consistent with the code snippet it intends to annotate, it would aid code comprehension. Otherwise, Code Comment Inconsistency (CCI) is not only detrimental to the understanding of code, but more importantly, it would negatively impact the development, testing, and maintenance of software. To tackle this issue, existing research has been primarily focused on detecting inconsistencies with varied performance. It is evident that detection alone does not solve the problem; it merely paves the way for solving it. A complete solution requires detecting inconsistencies and, more importantly, rectifying them by amending comments. However, this type of work is scarce. In this paper, we contribute C4RLLaMA, a fine-tuned large language model based on the open-source CodeLLaMA. It not only has the ability to rectify inconsistencies by correcting relevant comment content but also outperforms state-of-the-art approaches in detecting inconsistencies. Experiments with various datasets confirm that C4RLLaMA consistently surpasses both post hoc and just-in-time CCI detection approaches. More importantly, C4RLLaMA outperforms substantially the only known CCI rectification approach in terms of multiple performance metrics. To further examine C4RLLaMA's efficacy in rectifying inconsistencies, we conducted a manual evaluation, and the results showed that the percentage of correct comment updates by C4RLLaMA was 65.0% and 55.9% in just-in-time and post hoc, respectively, implying C4RLLaMA's real potential in practical use.

Commit messages are valuable resources for describing why code changes are committed to repositories in version control systems (e.g., Git).They effectively help developers understand code changes and better perform software maintenance tasks.Unfortunately, developers often neglect to write high-quality commit messages in practice.Therefore, a growing body of work is proposed to generate commit messages automatically.These works all demonstrated that how to organize and represent code changes is vital in generating good commit messages, including the use of fine-grained graphs or embeddings to better represent code changes.In this study, we choose an alternative way to condense code changes before generation, i.e., proposing brief yet concise text templates consisting of the following three parts: (1) summarized code changes, (2) elicited comments, and (3) emphasized code identifiers.Specifically, we first condense code changes by using our proposed templates with the help of a heuristic-based tool named ChangeScribe, and then fine-tune CodeLlama-7B on the pairs of our proposed templates and corresponding commit messages.Our proposed templates better utilize pre-trained language models, while being naturally brief and readable to complement generated commit messages for developers.

Log statements have become an integral part of modern software systems.Prior research efforts have focused on supporting the decisions of placing log statements, such as where/what to log.With the increasing adoption of Large Language Models (LLMs) for coderelated tasks such as code completion or generation, automated approaches for generating log statements have gained much momentum.However, the performance of these approaches still has a long way to go.This paper explores enhancing the performance of LLM-based solutions for automated log statement generation by post-training LLMs with a purpose-built dataset.Thus the primary contribution is a novel approach called AUCAD, which automatically constructs such a dataset with information extracting from log-related issues.Researchers have long noticed that a significant portion of the issues in the open-source community are related to log statements.However, distilling this portion of data requires manual efforts, which is labor-intensive and costly, rendering it impractical.Utilizing our approach, we automatically extract logrelated issues from 1,537 entries of log data across 88 projects and identify 808 code snippets (i.e., methods) with retrievable source code both before and after modification of each issue (including log statements) to construct a dataset.Each entry in the dataset consists of a data pair representing high-quality and problematic log statements, respectively.With this dataset, we proceed to post-train multiple LLMs (primarily from the Llama series) for automated * Corresponding author.

As code review is a tedious and costly software quality practice, researchers have proposed several machine learning-based methods to automate the process. The primary focus has been on accuracy, that is, how accurately the algorithms are able to detect issues in the code under review. However, human intervention still remains inevitable since results produced by automated code review are not 100% correct. To assist human reviewers in making their final decisions on automatically generated review comments, the comprehensibility of the comments underpinned by accurate localization and relevant explanations for the detected issues with repair suggestions is paramount. However, this has largely been neglected in the existing research. Large language models (LLMs) have the potential to generate code review comments that are more readable and comprehensible by humans, thanks to their remarkable processing and reasoning capabilities. However, even mainstream LLMs perform poorly in detecting the presence of code issues because they have not been specifically trained for this binary classification task required in code review. In this article, we contribute Comprehensibility of Automated Code Review using Large Language Models ( Carllm ), a novel fine-tuned LLM that has the ability to improve not only the accuracy but, more importantly, the comprehensibility of automated code review, as compared to state-of-the-art pre-trained models and general LLMs.

Incompletely declared build dependencies in MAKE-based build scripts can result in incorrect or inefficient incremental builds and parallel builds for C/C++ projects. In this sense, developing MAKE-based build scripts (e.g., Makefile) is a nontrivial task, since practitioners need to manually enumerate the dependencies between the parts involved in one build, which may result in serious dependency errors such as missing dependencies or redundant dependencies. To tackle this challenge, the software engineering community has invested considerable effort in dependency error detection. However, due to issues such as incomplete or even missing static dependencies (i.e., dependencies by users declared in Makefile), existing solutions either miss certain critical dependency errors or consume significant time when parsing build dependencies, posing a major challenge to ensure both detection effectiveness and efficiency. We propose a novel approach called BuildChecker to detect the above two critical types of dependency errors in MAKE dependencies that leverages a dynamically generated build execution-declaration model to improve error detection performance and reduce detection time. We evaluate BuildChecker with state-of-the-art tools (Mkcheck, Buildfs, VeriBuild, and VirtualBuild) on 30 projects. The experimental results show that BuildChecker is able to detect a total of 13,579 dependency errors with only 29 false positives, fewer than all the state-of-the-art tools. In terms of detection efficiency, BuildChecker outperforms Buildfs by 1.38 times and Mkcheck by 66.24 times. All dependency errors had been submitted to the practitioners and maintainers of these projects. At the time of writing this article, we received responses from the maintainers of four projects, who confirmed our error reports and fixes. BuildChecker demonstrates a great potential to support practitioners effectively detect build dependency errors.

Traceability allows stakeholders to extract and comprehend the trace links among software artifacts introduced across the software life cycle, to provide significant support for software engineering tasks. Despite its proven benefits, software traceability is challenging to recover and maintain manually. Hence, plenty of approaches for automated traceability have been proposed. Most rely on textual similarities among software artifacts, such as those based on Information Retrieval (IR). However, artifacts in different abstraction levels usually have different textual descriptions, which can greatly hinder the performance of IR-based approaches (e.g., a requirement in natural language may have a small textual similarity to a Java class). In this work, we leverage the consensual biterms and transitive relationships (i.e., inner- and outer-transitive links) based on intermediate artifacts to improve IR-based traceability recovery. We first extract and filter biterms from all source, intermediate, and target artifacts. We then use the consensual biterms from the intermediate artifacts to enrich the texts of both source and target artifacts, and finally deduce outer and inner-transitive links to adjust text similarities between source and target artifacts. We conducted a comprehensive empirical evaluation based on five systems widely used in other literature to show that our approach can outperform four state-of-the-art approaches in Average Precision over 15% and Mean Average Precision over 10% on average.

Incremental and parallel builds performed by build tools such as Make are the heart of modern C/C++ software projects. Their correct and efficient execution depends on build scripts. However, build scripts are prone to errors. The most prevalent errors are missing dependencies (MDs) and redundant dependencies (RDs). The state-of-the-art methods for detecting these errors rely on clean builds (i.e., full builds of a subset of software configurations in a clean environment), which is costly and takes up to a few hours for large-scale projects. To address these challenges, we propose a novel approach called EChecker to detect build dependency errors in the context of incremental builds. The core idea of EChecker is to automatically update actual build dependencies by inferring them from C/C++ pre-processor directives and Makefile changes from new commits, which avoids clean builds when possible. EChecker achieves higher efficiency than the methods that rely on clean builds while maintaining effectiveness. We selected 12 representative projects, with their sizes ranging from small to large, with 240 commits (20 commits for each project), based on which we evaluated the effectiveness and efficiency of EChecker. We compared the evaluation results with a state-of-the-art build dependency error detection tool. The evaluation shows that the F-1 score of EChecker improved by 0.18 over the state-of-the-art method. EChecker increases the build dependency error detection efficiency by an average of 85.14 times (with a median of 16.30 times). The results demonstrate that EChecker can support practitioners in detecting build dependency errors efficiently.

Traceability plays a vital role in facilitating various software development activities by establishing the traces between different types of artifacts (e.g., issues and commits in software repositories). Among the explorations for automated traceability recovery, the IR (Information Retrieval)-based approaches leverage textual similarity to measure the likelihood of traces between artifacts and show advantages in many scenarios. However, the globalization of software development has introduced new challenges, such as the possible multilingualism on the same concept (e.g., "[SEE PDF]" vs. "attribute") in the artifact texts, thus significantly hampering the performance of IR-based approaches. Existing research has shown that machine translation can help address the term inconsistency in bilingual projects. However, the translation can also bring in synonymous terms that are not consistent with those in the bilingual projects (e.g., another translation of "[SEE PDF]" as "property"). Therefore, we propose an enhancement strategy called AVIATE that exploits translation variants from different translators by utilizing the word pairs that appear simultaneously across the translation variants from different kinds artifacts (a.k.a. consensual biterms). We use these biterms to first enrich the artifact texts, and then to enhance the calculated IR values for improving IR-based trace-ability recovery for bilingual software projects. The experiments on 17 bilingual projects (involving English and 4 other languages) demonstrate that AVIATE significantly outperformed the IR-based approach with machine translation (the state-of-the-art in this field) with an average increase of 16.67 in Average Precision (31.43%) and 8.38 (11.22%) in Mean Average Precision, indicating its effectiveness in addressing the challenges of multilingual traceability recovery.

Abstract Software process simulation models (SPSMs) that are based on descriptive process models offer the executability that can demonstrate dynamic changes of software processes over time. Verification and validation (V&V) is critical in SPSMs for guaranteeing the quality and reliability of models. V&V of dynamic software process models is more complex and challenging than for static software process models. This work systematically summarizes and maps V&V studies in SPSM to provide guidelines for future research and practice. Specifically, this study aims at identifying the focus of research on V&V, the methods used for V&V, and how to implement V&V of SPSMs in software engineering research. We conducted a systematic mapping study on studies of SPSMs that report on their V&V activities. Under the guidance of a V&V meta‐model for SPSMs, we study four research questions about V&V process. We identified 107 primary studies from a pool of 313 papers on SPSMs until 2021. There are two main results of our study. The first one presents the relationship between quality aspects of SPSMs and the V&V methods to assure them. The second result reveals the relationships among the modeling process, three modeling steps, five quality aspects, and 10 V&V methods. Generally, researchers do not pay sufficient attention to V&V, as 65.8% ( ) failed to mention or elaborate on their V&V process. We systematically summarize and map the state‐of‐the‐art V&V research in software process modeling field to support modelers' practice and improve their V&V process.

Code review is an important practice in software development. One of its main objectives is for the assurance of code quality. For this purpose, the efficacy of code review is subject to the credibility of reviewers, i.e., reviewers who have demonstrated strong evidence of previously making quality-enhancing comments are more credible than those who have not. Code reviewer recommendation (CRR) is designed to assist in recommending suitable reviewers for a specific objective and, in this context, assurance of code quality. Its performance is susceptible to the relevance of its training dataset to this objective, composed of all reviewers’ historical review comments, which, however, often contains a plethora of comments that are irrelevant to the enhancement of code quality. Furthermore, recommendation accuracy has been adopted as the sole metric to evaluate a recommender's performance, which is inadequate as it does not take reviewers’ relevant credibility into consideration. These two issues form the ground truth problem in CRR as they both originate from the relevance of dataset used to train and evaluate CRR algorithms. To tackle this problem, we first propose the concept of Quality-Enhancing Review Comments (QERC), which includes three types of comments - change-triggering inline comments, informative general comments, and approve-to-merge comments. We then devise a set of algorithms and procedures to obtain a distilled dataset by applyingQERCto the original dataset. We finally introduce a new metric – reviewer's credibility for quality enhancement (RCQE) – as a complementary metric to recommendation accuracy for evaluating the performance of recommenders. To validate the proposed QERC-based approach to CRR, we conduct empirical studies using real data from seven projects containing over 82K pull requests and 346K review comments. Results show that: (a)QERCcan effectively address the ground truth problem by distilling quality-enhancing comments from the dataset containing original code reviews, (b)QERCcan assist recommenders in finding highly credible reviewers at a slight cost of recommendation accuracy, and (c) even “wrong” recommendations using the distilled dataset are likely to be more credible than those using the original dataset.

Logs record the behavioral data of running programs and are typically generated by executing log statements. Software developers generally carry out logging practices with clear intentions and associated concerns (I&Cs). However, I&Cs may not be properly fulfilled in source code as log placement - specifically determination of a log statement's context and content - is often susceptible to an individual's profile and experience. Some industrial studies have been conducted to discern developers' main logging I&Cs and the way I&Cs are fulfilled. However, the findings are only based on the developers from a single company in each individual study and hence have limited generalizability. More importantly, there lacks a comprehensive and deep understanding of the relationships between developers' profiles and experiences and their logging practices from a wider perspective. To fill this significant gap, we conducted an empirical study using mixed methods comprising questionnaire surveys, semi-structured interviews, and code analyses with practitioners from a wide range of companies across a variety of industrial domains. Results reveal that while developers share common logging I&Cs and conduct logging practices mainly in the coding stage, their profiles and experiences profoundly influence their logging I&Cs and the way the I&Cs are fulfilled. These findings pave the way to facilitate the acceptance of important logging I&Cs and the adoption of good logging practices by developers

Abstract By adopting agile and lean practices, DevOps aims to achieve rapid value delivery by speeding up development and deployment cycles, which however lead to more security concerns that cannot be fully addressed by an isolated security role only in the final stage of development. DevSecOps promotes security as a shared responsibility integrated into the DevOps process that seamlessly intertwines development, operations, and security from the start throughout to the end of cycles. While some companies have already begun to embrace this new strategy, both industry and academia are still seeking a common understanding of the DevSecOps movement. The goal of this study is to report the state‐of‐the‐practice of DevSecOps, including the impact of DevOps on security, practitioners' understanding of DevSecOps, and the practices associated with DevSecOps as well as the challenges of implementing DevSecOps. The authors used a mixed‐methods approach for this research. The authors carried out a grey literature review on DevSecOps, and surveyed the practitioners of DevSecOps in industry of China. The status quo of DevSecOps in industry is summarized. Three major software security risks are identified with DevOps, where the establishment of DevOps pipeline provides opportunities for security‐related activities. The authors classify the interpretations of DevSecOps into three core aspects of DevSecOps capabilities, cultural enablers, and technological enablers. To materialise the interpretations into daily software production activities, the recommended DevSecOps practices from three perspectives—people, process, and technology. Although a preliminary consensus is that DevSecOps is regarded as an extension of DevOps, there is a debate on whether DevSecOps is a superfluous term. While DevSecOps is attracting an increasing attention by industry, it is still in its infancy and more effort needs to be invested to promote it in both research and industry communities.

Continuity and steadiness are vital for services with massive users, which requires the anomalies of services should be detected and resolved in a timely manner. Our previous work proposed a tool, namelyImpAPTr (Impact Analysis based on Pruning Tree), to identify the combination of multiple dimensional attributes as the clues leading to the root cause of service anomalies. However,ImpAPTrapplies a threshold driven strategy, i.e., it needs to be triggered by a$\geq 0.05\%$drop of the success rate of the service calls (abbr.SRSC), which may face problems in an atypical yet pervasive situation in field application. For example, the combination of trivial anomalies (i.e., each causes a drop less than 0.05% toSRSC) can lead to a far more than 0.05% drop onSRSC. Besides, a suitable threshold is usually hard to be determined, etc. To address these problems, we propose a new method, namelyImpAPTr+in this paper to free the constraint of the 0.05% threshold. The basic idea is to involve time dimension and identify clues across multiple time intervals of data. We performed evaluation on three typical methods (i.e.,ImpAPTr+,R-AdtributorandSqueeze) with both production environment dataset and simulation dataset. The former dataset is directly retrieved from the service monitoring data inMeituan, one of the largest on-line service providers worldwide. The latter dataset is fabricated also using the monitoring data from the same company. The results indicate: (1)ImpAPTr+outperforms previous approaches to a large degree in terms of accuracy. (2) BothImpAPTr+andR-Adtributorare able to find proper clues within seconds. (3)ImpAPTr+tends to find proper clues with shorter time intervals (i.e., less data), which implies that the method is more suitable for near real-time monitoring scenarios.

The microservice architecture has been commonly adopted by large scale software systems exemplified by a wide range of online services. Service monitoring through anomaly detection and root cause analysis (RCA) is crucial for these microservice systems to provide stable and continued services. However, compared with monolithic systems, software systems based on the layered microservice architecture are inherently complex and commonly involve entities at different levels of granularity. Therefore, for effective service monitoring, these systems have a special requirement of multi-granular RCA. Furthermore, as a large proportion of anomalies in microservice systems pertain to problematic code, to timely troubleshoot these anomalies, these systems have another special requirement of RCA at the finest code-level. Microservice systems rely on telemetry data to perform service monitoring and RCA of service anomalies. The majority of existing RCA approaches are only based on a single type of telemetry data and as a result can only support uni-granular RCA at either application-level or service-level. Although there are attempts to combine metric and tracing data in RCA, their objective is to improve RCA's efficiency or accuracy rather than to support multi-granular RCA. In this article, we propose a new RCA solutionTrinityRCLthat is able to localize the root causes of anomalies at multiple levels of granularity including application-level, service-level, host-level, and metric-level, with the unique capability of code-level localization by harnessing all three types of telemetry data to construct a causal graph representing the intricate, dynamic, and nondeterministic relationships among the various entities related to the anomalies. By implementing and deployingTrinityRCLin a real production environment, we evaluateTrinityRCLagainst two baseline methods and the results show thatTrinityRCLhas a significant performance advantage in terms of accuracy at the same level of granularity with comparable efficiency and is particularly effective to support large-scale systems with massive telemetry data.

Background:Logging practices provide the ability to record valuable runtime information of software systems to support operations tasks such as service monitoring and troubleshooting. However, current logging practices face common challenges. On the one hand, although the importance of logging practices has been broadly recognized, most of them are still conducted in an arbitrary or ad-hoc manner, ending up with questionable or inadequate support to perform these tasks. On the other hand, considerable research effort has been carried out on logging practices, however, few of the proposed techniques or methods have been widely adopted in industry.Objective:This study aims to establish a comprehensive understanding of the research state of logging practices, with a focus on unveiling possible problems and gaps which further shed light on the potential future research directions.Method:We carried out a systematic mapping study on logging practices with 56 primary studies.Results:This study provides a holistic report of the existing research on logging practices by systematically synthesizing and analyzing the focus and inter-relationship of the existing research in terms of issues, research topics and solution approaches. Using3W1H—Why to log,Where to log,What to logandHow well is the logging—as the categorization standard, we find that: (1) the best known issues in logging practices have been repeatedly investigated; (2) the issues are often studied separately without considering their intricate relationships; (3) theWhere and Whatquestions have attracted the majority of research attention while little research effort has been made on theWhyandHow wellquestions; and (4) the relationships between issues, research topics, and approaches regarding logging practices appear many-to-many, which indicates a lack of profound understanding of the issues in practice and how they should be appropriately tackled.Conclusions:This study indicates a need to advance the state of research on logging practices. For example, more research effort should be invested onwhy to logto set the anchor of logging practices as well as onhow well is the loggingto close the loop. In addition, a holistic process perspective should be taken into account in both the research and the adoption related to logging practices.

Modern code review is a critical and indispensable practice in a pull-request development paradigm that prevails in Open Source Software (OSS) development. Finding a suitable reviewer in projects with massive participants thus becomes an increasingly challenging task. Many reviewer recommendation approaches (recommenders) have been developed to support this task which apply a similar strategy, i.e. modeling the review history first then followed by predicting/recommending a reviewer based on the model. Apparently, the better the model reflects the reality in review history, the higher recommender's performance we may expect. However, one typical scenario in a pull-request development paradigm, i.e. one Pull-Request (PR) (such as a revision or addition submitted by a contributor) may have multiple reviewers and they may impact each other through publicly posted comments, has not been modeled well in existing recommenders. We adopted the hypergraph technique to model this high-order relationship (i.e. one PR with multiple reviewers herein) and developed a new recommender, namely HGRec, which is evaluated by 12 OSS projects with more than 87K PRs, 680K comments in terms of accuracy and recommendation distribution. The results indicate that HGRec outperforms the state-of-the-art recommenders on recommendation accuracy. Besides, among the top three accurate recommenders, HGRec is more likely to recommend a diversity of reviewers, which can help to relieve the core reviewers' workload congestion issue. Moreover, since HGRec is based on hypergraph, which is a natural and interpretable representation to model review history, it is easy to accommodate more types of entities and realistic relationships in modern code review scenarios. As the first attempt, this study reveals the potentials of hypergraph on advancing the pragmatic solutions for code reviewer recommendation.

Software information sites (e.g., Jira, Stack Overflow) are now wide-ly used in software development. These online platforms for collaborative development preserve a large amount of Software Engineering (SE) texts. These texts enable researchers to detect developers’ attitudes toward their daily development by analyzing the sentiments expressed in the texts. Unfortunately, recent works reported that neither off-the-shelf tools nor SE-specified tools for sentiment analysis on SE texts can provide satisfying and reliable results. In this paper, we propose to incorporate pre-trained transformer models into the sentence-classification oriented deep learning framework named TextCNN to better capture the unique expression of sentiments in SE texts. Specifically, we introduce an optimized BERT model named RoBERTa as the word embedding layer of TextCNN, along with additional residual connections between RoBERTa and TextCNN for better cooperation in our training framework. An empirical evaluation based on four datasets from different software information sites shows that our training framework can achieve overall better accuracy and generalizability than the four baselines.

Traceability approves trace links among software artifacts based on whether two artifacts are related by system functionalities. The traces are valuable for software development, but are difficult to obtain manually. To cope with the costly and fallible manual recovery, automated approaches are proposed to recover traces through textual similarities among software artifacts, such as those based on Information Retrieval (IR). However, the low quality & quantity of artifact texts negatively impact the calculated IR values, thus greatly hindering the performance of IR-based approaches. In this study, we propose to extract co-occurred word pairs from the text structures of both requirements and code (i.e., consensual biterms) to improve IR-based traceability recovery. We first collect a set of biterms based on the part-of-speech of requirement texts, and then filter them through the code texts. We then use these consensual biterms to both enrich the input corpus for IR techniques and enhance the calculations of IR values. A nine-system-based evaluation shows that in general, when solely used to enhance IR techniques, our approach can outperform pure IR-based approaches and another baseline by 21.9% & 21.8% in AP, and 9.3% & 7.2% in MAP, respectively. Moreover, when used to collaborate with another enhancing strategy from different perspectives, it can outperform this baseline by 5.9% in AP and 4.8% in MAP.

Sentiment analysis on software engineering (SE) texts has been widely used in the SE research, such as evaluating app reviews or analyzing developers' sentiments in commit messages. To better support the use of automated sentiment analysis for SE tasks, researchers built an SE-domain-specified sentiment dictionary to further improve the accuracy of the results. Unfortunately, recent work reported that current mainstream tools for sentiment analysis still cannot provide reliable results when analyzing the sentiments in SE texts. We suggest that the reason for this situation is because the way of expressing sentiments in SE texts is largely different from the way in social network or movie comments. In this paper, we propose to improve sentiment analysis in SE texts by using sentence structures, a different perspective from building a domain dictionary. Specifically, we use sentence structures to first identify whether the author is expressing her sentiment in a given clause of an SE text, and to further adjust the calculation of sentiments which are confirmed in the clause. An empirical evaluation based on four different datasets shows that our approach can outperform two dictionary-based baseline approaches, and is more generalizable compared to a learning-based baseline approach.

Blockchain and smart contracts are being embraced by more and more industrial practitioners in multiple domains including agriculture, manufacturing, and healthcare. As a distributed, immutable, and partly public ledger, the consortium blockchain demonstrates its potential to enable trustworthy interoperability and collaboration between organizations. However, the mismatch between the unruled software engineering practices and the increased interest of the consortium blockchain technology may pose threats to the quality of systems implemented. To mitigate the possible threats, this study takes the angle of software engineering to systematically understand the challenges and possible solutions in terms of developing and operating a consortium blockchain-based system. For this purpose, we conducted a case study on a typical consortium blockchain-based system and exhaustively collected the data by two rounds in-depth interviews on practitioners of different roles in the case project. Based on the data analysis, eight pairs of challenges and potential solutions were identified, which cover the phases of the development and operation of consortium blockchains. Moreover, we also captured two implications after further analysis of the findings, which worth the special attention of researchers in the near future, i.e. DevOps and microservices for blockchain or smart contracts.

Background: Logs provide crucial information to understand the dynamic behavior of software systems in modern software development and maintenance. Usually, logs are produced by log statements which will be triggered and executed under certain conditions. However, current studies paid very limited attention to developers' Intentions and Concerns (I&C) on logging practice, leading uncertainty that whether the developers' I&C are properly reflected by log statements and questionable capability to capture the expected information of system behaviors in logs. Objective: This study aims to reveal the status of developers' I&C on logging practice and more importantly, how the I&C are properly reflected in software source code in real-world software development. Method: We collected evidence from two sources of a series of interviews and source code analysis which are conducted in a big-data company, followed by consolidation and analysis of the evidence. Results: Major gaps and inconsistencies have been identified between the developers' I&C and real log statements in source code. Many code snippets contained no log statements that the interviewees claimed to have inserted. Conclusion: Developers' original I&C towards logging practice are usually poorly realized, which inevitably impacted the motivation and purpose to conduct this practice.

For many on-line systems with massive users, to provide services continuously and steadily is vital for business, which requires the anomalies of services should be located and resolved in a timely manner. As a common IT infrastructure, various APM (Application Performance Management) systems/frameworks have been adopted to monitor each call request to a service. Nevertheless, the call request may contain multidimensional attributes (e.g., City, ISP, Platform, etc.), which may further contain multiple values (e.g., ISP could be T-Mobile, CMCC, etc.). As a result, an anomaly such as DSR (Declining Success Rate) to service typically occurs with a combination of such attribute values, which creates major challenges to locate the root cause of the anomaly due to potentially huge numbers of the combinations. In this paper, we propose a novel method, ImpAPTr (Impact Analysis based on Pruning Tree), to identify the combination of dimensional attributes as the clues leading to the root cause of anomalies regarding DSR timely. In the evaluation with the simulated dataset, ImpAPTr detects valid clues in milliseconds with an accuracy of 99.37% (within the top 10 candidate results), 97.72% (top 5), and 94.51% (top 3), respectively, which outperforms previous approaches to a large degree. A field test with a production environment dataset indicates that ImpAPTr is able to detect valid clues in a few seconds.

As a common IT infrastructure, APM (Application Performance Management) systems have been widely adopted to monitor call requests to an on-line service. Usually, each request may contain multi-dimensional attributes (e.g., City, ISP, Platform, etc.), which may become the reason for a certain anomaly regarding DSR (Declining Success Rate) of service calls either solely or as a combination. Moreover, each attribute may also have multiple values (e.g., ISP could be T-Mobile, Vodafone, CMCC, etc.), rendering intricate root causes and huge challenges to identify the root causes. In this paper, we propose a prototype tool, ImpAPTr (Impact Analysis based on Pruning Tree), to identify the combination of dimensional attributes as the clues to dig out the root causes of anomalies regarding DSR of a service call in a timely manner. ImpAPTr has been evaluated in MeiTuan, one of the biggest on-line service providers. Performance regarding the accuracy outperforms several previous tools in the same field.

Context: Emerging from the agile culture, DevOps particularly emphasizes development and deployment speed to achieve rapid value delivery, which however brings some security risks to the software development process. DevSecOps is an extension of DevOps, which is considered as a means to intertwine development, operation and security. Some companies with security concerns begin to take DevSecOps into consideration when it comes to the application of DevOps. Objective: The goal of this study is to report the state-of-the-practice of DevSecOps as well as calling for academia to pay more attention to DevSecOps. Method: Using Google search engine to collect articles on DevSecOps, we conducted a Grey Literature Review (GLR) on the selected articles. Results: Whilst there exists three major software security risks in DevOps, the establishment of DevOps pipeline provides opportunities for software security activities. Based on the preliminary consensus that DevSecOps is an extension of DevOps, it is observed that the interpretations of DevSecOps can be classified into three core aspects, which are: DevSecOps capabilities, cultural enablers, and technological enablers. Furthermore, to materialize the interpretations into daily software production activities, the recommended DevSecOps practices we obtain from Grey Literature (GL) can be categorized in terms of process, infrastructure and collaboration. Conclusion: Although DevSecOps is getting increasing attention by industry, it is still in its infancy and needs to be promoted by both academia and industry.

Summary The proliferation of DevOps enables significant acceleration and automation of the delivery and deployment of massive software products. Unfortunately, the development of supporting documents that is vital for large‐scale software systems in many cases does not keep pace with the rhythm of feature delivery using DevOps in practice, which becomes the bottleneck for many software organizations to deliver full value to the customers as claimed by the DevOps. This paper proposes, implements, and evaluates an integrated approach, DevDocOps, for continuous automated documentation, in particular for DevOps. With DevDocOps, supporting documents are created along with the development process simultaneously by various roles within a DevOps project, which largely guarantees the accuracy and integrity of documents as well as significantly increases their delivery speed. Within an established delivery chain, a set of templates are created to collect and transform the required information from its origin to the target documents for delivery. A real system, iDoc, is implemented to map, collect, and synthesize the information from document templates and automate the documentation process. DevDocOps has been successfully adopted in a top‐tier global telecommunication enterprise to support more than 5000 users with different roles related to documentation. The lag time between the releases of the product version and its supporting document has been shortened from 1 to 2 months on average to less than 2 days. DevDocOps extends the scope of DevOps and enhances the value delivery by supporting continuous documentation and bridges the gap between feature delivery and document delivery with automation.

Logs are the execution results of logging statements in software systems after being triggered by various events, which is able to capture the dynamic behavior of software systems during runtime and provide important information for software analysis, e.g., issue tracking, performance monitoring, etc. Obviously, to meet this purpose, the quality of the logs is critical, which requires appropriately placement of logging statements. Existing research on this topic reveals that where to log? and what to log? are two most concerns when conducting logging practice in software development, which mainly relies on developers' personal skills, expertise and preference, rendering several problems impacting the quality of the logs inevitably. One of the reasons leading to this phenomenon might be that several recognized best practices(strategies as well) are easily neglected by software developers. Especially in those software projects with relatively large number of participants. To address this issue, we designed and implemented a plug-in tool (i.e., JLLAR) based on the Intellij IDEA, which applied machine learning technology to identify and create a set of rules reflecting commonly recognized logging practices. Based on this rule set, JLLAR can be used to scan existing source code to identify issues regarding the placement of logging statements. Moreover, JLLAR also provides automatic code completion and semi code completion (i.e., to provide recommendations) regarding logging practice to support software developers during coding.

Knowledge management nowadays usually focuses on the choice among some models or methodologies as a whole, but not on some specific, quantitative contributions of particular goals of the organization. Such a simplification misses some important chances for knowledge integration and transformation. What's worse, this simplification depresses the motivation of team members to accumulate and use the knowledge. In this paper, we propose a knowledge management framework which features in its goal-driven philosophy to manage project development, organize the knowledge and effectively integrate the knowledge management process into the development process. This method helps software project teams comprehensively and systematically identify and track knowledge management goals as far as possible. With a common framework, an organization is able to exchange knowledge and expertise within itself, which helps to glue the company together; while at the same time ensures that knowledge is shared over time so that the company benefits from past experience. Team members come to a common understanding on how to accumulate knowledge by establishing goals and corresponding solutions to meet the goals, and this consensus and clear vision on knowledge management motivates members to create knowledge and reduce the "gulf" between knowledge creation and application. It was successfully applied in several projects of different companies. The framework helps them establish an initial knowledge and experience repository. Software engineers are able to have more information available than they could understand and apply.

Background: Logging practice is a critical activity in software development, which aims to offer significant information to understand the runtime behavior of software systems and support better software maintenance. There have been many relevant studies dedicated to logging practice in software engineering recently, yet it lacks a systematic understanding to the adoption state of logging practice in industry and research progress in academia. Objective: This study aims to synthesize relevant studies on the logging practice and portray a big picture of logging practice in software engineering so as to understand current adoption status and identify research opportunities. Method: We carried out a systematic review on the relevant studies on logging practice in software engineering. Results: Our study identified 41 primary studies relevant to logging practice. Typical findings are: (1) Logging practice attracts broad interests among researchers in many concrete research areas. (2) Logging practice occurred in many development types, among which the development of fault tolerance systems is the most adopted type. (3) Many challenges exist in current logging practice in software engineering, e.g., tradeoff between logging overhead and analysis cost, where and what to log, balance between enough logging and system performance, etc. Conclusion: Results show that logging practice plays a vital role in various applications for diverse purposes. However, there are many challenges and problems to be solved. Therefore, various novel techniques are necessary to guide developers conducting logging practice and improve the performance and efficiency of logging practice.

As an emerging approach to support fast delivery of software features with reliable quality, DevOps attracts more and more practitioners and shows the potential to become one of the mainstream approach for software development and operation. Many universities begin to offer DevOps related courses to the students majored in software engineering and computer science. However, as a critical part of a DevOps course, the project practicing using DevOps might cast big challenges for teachers, compared to traditional project practicing. For example, the more frequent than ever delivery in DevOps practicing will inevitably increase the workload vastly for teachers to conduct effective evaluation. In this paper, we introduce a web based system (DevOpsEnvy) to support the management and monitoring of student teams practicing DevOps. By integrating several popular open source tools, this system provides students with features such as group management, project status monitoring and student performance data analysis, etc. Meanwhile, DevOpsEnvy system also provides teachers with sufficient evidence to perform evaluation. Our preliminary trial in Nanjing University revealed several advantages of DevOpsEnvy system.

Background: Capture-ReCapture (CRC), as a technique for post-inspection defect estimation, has been studied in Software Engineering (SE) community since 1990s. While most studies focused on the performance evaluation of various CRC models and estimators, few have been done on the assessment of the credibility of estimation results, rendering the difficulty of decision-making for quality management when applying CRC for defect estimation. Objective: This research aims to explore and investigate a reliable and practical approach to assess the credibility of CRC based defect estimation. Method: One fundamental assumption of applying CRC method is the statistical independence of samples that can be measured by 'Coefficient of CoVariation' (CCV). We applied CCV as an indicator of the statistical dependence between the observations (i.e., the defects detected by inspectors), and assessed the estimation results of CRC with the published datasets in SE literature by examining the correlation between Relative Error (RE) and CCV. Based on the observed correlation, we further propose CĈV, which replaces the unknown N (the actual number of defects) with the estimated number (N), to assess the credibility of CRC estimates. Results: We found that most datasets are with non-zero CCVs and the R2 (Coefficient of Determination) of non-linear curve-fitting for their CCVs and REs is higher than 0.8. Conclusions: Our study shows the evidence that the statistical dependence among inspectors is ubiquitous in the existing CRC-related studies. Besides, the significant correlation between CCV (by CĈV in practice) and RE may enable the possibility of the assessment of CRC-based estimation in support of quality management.

Background: The Capture-recapture (CRC) method has been adopted in software inspection post-inspection defect estimation. One outstanding advantage of the CRC method is that it is able to produce objective estimates without relying on historical data. However, a common impression about the CRC method is its poor performance regarding estimation accuracy with small inspection teams. Involving more inspectors seems to be helpful, yet no conclusive results exist on the reasonable team size in order to get acceptable CRC estimates.

Very recently, an increasing number of software companies adopted DevOps to adapt themselves to the ever-changing business environment. While it is important to mature adoption of the DevOps for these companies, no dedicated maturity models for DevOps exist. Meanwhile, maturity models such as CMMI models have demonstrated their effects in the traditional paradigm of software industry, however, it is not clear whether the CMMI models could guide the improvements with the context of DevOps. This paper reports a case study aiming at evaluating the feasibility to apply the CMMI models to guide process improvement for DevOps projects and identifying possible gaps. Using a structured method(i.e., SCAMPI C), we conducted a case study by interviewing four employees from one DevOps project. Based on evidence we collected in the case study, we managed to characterize the maturity/capability of the DevOps project, which implies the possibility to use the CMMI models to appraise the current processes in this DevOps project and guide future improvements. Meanwhile, several gaps also are identified between the CMMI models and the DevOps mode. In this sense, the CMMI models could be taken as a good foundation to design suitable maturity models so as to guide process improvement for projects adopting the DevOps.

Background: Code inspection has been accepted as an effective method to detect and remove defects and code reading is a critical step in code inspection. However, there are very limited empirical studies on the content and appropriate forms of the suitable software artifacts as the supporting materials, hence inspectors may not be well-supported with necessary knowledge to carry out code reading. Objective: This research aims to investigate the impact of different common supporting materials (i.e., comments vs. design documents) on code reading. Method: A relatively large-scale controlled experiment with 135 senior students was designed and executed to compare the impacts of different supporting materials on code reading. The subjects were randomly separated into three groups with different treatments, i.e, the comments, the design documents and the comments+design documents, respectively. Two metrics regarding the code reading performance (i.e., Effectiveness and Defect Detection Rate) were used to compare the different impacts derived from the two different types of supporting materials. Qualitative feedbacks were also collected using questionnaires for the final analysis. Results: The results indicate that students performed better when being provided with comments than comments+design documents. Also, the removal of design documents shows little impact on inspection effectiveness and may lead to an increase in defect detection rate. Conclusion: Comments may provide more help and value than design documents as supporting material in small to median sized code reading.

Context: Capture-recapture method has long been adopted in software engineering as a relatively objective way for defect estimation. While many relevant studies have been carried out to evaluate various capture-recapture models and estimators, there still lacks common understanding on the adoption status of the method in software engineering. It is necessary to systematically collect empirical evidence of Capture-recapture adoption hence form necessary understanding on the method.

Training and learning are one important purpose of Software Process Simulation (SPS). Some previous reviews showed a noticeable number of studies that combine SPS and Soft- ware Engineering Education (SEE). The objective of this research is to present the latest state-of-the-art of this area, and more importantly provide practical support for the effective adoption of SPS in educational context. We conducted an extended Systematic Literature Review (SLR) based on our previous reviews. The review identified 42 primary studies from 1992 to 2013. This paper presents the preliminary results by answering the research questions. The overall findings confirmed the positive impact of SPS on education. The detailed discussions and recommendations may offer reference value to the community.

In order to enhance the understanding of important concepts and strengthen the awareness of software process, we designed a special project-practicing course in Nanjing University as an attempt to solve typical issues in these courses (e.g., focusing on aspects of software process, participation, limited time in a regular semester, etc.). The course is composed of 6-hour lecture and 32-hour bidding game. Preliminary results indicated several advantages with this new education approach on process-specific practicing course, which we already reported on CSEE&T2013. Since this course has been delivered to students from school (less experiences) and industry (more experiences), we noticed students' different performances on this course. In this paper, we collected course results from six classes, based on a comprehensive analysis from 8 different aspects; we try to understand where “EXPERIENCE” impacts students' difference performance and benefit from the understanding to improve our education on software engineering.

Code inspection is believed to be an effective technique to remove defects and improve software quality. However, the adoption of code inspection in industry is far less than it should be, which may lead to many novice inspectors in industry. For these novice inspectors, a suitable reading technique should be of the first step to begin this quality journey. While reports indicated that Checklist-Based Reading (CBR) and Ad Hoc Reading (AHR) had been the most adopted inspection techniques in industry, we deem it is necessary to investigate these two techniques first. In this paper, we present a case study of the adoption of code reading techniques in one small-sized software company. In this study, five engineers used different techniques (i.e., CBR vs. AHR) to read source code in 20 modules. Both quantitative data and qualitative data are collected during the case study. Initial analysis of these data indicates that industrial novice inspectors using CBR tended to have a lower reading speed than those using AHR. Both techniques could help these novice inspectors to remove a certain portion of defects during code review, and compared to AHR approach, CBR may help them find larger percentage of defects. However, there still exist several issues, for example, missing large portion of review-removable defects could not be avoided for novice inspectors. What's more, CBR may limit reviewers' ability to find defects outside the checklist, and to establish effective checklist remains a big challenge for novice inspectors. Besides, both internal factors (e.g., faith in inspection to achieve high quality) as well as external factors (e.g., schedule pressure) may also impact novice inspectors to adopt code reading.

Software Process Simulation Modeling (SPSM) has raised research interest since 1980s. However, it is observed that SPSM studies published in the ICSSP community may have dropped in recent years. The objective of this research is to update the recent status of this area. We conducted a Systematic Literature Review (SLR) using the QGS-based search strategy. The review identified 74 primary studies in the past five years (2008-2012). This paper presents the preliminary results from this updated SLR by answering the first four research questions. Based on the findings from this updated review, it can be concluded that in terms of the number of SPSM studies found in the overall software engineering community, there is no significant change (drop) compared to the former review stage (1998-2007).

Software projects are performed in different contexts and, thus, require a context-specific selection and adoption of adequate methods. The suitable selection and tailoring, however, still constitute a challenging task. For this, in this paper, we discuss several issues concerning process definition and adoption, and motivate more research regarding the improvement of evidence-based method selection and adoption for the respective context.

As an approach to develop suitable development processes for software projects, Software Process Selection, Tailoring and Composition (SP-STC) attract lots of attention from both industry and academia. However, without effective guidelines, how to do SP-STC often remains a mystery. This special panel aims to 1) initiate a discussion on the current research status of SP-STC, 2) identify main challenges of SP-STC and possible solutions, and 3) work out a research agenda for future work.

With the proliferation of embedded ubiquitous systems in all aspects of human life, the development of embedded systems has been facing more and more challenges (e.g., quality, time to market, etc.). Meanwhile, lots of software processes have been reported to be applied in Embedded Systems Development (ESD) with various advantages and disadvantages. Therefore, it’s important to portrait a big picture of the state-of-the-practice of the adoption of the software processes in ESD, which may benefit both practitioners and researchers in this area. This paper presents our investigation on this topic using systematic review that is intended to: 1) identify typical challenging factors and how software processes and practices address them; and 2) discover improvement opportunities from both academic and industrial perspectives.

In order to enhance the understanding of important concepts and strengthen the awareness of software process, students need to learn from their experiences in process-specific project practices. However, it's often difficult to design and carry out such practices in tertiary education environment. Typical challenges may include: 1) the difficulty to separate process-specific project practices from other (e.g., technical) practices in a software project, which may result in students paying more attention on technical aspects than process-specific aspects. 2) The limitation of a habitual technical-alone perspective may neglect concerns of other project stakeholders (e.g., project owner). We designed a special project-practicing course in Nanjing University as an attempt to solve these issues. The course is composed of 6-hour lecture and 32-hour bidding game. We found several positive results with this new education approach on process-specific practicing course. For example, it was short and flexible, which is easy to be placed in a regular semester. Besides, students were also forced to pay close attention only to process-specific aspects of the practice project. What's more, students were able to think from different perspectives, e.g., the senior management and customers.

Code review is believed to be an effective technique to remove defects in early development stage and improve software quality. Therefore, it is regarded as one of the basic skills of qualified software engineers. Consequently, most curricula for SE students incorporated knowledge about code review in different courses. However, how to teach students to conduct efficient code review remains challenging. Many reports claimed that using checklist during code review could increase review efficiency (percentage of defects removed in code review). Nevertheless, we found a quite different result through analyzing the data collected from a PSP course took by freshmen. Results indicate that checklist contributes more to helping beginners conduct code review than to improving review efficiency. This finding implies that educators need to properly recognize the role of checklist in code review for students and explore more approaches to help students master skills to conduct efficient code reviews.

The importance of delivering software process courses to software engineering students has been more and more recognized in China in recent years. However, students usually cannot fully appreciate the value of software process courses by only learning methodology and principle in the classroom. Therefore, a process-specific project course was designed to fill the gap between the software process theoretical and experiential knowledge. But to design the course also has many challenges, such as: to provide enough guideline for students; to monitor every process task; to gather and use the process data, especially considering the large class size. We designed a summer school 6-weeks project course in Nanjing University based on TSP (Team Software Process) methodology. To support the course, we developed a supporting tool, the Advance Process Improvement Solution (APIS), which can record and use historical data, support teamwork, and provide process data to both students and teachers in real time. This paper describes the methodology, course organization, supporting tool, and evaluation in details. Based on our two years' experience, this course plays a key role for SE students to better understand software process.

Handling large-sized classes and maintaining students' involvement are two of the major challenges in Personal Software Process (PSP) education in universities. In order to tackle these two challenges, we adapted and incorporated some typical practices of Pair Programming (PP) into the PSP class at summer school in Software Institute of Nanjing University in 2010, and received positive results, such as higher students' involvement and conformity of process discipline, as well as (half) workload reduction in evaluating assignments. However, the experiment did not confirm the improved performance of the paired students as expected. Based on the experience and feedbacks, we improved this approach in our PSP course in 2011. Accordingly, by analyzing the previous experiment results, we redesigned the experiment with a number of improvements, such as lab environment, evaluation methods and student selection, to further investigate the effects of this approach in PSP education, in particular students' performance. We also introduced several new metrics to enable the comparison analysis of the data collected from both paired students and solo students. The new experiment confirms the value of pairing practices in PSP education. The results show that in PSP class, compared to solo students, paired students can achieve better performance in terms of program quality and exam scores.

Nowadays, many universities include Personal Software Process (PSP) into their software engineering curriculum. However, delivering PSP course in tertiary education environment always faces at least two challenges. Firstly, in a typical PSP course in education environment, one teacher may teach much more students than a typical PSP class in industry, hence it is extremely difficult to provide evaluation of students' assignments in time. Secondly, participation of students in university often has significantly different characteristics compared to those trainees who had industry experiences. Based on education practice in Software Institute of Nanjing University, this paper proposed an approach to teaching PSP in tertiary education environment with higher efficiency and effectiveness. In this approach, a complete PSP course is delivered and cooperative learning (in pair) is encouraged. Besides, an evaluation team is established to provide timely evaluation on students' submissions and to help students correct their development behaviors. To validate this teaching approach, we conducted an experiment which involved all the freshman students enrolled in software engineering. We compared some process data collected from the submissions of both groups (individual and pair) of students. The results of the experiment show that the load of students' submissions reduced by half while students' interest of learning increased.

Technologies and methods for the development of embedded system projects are highly constrained by predefined hardware and software platforms. In this sense, embedded system projects may have more goals (derived from constraints) to achieve than regular software projects. Without pragmatic support, engineers from different disciplines are likely to neglect some project goals in the real-world embedded system projects. As a consequence, the success of embedded system projects may be more difficult to achieve than regular software projects. In this paper we report experiences gained during applying a goal driven project management methodology on several embedded system projects in a software company. We evaluated the effectiveness and efficiency of our Goal-Driven Development (GDD) methodology in practice by both projects results and feedbacks from relevant stakeholders. The results of our study show that GDD enables embedded system project teams to systematically and effectively identify, understand, track, and ultimately realize the project goals to meet relevant stakeholders' expectations. Being supported by GDD, explicit linkages and assignments are established between goals and solutions with project team's commitments.

With the research and debates on software process, the mainstream software processes can be grouped into two categories, the plan-driven (disciplined) processes and the agile processes. In terms of the classification, personal software process (PSP) is a typical plan-driven process while SCRUM is an agile-style instance. Although they are distinct from each other per se, our research found that PSP and SCRUM may also complement each other when SCRUM provides an agile process management framework, and PSP provides the skills and disciplines that a qualified team member needs to estimate, plan and manage his/her job. This paper proposes an integrated process model, SCRUM-PSP, which combines the strengths of each. We also verified that this integrated process by adopting it into a real project environment where typical agile processes are favored, i.e. change-prone requirements, rapid development, fast delivery, etc. As a result, manageability and predictability which traditional plan-driven processes usually benefit can also be achieved. The work described in this paper is a worthy attempt to embrace both process agility and discipline.