https://doi.org/10.5220/0013441300003928

https://doi.org/10.5220/0012350100003648

Abstract Refactoring, which aims to improve the internal structure of the software systems preserving their behavior, is the most common payment strategy for technical debt (TD) by removing the code smells. There exist many studies presenting code smell detection approaches/tools or investigating their impact on quality attributes. There are also studies that focus on refactoring techniques, their relation with quality attributes, tool supports, and opportunities for them. Although there are several studies addressing the gap between refactoring and TD indicators, the empirical evidence provided is still limited. In this study, we examine the distribution of 29 refactoring types among the different projects and their relation with code smells or faults. We explore the refactoring types that are most commonly performed together and other activities performed with refactorings. We conduct a large exploratory study with automatically detected 57,528 refactorings, 37,553 smells, 27,340 faults, and 134,812 commits of 33 Java projects. Results show that some refactoring types are more commonly applied by developers. Our analysis indicates that refactorings usually remove or do not affect the code smells, and this contradicts with the previous studies. Also, the commits in which refactoring(s) is performed are three times more fault inducing than those without refactoring.

Vulnerability prediction is a data-driven process that utilizes previous vulnerability records and their associated fixes in software development projects. Vulnerability records are rarely observed compared to other defects, even in large projects, and are usually not directly linked to the related code changes in the bug tracking system. Thus, preparing a vulnerability dataset and building a predicting model is quite challenging. There exist many studies proposing software metrics-based or embedding/token-based approaches to predict software vulnerabilities over code changes. In this study, we aim to compare the performance of two different approaches in predicting code changes that induce vulnerabilities. While the first approach is based on an aggregation of software metrics, the second approach is based on embedding representation of the source code using an Abstract Syntax Tree and skip-gram techniques. We employed Deep Learning and popular Machine Learning algorithms to predict vulnerability-inducing code changes. We report our empirical analysis over code changes on the publicly available SmartSHARK dataset that we extended by adding real vulnerability data. Software metrics-based code representation method shows a better classification performance than embedding-based code representation method in terms of recall, precision and F1-Score.

Background:The existing literature in software engineering reports adverse effects of confirmation bias on software testing. Confirmation bias among software testers leads to confirmatory behavior, which is designing or executing relatively more specification consistent test cases (confirmatory behavior) than specification inconsistent test cases (disconfirmatory behavior).Objective:We aim to explore the antecedents to confirmatory and disconfirmatory behavior of software testers. Furthermore, we aim to understand why and how those antecedents lead to (dis)confirmatory behavior.Method:We follow grounded theory method for the analyses of the data collected through semi-structured interviews with twelve software testers.Results:We identified twenty antecedents to (dis)confirmatory behavior, and classified them in nine categories. Experience and Time are the two major categories. Experience is a disconfirmatory category, which also determines which behavior (confirmatory or disconfirmatory) occurs first among software testers, as an effect of other antecedents. Time Pressure is a confirmatory antecedent of the Time category. It also contributes to the confirmatory effects of antecedents of other categories.Conclusion:The disconfirmatory antecedents, especially that belong to the testing process, e.g., test suite reviews by project team members, may help circumvent the deleterious effects of confirmation bias in software testing. If a team’s resources permit, the designing and execution of a test suite could be divided among the test team members, as different perspectives of testers may help to detect more errors. The results of our study are based on a single context where dedicated testing teams focus on higher levels of testing. The study’s scope does not account for the testing performed by developers. Future work includes exploring other contexts to extend our results.

Abstract Applying change‐level software defect prediction (SDP) in practice has several challenges regarding model validation techniques, data accuracy, and prediction performance consistency. A few studies report on these challenges in an industrial context. We share our experience in integrating an SDP into an industrial context. We investigate whether an “offline” SDP could reflect its “online” (real‐life) performance, and other deployment decisions: the model re‐training process and update period. We employ an online prediction strategy by considering the actual labels of training commits at the time of prediction and compare its performance against an offline prediction. We empirically assess the online SDP's performance with various lengths of the time gap between the train and test set and model update periods. Our online SDP's performance could successfully reach its offline performance. The time gap between the train and test commits, and model update period significantly impacts the online performance by 37% and 18% in terms of probability of detection (pd), respectively. We deploy the best SDP solution (73% pd) with an 8‐month time gap and a 3‐day update period. Contextual factors may determine the model performance in practice, its consistency, and trustworthiness. As future work, we plan to investigate the reasons for fluctuations in model performance over time.

Reviews on test-driven development (TDD) studies suggest that the conflicting results reported in the literature are due to unobserved factors, such as the tasks used in the experiments, and highlight that there are very few industry experiments conducted with professionals. The goal of this study is to investigate the impact of a new factor, the chosentask, and thedevelopment approachon external quality in an industrial experimental setting with 17 professionals. The participants are junior to senior developers in programming with Java, beginner to novice in unit testing, JUnit, and they have no prior experience in TDD. The experimental design is a$2\times 2$cross-over, i.e., we use two tasks for each of the two approaches, namely TDD and incremental test-last development (ITLD). Our results reveal that bothdevelopment approachandtaskare significant factors with regards to the external quality achieved by the participants. More specifically, the participants produce higher quality code during ITLD in which splitting user stories into subtasks, coding, and testing activities are followed, compared to TDD. The results also indicate that the participants produce higher quality code during the implementation of Bowling Score Keeper, compared to that of Mars Rover API, although they perceived both tasks as of similar complexity. An interaction between thedevelopment approachandtaskcould not be observed in this experiment. We conclude that variables that have not been explored so often, such as the extent to which the task is specified in terms of smaller subtasks, and developers’ unit testing experience might be critical factors in TDD experiments. The real-world appliance of TDD and its implications on external quality still remain to be challenging unless these uncontrolled and unconsidered factors are further investigated by researchers in both academic and industrial settings.

Software vulnerabilities may lead to crucial security risks in software systems. Thus, prioritization of the vulnerabilities is an important task for security teams, and assessing how severe the vulnerabilities are would help teams during fixing and maintenance activities. We replicated a prior work which aims to predict the severity of software vulnerabilities by grouping vulnerabilities into different severity levels. We follow their approach on feature extraction using word embeddings, and on prediction model using Convolutional Neural Networks (CNNs). In addition, Long Short Term Memory (LSTM) and Extreme Gradient Boosting (XGBoost) models are used. We also extend the replicated work by aiming to predict severity scores rather than levels. We carried out two experiments for predicting severity levels and severity scores of 82,974 vulnerabilities. On predicting the severity levels, our LSTM and CNN models perform similarly with an F1 score of 0.756 F1 score and 0.752, respectively. On predicting the severity scores, LSTM, CNN and XGBoost models perform 16.14%, 17.03%, 18.91% MAPE values, respectively.

Background. SZZ algorithm has been popularly used to identify bug-inducing changes in version history. It is still limited to link a fixing change to an inducing one, when the fix constitutes of code additions only. Goal. We improve the original SZZ by proposing a way to link the code additions in a fixing change to a list of candidate inducing changes. Method. The improved version, A-SZZ, finds the code block encapsulating the new code added in a fixing change, and traces back to the historical changes of the code block. We mined the GitHub repositories of two projects, Angular.js and Vue, and ran A-SZZ to identify bug-inducing changes of code additions. We evaluated the effectiveness of A-SZZ in terms of inducing and fixing ratios, and time span between the two changes. Results. The approach works well for linking code additions with previous changes, although it still produces many false positives. Conclusions. Nearly a quarter of the files in fixing changes contain code additions only, and hence, new heuristics should be implemented to link those with inducing changes in a more efficient way.

This extended abstract summarizes an article, which has been published in the Empirical Software Engineering Journal and was selected for the Journal-First presentations at the International Conference on Software and System Process (ICSSP 2018).

Background: Writing unit tests is one of the primary activities in test-driven development. Yet, the existing reviews report few evidence supporting or refuting the effect of this development approach on test case quality. Lack of ability and skills of developers to produce sufficiently good test cases are also reported as limitations of applying test-driven development in industrial practice. Objective: We investigate the impact of test-driven development on the effectiveness of unit test cases compared to an incremental test last development in an industrial context. Method: We conducted an experiment in an industrial setting with 24 professionals. Professionals followed the two development approaches to implement the tasks. We measure unit test effectiveness in terms of mutation score. We also measure branch and method coverage of test suites to compare our results with the literature. Results: In terms of mutation score, we have found that the test cases written for a test-driven development task have a higher defect detection ability than test cases written for an incremental test-last development task. Subjects wrote test cases that cover more branches on a test-driven development task compared to the other task. However, test cases written for an incremental test-last development task cover more methods than those written for the second task. Conclusion: Our findings are different from previous studies conducted at academic settings. Professionals were able to perform more effective unit testing with test-driven development. Furthermore, we observe that the coverage measure preferred in academic studies reveal different aspects of a development approach. Our results need to be validated in larger industrial contexts.

Existing tools for monitoring the quality of codebases modified by multiple developers tend to be centralized and inflexible. These tools increase the visibility of quality by producing effective reports and visualizations when a change is made to the codebase and triggering alerts when undesirable situations occur. However, their configuration is invariably both (a) centrally managed in that individual maintainers cannot define local rules to receive customized feedback when a change occurs in a specific part of the code in which they are particularly interested, and (b) coarse-grained in that analyses cannot be turned on and off below the file level. Triggr, the tool proposed in this paper, addresses these limitations by allowing distributed, customized, and fine-grained monitoring. It is a lightweight re-implementation of our previous tool, CodeAware, which adopts the same paradigm. The tool listens on a codebase’s shared repository using an event-based approach, and can send alerts to subscribed developers based on rules defined locally by them. Triggr is open-source and available at https://github.com/lyzerk/Triggr. A demonstration video can be found at https://youtu.be/qQs9aDwXJjY.

Defect prediction studies have proposed several data-driven approaches, and recently, this field has put more emphasis on whether the people factor is associated software defects. Developer metrics can capture experience, code ownership, coding skills and techniques, and commit activities. These metrics have so far been measured at a specified snapshot of the codebase although developer's knowledge on a source module could change over time. In this paper, we propose to measure periodic developer experience with regard to contextual knowledge on files and directories. We extract periodic experience metrics capturing the previous activities of developers on source files and investigate the explanatory effect of these metrics on defects. We also use activity-based (churn) metrics to observe the performance of both metric types on defect prediction. We used two large-scale open source projects, Lucene and Jackrabbit, for model evaluation. We calculate periodic developer experience metrics and churn metrics at two granularity levels: file level and commit level. We build the models using five popular machine learning algorithms in defect prediction literature. The models with the two best performing algorithms are assessed in terms of Precision, Recall, False Positive Rate, and F-measure. The set of metrics that explains software defects the best is also identified using correlation-based feature selection method. Results show that periodic developer experience metrics extracted at file level are good merits for defect prediction, accompanied with churn. When there is not enough data to extract the contextual knowledge of developers on source files, churn metrics play an important role on defect prediction.

Background: Most of the experiments in software engineering (SE) employ students as subjects. This raises concerns about the realism of the results acquired through students and adaptability of the results to software industry. Aim: We compare students and professionals to understand how well students represent professionals as experimental subjects in SE research. Method: The comparison was made in the context of two test-driven development experiments conducted with students in an academic setting and with professionals in a software organization. We measured the code quality of several tasks implemented by both subject groups and checked whether students and professionals perform similarly in terms of code quality metrics. Results: Except for minor differences, neither of the subject groups is better than the other. Professionals produce larger, yet less complex, methods when they use their traditional development approach, whereas both subject groups perform similarly when they apply a new approach for the first time. Conclusion: Given a carefully scoped experiment on a development approach that is new to both students and professionals, similar performances are observed. Further investigation is necessary to analyze the effects of subject demographics and level of experience on the results of SE experiments.

Previous studies suggest that motivation is a critical factor in developer productivity and project outcome, i.e., software project failures are significantly associated with low motivation of software teams. Surveys with software engineers also indicate that culture can affect software development team motivation through differences in developers' motivational factors. In this study, we conduct a survey with 15 Finnish software engineers and 21 non-Finnish software engineers who live in Finland to investigate 1) the relationship between team motivation and project outcome, and 2) the factors that motivate Finnish software engineers. We compare the motivational factors found from the Finnish data with those identified in prior research with developers from four different countries. An analysis of the data from our 36 subjects indicates that project outcome is not associated with team motivation in Finland though this result contradicts prior research. There is a single motivational factor, “team work” that appears to be culturally independent. Unique team motivational factors for Finnish software engineers are found to be “the authority of project manager” and “the vision of project manager”. Our results indicate that cultural differences affect team motivation and that project managers need to consider these when working in a global environment.

Recommendation systems in software engineering (SE) should be designed to integrate evidence into practitioners experience. Bayesian networks (BNs) provide a natural statistical framework for evidence-based decision-making by incorporating an integrated summary of the available evidence and associated uncertainty (of consequences). In this study, we follow the lead of computational biology and healthcare decision-making, and investigate the applications of BNs in SE in terms of 1) main software engineering challenges addressed, 2) techniques used to learn causal relationships among variables, 3) techniques used to infer the parameters, and 4) variable types used as BN nodes. We conduct a systematic mapping study to investigate each of these four facets and compare the current usage of BNs in SE with these two domains. Subsequently, we highlight the main limitations of the usage of BNs in SE and propose a Hybrid BN to improve evidence-based decision-making in SE. In two industrial cases, we build sample hybrid BNs and evaluate their performance. The results of our empirical analyses show that hybrid BNs are powerful frameworks that combine expert knowledge with quantitative data. As researchers in SE become more aware of the underlying dynamics of BNs, the proposed models will also advance and naturally contribute to evidence based-decision-making.

We present an integrated measurement and defect prediction tool: Dione. Our tool enables organizations to measure, monitor, and control product quality through learning based defect prediction. Similar existing tools either provide data collection and analytics, or work just as a prediction engine. Therefore, companies need to deal with multiple tools with incompatible interfaces in order to deploy a complete measurement and prediction solution. Dione provides a fully integrated solution where data extraction, defect prediction and reporting steps fit seamlessly. In this paper, we present the major functionality and architectural elements of Dione followed by an overview of our demonstration.

Software defect prediction aims to reduce software testing efforts by guiding testers through the defect-prone sections of software systems. Defect predictors are widely used in organizations to predict defects in order to save time and effort as an alternative to other techniques such as manual code reviews. The application of a defect prediction model in a real-life setting is difficult because it requires software metrics and defect data from past projects to predict the defect-proneness of new projects. It is, on the other hand, very practical because it is easy to apply, can detect defects using less time and reduces the testing effort. We have built a learning-based defect prediction model for a telecommunication company during a period of one year. In this study, we have briefly explained our model, presented its pay-off and described how we have implemented the model in the company. Furthermore, we have compared the performance of our model with that of another testing strategy applied in a pilot project that implemented a new process called Team Software Process (TSP). Our results show that defect predictors can be used as supportive tools during a new process implementation, predict 75% of code defects, and decrease the testing time compared with 25% of the code defects detected through more labor-intensive strategies such as code reviews and formal checklists.

Software defect data has an imbalanced and highly skewed class distribution. The misclassification costs of two classes are not equal nor are known. It is critical to find the optimum bound, i.e. threshold, which would best separate defective and defect-free classes in software data. We have applied decision threshold optimization on Naïve Bayes classifier in order to find the optimum threshold for software defect data. ROC analyses show that decision threshold optimization significantly decreases false alarms (on the average by 11%) without changing probability of detection rates.

In this paper, we present a defect prediction model based on ensemble of classifiers, which has not been fully explored so far in this type of research. We have conducted several experiments on public datasets. Our results reveal that ensemble of classifiers considerably improve the defect detection capability compared to Naive Bayes algorithm. We also conduct a cost-benefit analysis for our ensemble, where it turns out that it is enough to inspect 32% of the code on the average, for detecting 76% of the defects.