Programmers can make the variables easy to understand by choosing meaningful names. However, even though individual names are meaningful, a collection of them might adversely affect the code readability when their names are highly similar, such as “bottomRight” vs. “bottomHeight,” and they may cause mixing up or misreading of variables. Such a pair is referred to as a “confusing variable pair.” This paper conducts a large-scale investigation to examine the change trends of confusing variable pairs over commits, and it reports the following findings. (1) The average number of confusing variable pairs appearing in a source file is 1.4 in Java and 1.3 in Python. (2) About 67–75% of confusing variable pairs survive through commits. (3) Confusing variable pairs tend to appear in a subset of source files within a project (the median rates in Java and Python projects: 26% and 13%) and disappear from fewer files (the median rates in Java and Python projects: 6% and 2%). (4) Although the change trends do not vary among projects, some outlier projects have significantly more confusing variable pairs.

CONTEXT: Information retrieval-based bug localization (IRBL) is a promising approach for efficiently identifying buggy software modules in response to user bug reports. Supervised learning techniques were adopted to improve the bug localization performance, but they brought the cold-start problem due to insufficient training data. Recent studies focused on transfer learning techniques to utilize cross-project data. These techniques improved performance but left a question regarding better cross-project data selection. OBJECTIVE: To evaluate the effectiveness of bellwether projects, which are exemplary cross-project data better than the others for training a cross-project bug localization model. METHOD: With a bug localization method, the performance of cross-project bug localization was observed to find bellwether projects and to evaluate its effective usage for bug localization. RESULTS: One cross-project was dominantly better than the others. Also, it was often helpful to mix a bellwether project with the small available within-project data to improve the localization performance. CONCLUSION: A practical implication is to select cross-project data supported by cross-project bug localization on other projects. Mixing it with target project data is often beneficial at an early phase.

Software testing is one of the most crucial quality assurance activities, and test results are of great concern to software developers. However, the quality assurance of the test code (test case) itself also becomes critical because a poor-quality test case may fail to detect latent faults and give developers false comfort regarding the test result. A code smell threatening test code quality has been studied as “test smell.” This paper conducts an investigation of test smells in 775 Python open-source programs and reports the results of a quantitative analysis regarding whether test smells impact the fault-proneness of the product code under test. The analysis results show the following two findings. (1) When a test code has one of the reported ten kinds of test smells, the production code under test is more fault- prone than the others. (2) The fault-proneness of a production code tends to get higher when the corresponding test code has two or more different kinds of test smells-over 75% of test smell combinations showed such a trend of increasing the risk of being faulty production code.

CONTEXT: Web applications are exposed to malicious accesses through the Internet. SQL injection (SQLi) attacks are still a typical threat to web application providers. Although recent studies proposed deep learning-based SQLi attack detection models with high performance, those studies were not evaluated under the same conditions. Crucial aspects other than the predictive performance were also overlooked. OBJECTIVE: To evaluate SQLi attack detection models from multi-aspects related to its operation. METHOD: Three aspects, namely, predictive performance, detection speed, and operation costs, were applied to deep learning-based SQLi attack detection models. RESULTS: No DL-based model beaten a conventional machine learning-based model, Random Forests. CONCLUSION: Researchers must evaluate DL-based SQLi attack detection models with conventional ones with hyper-parameter tuning.

Variables are essential for handling objects and data in a program, and their names can provide helpful clues to understanding the program. Well-chosen names enhance code readability. On the other hand, ill-chosen names hinder the comprehension of the program or cause misunderstanding. Al-though a variable's name is worthy of attention, it is challenging to judge whether it is appropriate or not automatically. This paper proposes a method for checking variable names using the program slicing technique and CodeBERT to automate the assessment of variable names. Given a variable in a program, the proposed method extracts the program slice regarding the variable and masks the variable's name. Then, that method tries to predict the masked part using CodeBERT and assesses the original name's adequacy by comparing the predicted names with the original name. A case study shows that the proposed method may detect ill-chosen names with high accuracy (higher than 0.9).

CONTEXT: Functional size measurement (FSM) is a basis of project planning. FSM methods usually need experts and well-specified requirements. To relax these constraints, automated size approximation models have been developed. A recent study proposed a deep learning based model named DEEP-COSMIC-UC for COSMIC FP approximation based on use cases. Performance comparison in that study did not consider conventional machine learning techniques and contextualized word embeddings. OBJECTIVE: To evaluate the effectiveness of the deep learning based COSMIC approximation. METHOD: Empirical experiments were conducted with actual project data. Random Forests and Support Vector Regression were used as representative conventional machine learning techniques. BERT-based model was used as a representative contextualized word embedding model. RESULTS: No clear superiority of DEEP-COSMIC-UC was observed. CONCLUSION: Simple off-the-shelf models are enough at least for the dataset we used.

Software testing is an essential activity for developing and maintaining high-quality software. Unit testing with test code (test cases) is a fundamental testing activity, and developers can test their production code whenever they create or modify the code. However, such quality assurance relies on the correctness of the test code. If a test code had a flaw, it would mislead the developers about the hidden faults and prevent early detection of the faults. This paper focuses on "test smells," which may cause test code flaws in Python programs, and analyzes their changing trends over commit history (code changes) toward better Python test code management. Through an empirical data analysis of 100 open-source projects, the paper reports the following findings: (1) a few kinds of test smells constitute the majority of smells detected in the studied projects, and (2) most kinds of smells tend to increase over commits, i.e., many test smells are likely to have remained in test code as technical debt.

Abstract Variable names represent a significant source of information regarding the source code, and a successful naming of variables is key to producing readable code. Programmers often use a compound variable name by concatenating two or more words to make it more informative and enhance the code readability. While each compound variable name is descriptive, a collection of them sometimes produces “confusing” variable pairs if their names are highly similar, e.g., “shippingHeight,” vs. “shippingWeight.” A confusing variable pair would adversely affect the code readability because it can cause a misreading or mix-up of variables during the programming or code review activities. Toward automated support for enhancing code readability, this paper conducts a large-scale investigation of compound variable names in Java and Python programs. The investigation collects 116,921,127 pairs of compound-named variables from 1,876 open-source Java projects and 106,943,523 pairs of such variables from 2,427 open-source Python projects. Then, this study analyzes those variable pairs from two perspectives of name similarity: string similarity and semantic similarity. Through an evaluation study with 30 human participants, the data analyses show that both string and semantic similarity can help detect confusing variable pairs in Java and Python programs. In order to distill confusing variable pairs automatically, support tools for detecting confusing variable pairs are also developed in this study.

CONTEXT: Software defect prediction (SDP) is an active research topic to support software quality assurance (SQA) activities. It was observed that unsupervised prediction models were often competitive with supervised ones at release-level and change-level defect prediction. Fine-grained just-in-time defect prediction focuses on defective files in a change, rather than the whole change. A recent study showed that the fine-grained just-in-time defect prediction was cost-effective in terms of effort-aware performance measures. Those studies did not explore the effectiveness of supervised and unsupervised models at that finer level in terms of effort-aware performance measures. OBJECTIVE: To examine the performance of supervised and unsupervised prediction models in the context of fine-grained defect prediction in terms of effort-aware performance measures. METHOD: Experiments with a time-sensitive approach were conducted to evaluate the predictive performance of supervised and unsupervised methods proposed in past studies. Datasets from OSS projects with manually validated defect links were employed from a past study. RESULTS: The use of manually validated links led to low-performance results. No clear difference among supervised and unsupervised methods was found while CBS+, a supervised method, was the best method in terms of F-measure. Even CBS+ did not achieve reasonable performance. A non-linear learning algorithm did not help the performance improvement. CONCLUSION: No clear preference among unsupervised and supervised methods. CBS+ was the best method on average. The predictive performance was still a challenge.

Any source code of a software product (production code) is expected to be tested to ensure its correct behavior. Whenever a developer updates production code, the developer should also update or create the corresponding test code to check if the updated parts still work correctly. Such a desirable co-evolution relationship between production and test code forms a logical coupling. Although the logical coupling is detectable through an association analysis on the code repository such as Git, the detection granularity is coarse because the conventional repository is at the file level. For observing those logical couplings as precisely as possible, this paper utilizes the finer-grained, Java method-level repository (FinerGit). Then the paper proposes a metric measuring the extent to which a production method has co-evolved with test methods and conducts a case study using ten open-source projects. The results show that most Java methods (98% on average) have co-evolved with test methods, but some have not; The proposed metric helps detect those methods having the potential risk that the developers might not test adequately.

This paper examines modeling methods for applying model checking to game programs created with Unreal Engine 5 Blueprint scripting system (hereinafter UE5 Blueprint). UE5 Blueprint can visually describe game logic by combining various processing nodes, but as the size of the game grows, it becomes more difficult to find and fix bugs that prevent the game from progressing. In this paper, a formal verification technique, model checking, is used to automatically detect game logic bugs. We convert a game program created in UE5 Blueprint into an input model for the model-checker NuSMV to achieve verification by NuSMV. The proposed framework enables the automatic generation of models by formally defining the semantics of nodes. We also propose methods for data flow optimization and abstraction of variable domain for the purpose of reducing the number of states in the model. We applied the proposed method to a blueprint containing a typical flag management bug and confirmed that the bug was correctly detected by NuSMV. Furthermore, we show that the number of states can be significantly reduced by the optimization and abstraction.

CONTEXT: Just-in-Time defect prediction is to specify the suspicious code commits that might make a product cause defects. Building JIT defect prediction models require a commit history and their fixed defect records. The shortage of commits of new projects motivated research of JIT cross-project defect prediction (CPDP). CPDP approaches proposed for component-level defect prediction were barely evaluated under JIT CPDP. OBJECTIVE: To explore the effects of CPDP approaches for component-level defect prediction where JIT CPDP is adopted. METHOD: A case study was conducted through two commit dataset suites provided in past studies for JIT defect prediction. JIT defect predictions with and without 21 CPDP approaches were compared regarding the classification performance using AUC. The CPDP approaches were also compared with each other. RESULTS: Most CPDP approaches changed the prediction performance of a baseline that simply combined all CP data. A few CPDP approaches could improve the prediction performance significantly. Not a few approaches worsened the performance significantly. The results based on the two suites could specify two CPDP approaches safer than the baseline. The results were inconsistent with a previous study. CONCLUSIONS: CPDP approaches for component-level might be effective for JIT CPDP. Further evaluations were needed to bring a firm conclusion.

Once a software bug is reported, it is crucial to locate the source file causing the bug and fix it as soon as possible. To this end, there have been various studies for locating bugs with the version control system and the bug reports. AmaLgam+ is one of the most promising methods for localizing bugs. This method quantifies the degree to which a source file causes the bug from different five perspectives (metrics), and combines those values (suspicious scores) into a single integrated score of the source file. However, the method has a challenge regarding the computation time because it uses the genetic algorithm (GA) to combine the above five metrics. This paper proposes an application of the Mahalanobis distance to the suspicious score integration to overcome the above challenge. The proposed method considers the above five metrics to be a five-dimensional vector. It computes the Mahalanobis distance of the vector from the origin as an alternative integrated suspicious score. The empirical study using six open source software projects proves that the proposed method has almost the same bug localization accuracy as AmaLgam+ and can reduce the computation time by up to 98%: e.g., while AmaLgam+ took about 3.5 hours, the proposed method did it about 4 minutes.

To enhance the efficiency of software testing, researchers have studied various test case prioritization (TCP) methods. A topic model-based TCP is one of the promising methods, which expresses test cases by topic vectors and prioritizes them in the order such that the set of already-prioritized test cases have the maximum dispersion in the vector space. However, the topic model is not the only option available for vectorizing test cases. Moreover, the distance metric in the vector space and the scheme to prioritize test cases (the way to find the test case that is the farthest from the set of already-prioritized ones) also have some available options. Because the combinations of the above options have not been well-discussed in the past, this paper conducts a comparative study of 36 TCP methods, which are the combinations of (1) three vectorization methods, (2) three distance metrics, and (3) four prioritization schemes (36=3x3x4). The empirical results show the following findings. The choice of the vectorization method has a significant impact on the testing efficiency: a promising option is Doc2Vec (PVDBoW). The combination with the distance metric may also be impactful: a useful combination is Doc2Vec (PV-DBoW) and Euclidean distance. The third aspect, i.e., the choice of the scheme to find the farthest test case, is not always influential.

CONTEXT: Automated bug localization is expected to help developers reducing effort and has been studied for years. One of the major approaches for this topic is Information Retrieval (IR) based bug localization. It takes a bug report and gives a ranking list of suspicious software modules based on the text similarity. Method-level bug localization is more desirable than file-level to reduce developers' cost to examine the ranking list. A challenge is that methods have little information inside. OBJECTIVE: To examine whether method-level bug localization can be improved with file-level information around a method. METHOD: An empirical evaluation was conducted with public data sets. A bug localization method BLUiR was used as a testbed. RESULTS: We found the class name of a method contributed to improving method-level bug localization. The comments outside the method often improved the performance but sometimes degraded it. The other elements hardly improved the performance. CONCLUSIONS: It is worth to add the class name for improving method-level bug localization. The comments are also worth trying.

Abstract Programmers are familiar with local variables, and in many cases, they can freely define the local variables they use. Thus, the properties of these variables are widely diverse, and this may cause variations in the quality of code. Although variables are named in accordance with coding conventions, the following matters have not received much attention from an empirical viewpoint: automatically deciding whether a local variable is “abnormal” and determining the harmful effect of an abnormal variable. This study focuses on the trends in the name, type, and scope of local variables, then proposes the use of the Mahalanobis distance to evaluate their abnormality. The empirical study entailed collecting local variables from eight open‐source software projects, and the paper reports the following findings: (a) the trend in the variation of the names of variables according to their type; (b) the majority of variables have short names with narrow scopes, where a name is often a word or an abbreviation thereof; (c) methods with an abnormal variable are approximately 1.4 times more likely to be fault prone than methods that contain only normal variables; (d) the proposed abnormality metric can be useful in a random forest‐based fault‐prone method analysis model.

This research aims to tackle a data shift problem of software effort estimation. Cross project defect prediction approaches were found to be helpful for the same problem of software defect prediction. We examined the CPDP approaches and explored its applicability and adaptability for the problem of software effort estimation.

CONTEXT: Debugging is a labor-intensive and time-consuming activity. Automatic bug localization techniques have been proposed for reducing this effort. Among the techniques, Information retrieval (IR) based bug localization techniques take a bug report and give a rank list of source modules which are likely to cause the bug. Those techniques use a few variants of tf-idf vectorizations for bug reports and software modules though different vectorizations may give vastly different performances. OBJECTIVE: To explore the effects of vectorization methods on IR-based bug localization. METHOD: An empirical evaluation was conducted with 46 public data sets and 6 vectorization methods. BugLocator was used as a test bed. RESULTS: We found a vectorization used in BugLocator was one of the best. However, we found a better vectorization for representing software modules. CONCLUSIONS: It is worth to examine different vectorization methods for better IR-based bug localization because a preference for the methods can change as demonstrated in this study.

Comments in a source program can be helpful artifacts for program comprehension. While many comments are useful documents embedded in source programs, there are also poorly-informative comments in the real world. In order to quantitatively assess the value of comments, this paper proposes applying the Doc2Vec model to comment evaluation. Doc2Vec is a useful model for vectorizing the content of a document. In this paper, a Java method is regarded as a document, and its content is expressed as a vector. Then, two vectors corresponding to different versions of a method are prepared-the original version and the comment-erased version-, and the vector similarity between these two versions are computed. If the erased comments provided richer information for the source code, the corresponding vector would have a larger change through the comment elimination. A method having poorly-informative comments may be low-quality and might require more code modifications. This paper analyzes the relationship between the value of comments in a method and the change-proneness, using the data collected from five popular open source software projects. The results show that a method having poorly-informative comments is likely to be change-prone, i.e., such a method could not survive unscathed after release.

Giving a name to a local variable is usually a programmer's discretion. Since it depends on the programmer's preference and experience, there is a lot of individual variation which may cause a variability in the code quality such as the readability. While there have been studies on the naming of local variables in the past, a relationship of names among local variables within a method (function) has not been well-discussed. This paper focuses on a pair of local variables with similar, confusing names, e.g., "lineIndex" vs. "lineIndent." Since such local variables are confusable with each other, the presence of such a confusing pair may be related to the fault-proneness of the method. An empirical analysis for five major open source Java projects is conducted, and the following results are reported: (1) a method having a confusing variable pair is about 1.1 - 2.6 times more fault-prone than a method having only dissimilar (non-confusing) pairs; (2) the proposed metric of how confusing the local variables are is equivalent to or better than the conventional cyclomatic complexity in predicting fault-prone methods.

This paper focuses on comments written in source programs. While comments can work for improving the readability of code, i.e., the quality of programs, there have also been concerns thatcomments can be added for complicated source code in order to compensate for a lack of readability. That is to say, well-written comments might be associated with problematic parts to be refactored. This paper collected Java methods (programs) from six popular open source products, and performs analyses on words which appear in their comments. Then, the paper shows that a method having a longer comments (more words)tends to be more change-prone and would be required more fixes after their releases.

Many empirical studies have reported notable theories or methods for evaluating or predicting code quality through analyses of code repositories. This paper has yet another point of view: it focuses on "commits" rather than source code. That is to say, this paper proposes to evaluate commits themselves. When an aim of a commit is to fix a bug, there can be another preceding commit which made a reason of the bug fixing. Those commits are linked by a bug fixing-based causal relationship. Then, commits can be modeled as a directed graph model of causal relationships. This paper applies Google's PageRank algorithm to the graph modelin order to evaluate commits' influences on the others. Through an empirical study with Git repositories of six open source projects, the following factors are showed to be noteworthy:(1) the number of added files at the commit,(2) the length of commit message,(3) the experience of committing author, and (4) the number of developers who have been involved in the modified files at the commit.

Stepwise regression has often been used for variable selection of effort estimation models. However it has been criticized for inappropriate selection, and another method is recommended. We thus examined the effects of Lasso, which is one of such variable selection methods. An experiment with datasets from PROMISE repository revealed that Lasso-based selection stably selected better variables than stepwise in predictive performance. We thus concluded Lasso-based selection is preferable to stepwise regression.

In the development of a software system using UML, consistency between state machine diagrams and sequence diagrams is crucial. This study proposes a verification method for the consistency of a sequence diagram and state machine diagrams. The proposed method represents state machine diagrams and a sequence diagram as processes, and can verify the consistency by checking weak simulation of the processes. We confirms the method could detect inconsistency with an example.

Background: In effort estimation model construction, it seems effective to window training project data so that only recently finished projects are used. This is because old projects might be less representative of an organization. The past study demonstrated windowing approach works with linear regression, which is one of global models. However, this approach has not been examined with local models. Local models use subset of historical data for model construction and thus windowing approach may influence on its performance more weakly. Aim: To investigate whether windowing approach works with local models. Method: We replicated the past study with EbA. Maxwell and CSC datasets were used for an experiment. Results: Windowing approach improved predictive performance. Although the difference was insignificant in any window size, the result indicated using windowing approach has positive effect on average. Conclusions: This result contributes to understand where windowing approach works well.

Model checking is a technique that can make a verification for finite state systems absolutely automatic. We propose a method for automatic verification of fault-tolerant systems using this technique. Unlike other related work, which is tailored to specific systems, we are aimed at providing a general approach to verification of fault tolerance. The main obstacle in model checking is state explosion. To avoid the problem, we design this method so that it can use SMV, a symbolic model checking tool. Symbolic model checking can overcome the problem by expressing the state space and the transition relation by Boolean functions. Assuming that a system to be verified is specified by guarded commands, we define a modeling language suited for describing guarded command programs and propose a translation method from the modeling language to the input language of SMV. We show the results of applying the proposed method to various examples to demonstrate the usefulness.