Several studies have raised concerns about the performance of estimation techniques if employed with default parameters provided by specific development toolkits, e.g., Weka. In this paper, we evaluate the impact of parameter optimization with nine different estimation techniques in the Software Development Effort Estimation (SDEE) and Software Fault Prediction (SFP) domains to provide more generic findings of the impact of parameter optimization. To this aim, we employ three datasets from the domain of SDEE (China, Maxwell, Nasa) and three different regression-based datasets from the SFP domain (Ant, Xalan, Xerces). Regarding parameter optimization, we consider four optimization algorithms from different families: Grid Search and Random Search, Simulated Annealing, and Bayesian Optimization. The estimation techniques are: Support Vector Machine, Random Forest, Classification and Regression Tree, Neural Networks, Averaged Neural Networks, k-Nearest Neighbor, Partial Least Square, MultiLayer Perceptron, and Gradient Boosting Machine. Results reveal that, with both SDEE and SFP datasets, seven out of nine estimation techniques require optimization/configuration of at least one parameter. In majority of the cases, the parameters of the employed estimation techniques are sensitive to the optimization of specific types of data. Moreover, not all the parameters need to be optimized as some of them are not sensitive to optimization.

Abstract Model validation methods (e.g., k‐fold cross‐validation) use historical data to predict how well an estimation technique (e.g., random forest) performs on the current (or future) data. Studies in the contexts of software development effort estimation (SDEE) and software fault prediction (SFP) have used and investigated different model validation methods. However, no conclusive indications to suggest which model validation method has a major impact on the prediction accuracy and stability of estimation techniques. Some studies have investigated model validation methods specific to data about either SDEE or SFP. To the best of our knowledge, there is no study in the literature, which has employed different validation methods both with SDEE and SFP data. The aim of this paper is to consider different methods (10) from the family of cross‐validation (CV) and bootstrap validation methods to identify which one contributes to obtaining a better prediction accuracy for both types of data. We also evaluate which model validation methods allow the estimation techniques to provide stable performances (i.e., with lower variance). To this aim, we present an empirical study involving six datasets from the domain of SDEE and six datasets from the SFP domain. The results reveal that repeated 10‐fold CV with SDEE and optimistic boot with SFP data are the model validation methods that provide a better prediction accuracy in a greater number of experiments than the other model validation methods. Furthermore, a model validation method can improve the prediction accuracy up to 60% with SDEE data and up to 36% when employing SFP data. The analysis also reveals that repeated fivefold CV produces more stable performances when the experiments are repeated on the same data.

Feature selection algorithms select the best and relevant set of features of the datasets which leads to an increase in the accuracy of predictions when employed with the machine learning techniques. Different feature selection algorithms are used in the domain of Software Development Effort Estimations (SDEE) and recently the use of bio-inspired feature selection algorithms got the attention of the researchers, which provided the best results in terms of the accuracy measures. In this paper, we manage to systematically evaluate and assess different bio-inspired feature selection algorithms which have been employed and investigated in the studies related to SDEE with the aim of increasing the accuracy of estimations. To the best of our knowledge, there is no Systematic Literature Review (SLR) which investigated the use of bio-inspired algorithms in SDEE. Since, the use of bio-inspired algorithms in the area of SDEE started in the late 2000, we have considered the studies published between 2007-2018. We have selected about 30 different studies from five digital libraries, i.e., IEEE explore, Springer, ScienceDirect, ACM digital library, and Google Scholar, after the filtering of inclusion/exclusion and quality assessment criteria. The main findings of our SLR are that Genetic Algorithms (GA) and Particle Swarm Optimizations (PSO) are widely used bio-inspired algorithms. Moreover, GA and PSO are the algorithms which outperform baseline estimation techniques (estimation techniques employed without any feature selection algorithms) in more number of experiments, in terms of prediction accuracy.

Abstract Machine learning (ML) techniques have been widely investigated for building prediction models, able to estimate software development effort as well as to improve the accuracy of other estimation techniques. The objective of this paper is to systematically review the recent studies which used and discussed the software effort estimation models built using ML techniques. The performed literature review is based on the empirical studies published in the time period of January 1991 to December 2017, by employing widely used guidelines. The review has selected a total of 75 primary studies after the careful filtering of inclusion/exclusion and quality assessment criteria. The performed analysis reveals that artificial neural network (ANN) as ML model, NASA as dataset, and mean magnitude of relative error (MMRE) as accuracy measure are widely used in the selected studies. ANN and support vector machine (SVM) are the two techniques which have outperformed other ML techniques in more studies. Regression techniques are the mostly used among the non‐ML techniques, which outperformed other ML techniques in about 19 studies. Moreover, SVM and regression techniques in combination are characterized by better predictions when compared with other ML and non‐ML techniques.