Software defect prediction strives to improve software quality and testing efficiency by constructing predictive classification models from code attributes to enable a timely identification of fault-prone modules. Several classification models have been evaluated for this task. However, due to inconsistent findings regarding the superiority of one classifier over another and the usefulness of metric-based classification in general, more research is needed to improve convergence across studies and further advance confidence in experimental results. We consider three potential sources for bias: comparing classifiers over one or a small number of proprietary data sets, relying on accuracy indicators that are conceptually inappropriate for software defect prediction and cross-study comparisons, and, finally, limited use of statistical testing procedures to secure empirical findings. To remedy these problems, a framework for comparative software defect prediction experiments is proposed and applied in a large-scale empirical comparison of 22 classifiers over 10 public domain data sets from the NASA Metrics Data repository. Overall, an appealing degree of predictive accuracy is observed, which supports the view that metric-based classification is useful. However, our results indicate that the importance of the particular classification algorithm may be less than previously assumed since no significant performance differences could be detected among the top 17 classifiers.

Support Vector Regression is one of the promising contenders in predicting the 111 time series of the NN3 Neural Forecasting Competition. As they offer substantial degrees of freedom in the modeling process, in selecting the kernel function and its parameters, cost and epsilon parameters, issues of model parameterization and model selection arise. In lack of an established methodology or comprehensive empirical evidence on their modeling, a number of heuristics and ad-hoc rules have emerged, that result in selecting different models, which show different performance. In order to determine a lower bound for Support Vector Regression accuracy in the NN3 competition, this paper seeks to compute benchmark results using a naive methodology with a fixed parameter grid-search and exponentially increasing step sizes for radial basis function kernels, estimating 43,725 candidate models for each of the 111 time series. The naive approach attempts to mimic many of the common mistakes in model building, providing error as a lower bound to support vector regression accuracy. The in-sample results parameters are evaluated to estimate the impact of potential shortcomings in the grid search heuristic and the interaction effects of the parameters.

Recently, novel algorithms of support vector regression and neural networks have received increasing attention in time series prediction. While they offer attractive theoretical properties, they have demonstrated only mixed results within real world application domains of particular time series structures and patterns. Commonly, time series are composed of a combination of regular patterns such as levels, trends and seasonal variations. Thus, the capability of novel methods to predict basic time series patterns is of particular relevance in evaluating their initial contribution to forecasting. This paper investigates the accuracy of competing forecasting methods of NN and SVR through an exhaustive empirical comparison of alternatively tuned candidate models on 36 artificial time series. Results obtained show that SVR and NN provide comparative accuracy and robustly outperform statistical methods on selected time series patterns.