Background: Using design metrics to predict fault-prone elements of a software design can help to focus attention on classes that need redesign and more extensive testing. However, some design metrics have been pointed out to be theoretically invalid, and the usefulness of some metrics is questioned.

CONTEXT: Several studies in effort estimation havefound that it can be effective to use only recent project data for building an effort estimation model. The generality of this timeaware approach has been explored across a variety of effort estimation model approaches, organizations and definitions of recency. However, other studies have shown that it is not alwayshelpful. A question arises: how can one tell whether the approachwould be effective for a given target project? OBJECTIVE: Toinvestigate a potential method to decide between selecting recentor all project data. METHOD: Using a single-company ISBSGdata set1 studied previously in similar research, we propose andevaluate a selection method. The method utilizes a variant ofcross-validation based on recent projects to make the decision.RESULTS: There are significant differences in the estimation accuracybetween using the proposed method and using the growingportfolio (always using all available data). The method could alsoselect the better approach on average. However, the differencein estimation accuracy between using the proposed method andalways using moving windows was not statistically significant.CONCLUSIONS: The selection method could select the betterapproach on average. The results contribute to developing amethod for suggesting a better approach for practitioners.

Context The research literature on software development projects usually assumes that effort is a good proxy for cost. Practice, however, suggests that there are circumstances in which costs and effort should be distinguished. Objectives: We determine similarities and differences between size, effort, cost, duration, and number of defects of software projects. Method: We compare two established repositories (ISBSG and EBSPM) comprising almost 700 projects from industry. Results: We demonstrate a (log)-linear relation between cost on the one hand, and size, duration and number of defects on the other. This justifies conducting linear regression for cost. We establish that ISBSG is substantially different from EBSPM, in terms of cost (cheaper) and duration (faster), and the relation between cost and effort. We show that while in ISBSG effort is the most important cost factor, this is not the case in other repositories, such as EBSPM in which size is the dominant factor. Conclusion: Practitioners and researchers alike should be cautious when drawing conclusions from a single repository.

Ensemble methods aim at combining multiple learning machines to improve the efficacy in a learning task in terms of prediction accuracy, scalability, and other measures. These methods have been applied to evolutionary machine learning techniques including learning classifier systems (LCSs). In this article, we first propose a conceptual framework that allows us to appropriately categorize ensemble‐based methods for fair comparison and highlights the gaps in the corresponding literature. The framework is generic and consists of three sequential stages: a pre‐gate stage concerned with data preparation; the member stage to account for the types of learning machines used to build the ensemble; and a post‐gate stage concerned with the methods to combine ensemble output. A taxonomy of LCSs‐based ensembles is then presented using this framework. The article then focuses on comparing LCS ensembles that use feature selection in the pre‐gate stage. An evaluation methodology is proposed to systematically analyze the performance of these methods. Specifically, random feature sampling and rough set feature selection‐based LCS ensemble methods are compared. Experimental results show that the rough set‐based approach performs significantly better than the random subspace method in terms of classification accuracy in problems with high numbers of irrelevant features. The performance of the two approaches are comparable in problems with high numbers of redundant features.

Context: Recent studies have shown that estimation accuracy can be affected by only using a window of recent projects as training data for building an effort estimation model. The idea has been extended for regression-based estimation by weighting projects differently according to their order within the window. This significantly improved the accuracy of estimation in a single-company dataset from the ISBSG repository.

BACKGROUND: Several studies in software effort estimation have found that it can be effective to use a window of recent projects as training data for building an effort estimation model. The generality of the windowing approach still remains uncertain across the variety of effort estimation approaches that are based on different theory. Recent studies have focused on the use of windows with effort estimation models based on a machine learning approach, which could make better estimates than conventional linear regression. OBJECTIVE: To investigate the effect of using a window on estimation accuracy with a machine learning-based method, Artificial Neural Networks (ANN). ANN was recently found as a popular and good performance method, and is based on a different theory from other Machine Learning-based methods used in past studies. METHOD: Using a single-company ISBSG dataset studied previously in similar research, we examine the effect of using a fixed-size windowing policy on the accuracy of estimates using ANN. RESULTS: There is a difference in the estimation accuracy between using a window and not using a window. Using windows of 50 to 120 projects reduced mean absolute errors by 5-7%. The effective range of window sizes was different from previous studies. CONCLUSIONS: Windowing significantly improves estimation accuracy with ANN. The results support past studies, in that the effective window sizes were different among estimation models. The results contribute to understanding characteristics of the windowing approach.

The International Software Benchmarking Standards Group (ISBSG) maintains a repository of data about completed software projects. A common use of the ISBSG dataset is to investigate models to estimate a software project's size, effort, duration, and cost. The aim of this paper is to determine which and to what extent variables in the ISBSG dataset have been used in software engineering to build effort estimation models. For that purpose a systematic mapping study was applied to 107 research papers, obtained after a filtering process, that were published from 2000 until the end of 2013, and which listed the independent variables used in the effort estimation models. The usage of ISBSG variables for filtering, as dependent variables, and as independent variables is described. The 20 variables (out of 71) mostly used as independent variables for effort estimation are identified and analysed in detail, with reference to the papers and types of estimation methods that used them. We propose guidelines that can help researchers make informed decisions about which ISBSG variables to select for their effort estimation models.

Abstract In construction of an effort estimation model, it seems effective to use a window of training data so that the model is trained with only recent projects. Considering the chronological order of projects within the window, and weighting projects according to their order within the window, may also affect estimation accuracy. In this study, we examined the effects of weighted moving windows on effort estimation accuracy. We compared weighted and non‐weighted moving windows under the same experimental settings. We confirmed that weighting methods significantly improved estimation accuracy in larger windows, although the methods also significantly worsened accuracy in smaller windows. This result contributes to understanding properties of moving windows. Copyright © 2014 John Wiley & Sons, Ltd.

Classification rule induction involves two main processes: finding the optimal conjuncts (attribute intervals or attribute-value pairs) and their combination (disjuncts or rules) to classify different concepts in the data. The evolutionary rule learning approaches employ an evolutionary algorithm, such as a genetic algorithm, to perform both these search operations simultaneously. This approach often leads to significant problems including population bloating and stalled evolutionary search in real-valued attribute problems, especially with higher dimensions. In this paper, we present an online evolutionary rule learning approach referred to as ERL-AID that decouples the above search processes and employs a discretization algorithm that works on the attribute space and a genetic algorithm to combine the discretized attributes into appropriate classification rules. ERL-AID applies a sliding window approach to process inputs in an online fashion. The proposed system is able to produce compact rule sets with competitive performance and could scale to higher dimensions. The experimental results show the competitiveness of our algorithm.

Effort estimation for software customisation projects is a challenging task. Similar to the COCOMO approach, an initial estimate can be based on basic project characteristics, and then modified according to various productivity drivers. The aim of this paper is to build on an initial estimation model, generated from a set of contributing factors for software customisation projects, and to improve the accuracy of the model through calibration of effort multipliers derived from customisation project characteristics. Projects from Release 12 of the ISBSG Data Repository were analysed to see which of the productivity drivers were usefully related to variations in software productivity, after other project characteristics had been taken into account. The effect on estimation accuracy of multiplying an initial estimate by each of these productivity drivers was evaluated, in terms of changes in mean and median absolute error, and mean and median magnitude of relative error. Finally, a set of useful productivity drivers is identified for customisation projects, and their effect is quantified. They are found to improve effort estimation accuracy significantly.

Software development effort estimation is an essential aspect of software project management. An effort estimation model expresses relationships between effort and factors such as organizational and project features (e.g. software functional size, and the programming language used in a project). However, software development practices and tools change over time, to environmental changes. This can affect some relationships assumed in an effort estimation model. A moving windows method (a method for treating the timing information of projects), has thus been proposed for estimation models. The moving windows method uses data from a fixed number of the most recent projects data for model construction. However, it is not clear that moving windows is the best way to handle the timing information in an estimation model. The goal of our research is to determine how best to treat timing information in constructing effort estimation models. To achieve the goal, we compared six different methods (moving windows, dummy variable of moving windows, dummy variables of equal bins, dummy variables of year, year predictor, and serial number) for treating timing data, in terms of estimation accuracy. In the experiment, we use three software development project datasets. We found that moving windows is best when the number of projects included in the dataset is not small, and dummy variable of moving windows is the best when the number is small.

To date most research in software effort estimation has not taken into account any form of chronological split when selecting projects for training and testing sets. A chronological split represents the use of a project's starting and completion dates, such that any model that estimates effort for a new project p only uses as its training set projects that were completed prior to p's starting date. Three recent studies investigated the use of chronological splits, using a type of chronological split called a moving window, which represented a subset of the most recent projects completed prior to a project p's starting date. They found some evidence in favour of using windows whenever projects were recent. These studies all defined window sizes as being fixed numbers of recent projects. In practice, we suggest that estimators are more likely to think in terms of elapsed time than the size of the data set, when deciding which projects to include in a training set. Therefore, this paper investigates the effect on accuracy when using moving windows of various durations to form training sets on which to base effort estimates. Our results show that the use of windows based on duration can affect the accuracy of estimates (in this data set, a window of about three years duration appears best), but to a lesser extent than windows based on a fixed number of projects.

With the continued growth in Air Traffic, researchers are investigating innovative ways to increase airspace capacity while maintaining safety. A key safety indicator for an airspace is its Collision Risk estimate, which is compared against a Target Level of Safety (TLS) to provide a quantitative basis for judging the safety of operations in an airspace. However this quantitative value does not give an insight into the overall collision risk picture for an airspace, and how the risk changes given the interaction of a multitude of factors such as sector/traffic characteristics and controllers actions for flow management. In this paper, we propose an evolutionary framework with multi-objective optimization to evolve collision risk of air traffic scenarios. We attempt to identify, through evolutionary mechanism, the flight events resulting from Air Traffic Controller's actions that can lead to higher collision risks, thereby identifying the contributing factors or the events leading to collision risk. Computational experiments were conducted in an hi-fidelity air traffic simulation environment with collision risk model. Results indicate that “risk-free” traffic scenarios having collision risk below TLS can become “risk-prone” by few flight events, with Climb and Turn maneuvers, specifically during entering and exiting a sector, contributing significantly to increased collision risk.

Dynamic Airspace Sectorization (DAS) is a future concept in Air Traffic Management. Its main goal is to increase airspace capacity by reshaping - thus optimizing - airspace sector boundaries based on the specifics of different air traffic situations, weather conditions and other factors. The primary objective for the optimization is to balance and reduce the workload of Air Traffic Controllers (ATCs). Many researchers have made efforts in this topic in the past years. However, air traffic changes continually, and DAS has to be adaptive to each change; be it in terms of aircraft density, dynamic routes, fleet mix, etc. Therefore, instead of sectorizing the airspace each time a change occurs, we should re-sectorize it by maintaining maximum similarities between each sectorization. In this paper, we propose a multi-objective evolutionary computation methodology to re-sectorize an airspace. We use a similarity measure between the existing sectorization and the re-sectorization as an objective to maximize during the evolution.We test the methodology with different air traffic conditions with four objective functions: minimize ATC task load standard deviation, maximize average flight sector time, maximize the minimum distance between traffic crossing points and sector boundaries, and maximize the similarity of two airspace sectorizations. Experimental results show that our re-sectorization method is able to perform airspace re-sectorization under different changes in the air traffic, while satisfying the predefined objectives.

BACKGROUND: Models for estimating software development effort are constructed using a set of historical data for training. In construction, it seems effective to use a window of training data that consists of only recently finished projects. Two previous studies evaluated the use of a window with linear regression (LR) and estimation by analogy (EbA). However, these studies were based on different datasets and thus their findings could not be compared directly. OBJECTIVE: This study investigates the effect of using a window on estimation accuracy with EbA and LR. The difference between the results with the two modeling techniques was also investigated. METHOD: We compared the effectiveness of using a window with both LR and EbA, with the same experiment settings and data. RESULTS: There is a difference in accuracy between using a window and not using a window, with both software estimation methods. However, the effect of the use of a window is weaker with EbA than with LR. CONCLUSIONS: Windowing is effective with EbA and LR. However, the degree of effectiveness is weaker with EbA than with LR. The results contribute to understand how the windowing approach interrelates with software estimation models.

What makes a combinatorial optimization problem hard? The concept of phase transition was introduced in combinatorial decision problems to explain that not all NP-Complete problems are hard, and that there exists a phase transition from solvable to unsolvable problems, within which hard problems exist. Phase transition has been studied using randomly generated problems in which variables have uniform distributions across the different constraints. Real-world problems demonstrate different distributions, however. This paper reveals the relationship between the difficulty of a 3-SAT problem and graph properties. It establishes for the first time a link between the theory of phase-transition in 3-SAT and phase transitions in complex systems and networks. This paper also addresses the question of whether the phase transition phenomenon exists for non-uniform randomly generated problems. A positive answer to this question means in principle that (1) we can generate test problems for combinatorial optimization that are not uniform; (2) we can generate test problems that resemble hard versions of real-world problems; (3) we can identify the features that we need to look for in a problem to test whether or not it is hard. We use a method that we call Adversarial Evolution (AE). In AE, an evolutionary computation method is used to generate hard problem instances by evolving solutions towards the failure of an algorithm and the phase transition region.

An efficient design of a Multi-Objective Learning Classifier System for multi-flight navigation is presented. A classifier is represented by a set of rules, which are used to simultaneously navigate all the flights in the airspace. Navigation of a flight is based on the relation of the flight with factors of the air traffic environment such as wind, storm as well as other flights. This system continually learns and refines the rules of classifiers by a multi-objective optimization algorithm - NSGAII - to discover the trade-off set of classifiers which navigate flights without any conflict, minimal distance of flying, minimal discomfort defined by storm level and the time duration of flights passing through storm areas, and minimizing total delay time of flights. We propose to detect conflicts between flights by grouping trajectory segments in 3-D (abscissa-x, ordinate-y, and time-t) boxes. The conflict detection is only implemented in a box, thus the number of conflict detection times approximates to the number of conflicts. Further, conflicts between flights are resolved using a hill climber by propagating delays in the takeoff time of conflicting flights. The advantage of the proposed system is that the classifier outputs its rules in a symbolic representation, making the overall process transparent to the user and reusable. Moreover, the system successfully discovered rules in all runs to optimize its performance.

Machine learning has become a prominent tool in various domains owing to its adaptability. However, this adaptability can be taken advantage of by an adversary to cause dysfunction of machine learning; a process known as Adversarial Learning. This paper investigates Adversarial Learning in the context of artificial neural networks. The aim is to test the hypothesis that an ensemble of neural networks trained on the same data manipulated by an adversary would be more robust than a single network. We investigate two attack types: targeted and random. We use Mahalanobis distance and covariance matrices to selected targeted attacks. The experiments use both artificial and UCI datasets. The results demonstrate that an ensemble of neural networks trained on attacked data are more robust against the attack than a single network. While many papers have demonstrated that an ensemble of neural networks is more robust against noise than a single network, the significance of the current work lies in the fact that targeted attacks are not white noise.

Evaluating conflict resolution algorithms in the air-traffic domain is a challenging task. These algorithms are usually tested using a pair of aircraft or a limited number of geometries involving multiple aircraft. Our previous work demonstrated the use of evolutionary computation for risk assessment of air-traffic conflict detection algorithms using a red-teaming (or playing the devil) approach. This paper extends our previous work to conflict resolution and investigate the effect of symmetry in the representation on the performance of the evolutionary operators.

Over the last two decades several methods have been proposed for handling functional constraints while solving nonlinear optimization problems using Evolutionary Algorithms (EA). However EAs have inherent difficulty in dealing with equality constraints. This paper presents an Agent-based Memetic Algorithm (AMA) for solving nonlinear optimization problems with equality constraints. A new learning process for agents is introduced specifically for handling the equality constraints in the evolutionary process. The basic concept is to reach a point on the equality constraint from its current position by the selected individual agents. The proposed algorithm is tested on a set of standard benchmark problems. The preliminary results show that the proposed technique works very well on those benchmark problems.

Models for estimating software development effort are commonly built and evaluated using a set of historical projects. An important question is which projects to use as training data to build the model: should it be all of them, or a subset that seems particularly relevant? One factor to consider is project age: is it best to use the entire history of past projects, or is it more appropriate in a rapidly changing world to use a window of recent projects? We investigate the effect on estimation accuracy of using a moving window, using projects from the ISBSG data set. We find that using a moving window can improve accuracy, and we make some observations about factors that influence the range of possible window sizes and the best window size.

In this paper we demonstrate our signature based detector for self-propagating worms. We use a set of worm and benign traffic traces of several endpoints to build benign and worm profiles. These profiles were arranged into separate n-ary trees. We also demonstrate our anomaly detector that was used to deal with tied matches between worm and benign trees. We analyzed the performance of each detector and also with their integration. Results show that our signature based detector can detect very high true positive. Meanwhile, the anomaly detector did not achieve high true positive. Both detectors, when used independently, suffer high false positive. However, when both detectors were integrated they maintained a high detection rate of true positive and minimized the false positive.

Learning classifier systems (LCSs) are rule-based inductive learning systems that have been widely used in the field of supervised and reinforcement learning over the last few years. This paper employs sUpervised Classifier System (UCS), a supervised learning classifier system, that was introduced in 2003 for classification tasks in data mining. We present an adaptive framework of UCS on top of a self-organized map (SOM) neural network. The overall classification problem is decomposed adaptively and in real time by the SOM into subproblems, each of which is handled by a separate UCS. The framework is also tested with replacing UCS by a feedforward artificial neural network (ANN). Experiments on several synthetic and real data sets, including a very large real data set, show that the accuracy of classifications in the proposed distributed environment is as good or better than in the nondistributed environment, and execution is faster. In general, each UCS attached to a cell in the SOM has a much smaller population size than a single UCS working on the overall problem; since each data instance is exposed to a smaller population size than in the single population approach, the throughput of the overall system increases. The experiments show that the proposed framework can decompose a problem adaptively into subproblems, maintaining or improving accuracy and increasing speed.

UCS is a supervised learning classifier system that was introduced in 2003 for classification in data mining tasks. The representation of a rule in UCS as a univariate classification rule is straightforward for a human to understand. However, the system may require a large number of rules to cover the input space. Artificial neural networks (NNs), on the other hand, normally provide a more compact representation. However, it is not a straightforward task to understand the network. In this paper, we propose a novel way to incorporate NNs into UCS. The approach offers a good compromise between compactness, expressiveness, and accuracy. By using a simple artificial NN as the classifier's action, we obtain a more compact population size, better generalization, and the same or better accuracy while maintaining a reasonable level of expressiveness. We also apply negative correlation learning (NCL) during the training of the resultant NN ensemble. NCL is shown to improve the generalization of the ensemble.

In recent years, memetic algorithms (MAs) have been proposed to enhance the performance of evolutionary algorithms by incorporating local search techniques with evolutionary algorithms' global search ability, and applied successfully to solve different type of optimization problems. This paper proposes a new memetic algorithm and then introduces an agent-based memetic algorithm (AMA), for the first time, to further enhance the ability of MA in solving constrained optimization problems. In a lattice-like environment, each of the agents represents a candidate solution of the problem. The agents are able to sense and act on the society, and their performances i.e. fitness of the solution improves through co-evolutionary adaptation of society with the individual learning of the agents. The proposed algorithm is tested on 13 benchmark problems and the experimental results show promising performance.

XCS is widely accepted as one of the most reliable Michigan-style learning classifier system for data mining. Many studies found that XCS is able to provide good generalization using a ternary representation for binary inputs as well as interval representation for continuous-valued inputs. Since distributed data mining is becoming more popular due to massive data sets spread across a network at many organizations, we have proposed an XCS system for distributed data mining called DXCS. DXCS has been tested on binary inputs. The results showed that DXCS does not only achieve as good performance as the centralized XCS system, but also reduces data transmission in the network. In this paper, we further examine DXCS with real-valued inputs in stationary and dynamic environments

XCS is a flexible system for data mining due to its ability to deal with environmental changes, learn online with little prior knowledge and evolve accurate and maximally general classifiers. In this paper, we propose DXCS which is an XCS-based distributed data mining system. A MDL metric is proposed to quantify and analyze network load, and study the balance between network load and classifier accuracy in the presence of noise. The DXCS system shows promising results.