Test case prioritisation (TCP) involves ordering and selecting the most relevant test cases to verify that the current functionality of a software system remains unaffected by code changes. Recently, TCP has been addressed by machine learning (ML), predicting the failure probability of each test case. However, software engineers may struggle to identify and implement the most suitable predictive models for TCP. As new builds adapt the test suite being tested, the model performance may decline with the incorporation of these new builds. In this study, we address these challenges by applying automated workflow composition, including algorithm selection and hyperparameter optimisation. They are considered tasks within automated machine learning (AutoML). With this aim, our proposal employs grammar-guided genetic programming as the underlying mechanism for implementing the AutoML algorithm. Our experimental results demonstrate that our approach can adapt to the particularities of the system under test, selecting the most appropriate ML pipeline and hyperparameters for each build. More importantly, our approach reduces the ML knowledge required by testers—particularly regarding the selection and tuning of algorithms—while enabling them to generate pipelines suited to successive changes in SUT builds. This research showcases the potential of AutoML in software engineering, specifically for the TCP problem.

Abstract Multi‐objective optimization problems frequently appear in many diverse research areas and application domains. Metaheuristics, as efficient techniques to solve them, need to be easily accessible to users with different expertise and programming skills. In this context, metaheuristic optimization frameworks are helpful, as they provide popular algorithms, customizable components and additional facilities to conduct experiments. Due to the broad range of available tools, this paper presents a systematic evaluation and experimental comparison of 10 frameworks, covering from multi‐purpose, consolidated tools to recent libraries specifically designed for multi‐objective optimization. The evaluation is organized around seven characteristics: search components and techniques, configuration, execution, utilities, external support and community, software implementation and performance. An analysis of code metrics and a series of experiments serves to assess the last two features. Lesson learned and open issues are also discussed as part of the comparative study. The outcomes of the evaluation process reveal a contrasted support to recent advances in multi‐objective optimization, with a lack of novel algorithms and variety of metaheuristics other than evolutionary algorithms. The experimental comparison also reports significant differences in terms of both execution time and memory usage under demanding configurations.

Abstract Knowledge extraction through machine learning techniques has been successfully applied in a large number of application domains. However, apart from the required technical knowledge and background in the application domain, it usually involves a number of time-consuming and repetitive steps. Automated machine learning (AutoML) emerged in 2014 as an attempt to mitigate these issues, making machine learning methods more practicable to both data scientists and domain experts. AutoML is a broad area encompassing a wide range of approaches aimed at addressing a diversity of tasks over the different phases of the knowledge discovery process being automated with specific techniques. To provide a big picture of the whole area, we have conducted a systematic literature review based on a proposed taxonomy that permits categorising 447 primary studies selected from a search of 31,048 papers. This review performs an extensive and rigorous analysis of the AutoML field, scrutinising how the primary studies have addressed the dimensions of the taxonomy, and identifying any gaps that remain unexplored as well as potential future trends. The analysis of these studies has yielded some intriguing findings. For instance, we have observed a significant growth in the number of publications since 2018. Additionally, it is noteworthy that the algorithm selection problem has gradually been superseded by the challenge of workflow composition, which automates more than one phase of the knowledge discovery process simultaneously. Of all the tasks in AutoML, the growth of neural architecture search is particularly noticeable.

Most software companies have extensive test suites and re-run parts of them continuously to ensure that recent changes have no adverse effects. Since test suites are costly to execute, industry needs methods for test case prioritisation (TCP). Recently, TCP methods use machine learning (ML) to exploit the information known about the system under test and its test cases. However, the value added by ML-based TCP methods should be critically assessed with respect to the cost of collecting the information. This article analyses two decades of TCP research and presents a taxonomy of 91 information attributes that have been used. The attributes are classified with respect to their information sources and the characteristics of their extraction process. Based on this taxonomy, TCP methods validated with industrial data and those applying ML are analysed in terms of information availability, attribute combination and definition of data features suitable for ML. Relying on a high number of information attributes, assuming easy access to system under test code and simplified testing environments are identified as factors that might hamper industrial applicability of ML-based TCP. The TePIA taxonomy provides a reference framework to unify terminology and evaluate alternatives considering the cost-benefit of the information attributes.

Automated test case generation has proven to be useful to reduce the usually high expenses of software testing. However, several studies have also noted the skepticism of testers regarding the comprehension of generated test suites when compared to manually designed ones. This fact suggests that involving testers in the test generation process could be helpful to increase their acceptance of automatically-produced test suites. In this paper, we propose incorporating interactive readability assessments made by a tester into EvoSuite, a widely-known evolutionary test generation tool. Our approach,InterEvo-TR, interacts with the tester at different moments during the search and shows different test cases covering the same coverage target for their subjective evaluation. The design of such an interactive approach involves a schedule of interaction, a method to diversify the selected targets, a plan to save and handle the readability values, and some mechanisms to customize the level of engagement in the revision, among other aspects. To analyze the potential and practicability of our proposal, we conduct a controlled experiment in which 39 participants, including academics, professional developers, and student collaborators, interact withInterEvo-TR. Our results show that the strategy to select and present intermediate results is effective for the purpose of readability assessment. Furthermore, the participants’ actions and responses to a questionnaire allowed us to analyze the aspects influencing test code readability and the benefits and limitations of an interactive approach in the context of test case generation, paving the way for future developments based on interactivity.

Test generation is a costly but necessary testing activity to increase the quality of software projects. Automated testing tools based on evolutionary computation principles constitute an appealing modern approach to support testing tasks. However, these tools still find difficulties to detect certain types of plausible faults in real-world projects. Besides, recent studies have shown that, in general, automatically-generated tests do not resemble those manually written and, consequently, testers are reluctant to adopt them. We observe two key issues, namely the opacity of the process and the lack of cooperation with the tester, currently hampering the acceptance of automated results. Based on these findings, we explore in this paper how the interaction between current tools and expert testers would help address the test case generation problem. More specifically, we identify a number of interaction opportunities related to the object-oriented test case design driven to boost their readability and detection power. Using EvoSuite as base implementation, we present a proof of concept focused on the possibility to integrate readability assessment of the most promising test suites into a genetic algorithm.

Search-Based Software Engineering (SBSE) has been successfully applied to automate a wide range of software development activities. Nevertheless, in those software engineering problems where human evaluation and preference are crucial, such insights have proved difficult to characterize in search, and solutions might not look natural when that is the expectation. In an attempt to address this, an increasing number of researchers have reported the incorporation of the 'human-in-the-loop' during search and interactive SBSE has attracted significant attention recently. However, reported results are fragmented over different development phases, and a great variety of novel interactive approaches and algorithmic techniques have emerged. To better integrate these results, we have performed a systematic literature review of interactive SBSE. From a total of 669 papers, 26 primary studies were identified. To enable their analysis, we formulated a classification scheme focused on four crucial aspects of interactive search, i.e., the problem formulation, search technique, interactive approach, and the empirical framework. Our intention is that the classification scheme affords a methodological approach for interactive SBSE. Lastly, as well as providing a detailed cross analysis, we identify and discuss some open issues and potential future trends for the research community.

Software architects devote substantial efforts to find the most fitting architectural description for their system, which should not only specify its structure, but is also required to meet multiple, simultaneous quality criteria. Evolutionary computation has recently demonstrated to provide insightful support during the design phase by automatically deciding how to organise internal software components and how they should interact each other. Observed from a multi-objective perspective, particular care has to be taken in order to reach an appropriate trade-off among design metrics, while providing the software engineer with diverse alternatives to choose among. However, multi-objective evolutionary algorithms may find difficulties to control both aspects and, at the same time, to explore the entire search space in depth. Under these circumstances, local search can be applied to complement the evolution by scrutinising the most promising search directions. This paper proposes two different approaches that take advantage of the benefits of local search within the multi-objective evolutionary discovery of component-based software architectures. A detailed analysis and comparative study provides interesting findings like the importance of assigning a sufficient number of evaluations to the local improvement. The way in which local search explores and compares solutions for acceptance is a relevant aspect to promote diversity during the discovery process as well.

During the design of complex systems, software architects have to deal with a tangle of abstract artefacts, measures and ideas to discover the most fitting underlying architecture. A common way to structure these systems is in terms of their interacting software components, whose composition and connections need to be properly adjusted. Its abstract and highly combinatorial nature increases the complexity of the problem. In this scenario, Search-based Software Engineering (SBSE) may serve to support this decision making process from initial analysis models, since the discovery of component-based architectures can be formulated as a challenging multiple optimisation problem, where different metrics and configurations can be applied depending on the design requirements and its specific domain. Many-objective optimisation evolutionary algorithms can provide an interesting alternative to classical multi-objective approaches. This paper presents a comparative study of five different algorithms, including an empirical analysis of their behaviour in terms of quality and variety of the returned solutions. Results are also discussed considering those aspects of concern to the expert in the decision making process, like the number and type of architectures found. The analysis of many-objectives algorithms constitutes an important challenge, since some of them have never been explored before in SBSE.

This paper proposes a novel grammar-guided genetic programming algorithm for subgroup discovery. This algorithm, called comprehensible grammar-based algorithm for subgroup discovery (CGBA-SD), combines the requirements of discovering comprehensible rules with the ability to mine expressive and flexible solutions owing to the use of a context-free grammar. Each rule is represented as a derivation tree that shows a solution described using the language denoted by the grammar. The algorithm includes mechanisms to adapt the diversity of the population by self-adapting the probabilities of recombination and mutation. We compare the approach with existing evolutionary and classic subgroup discovery algorithms. CGBA-SD appears to be a very promising algorithm that discovers comprehensible subgroups and behaves better than other algorithms as measures by complexity, interest, and precision indicate. The results obtained were validated by means of a series of nonparametric tests.

As the complexity of open distributed systems grows, the need to rely on concepts and notations for expressing and structuring their specifications becomes essential. However, having concepts and notations is not enough. The large size and complexity of the set of models that constitute the system specifications also forces the need to have tools for properly creating, handling, validating and maintaining these large models. This paper presents xODP, a tool for the model-based specification of open distributed systems according to the Reference Model of Open Distributed Processing (RM-ODP) standards. Using the UML4ODP notation, it provides edition and validation facilities for ODP viewpoint models and for the correspondences between them, as well as early prototyping and execution of the ODP computational models.

This paper presents VisualJCLEC, a visual framework based on JCLEC for Evolutionary Computing. In order to have a high degree of adaptability, the architecture and pattern design followed are focused on enhancing the f exibility and scalability. For illustrative purposes, a case study of an optimization classical problem (the knapsack problem) using this framework is presented, as well as some guidelines on how to add new elements to the environment by means of CDL descriptors.

This paper presents a free-parameter grammar-guided genetic programming algorithm for mining association rules. This algorithm uses a contex-free grammar to represent individuals, encoding the solutions in a tree-shape conformant to the grammar, so they are more expressive and flexible. The algorithm here presented has the advantages of using evolutionary algorithms for mining association rules, and it also solves the problem of tuning the huge number of parameters required by these algorithms. The main feature of this algorithm is the small number of parameters required, providing the possibility of discovering association rules in an easy way for non-expert users. We compare our approach to existing evolutionary and exhaustive search algorithms, obtaining important results and overcoming the drawbacks of both exhaustive search and evolutionary algorithms. The experimental stage reveals that this approach discovers frequent and reliable rules without a parameter tuning.

Classification in imbalanced domains is a challenging task, since most of its real domain applications present skewed distributions of data. However, there are still some open issues in this kind of problem. This paper presents a multi-objective grammar-based ant programming algorithm for imbalanced classification, capable of addressing this task from both the binary and multiclass sides, unlike most of the solutions presented so far. We carry out two experimental studies comparing our algorithm against binary and multiclass solutions, demonstrating that it achieves an excellent performance for both binary and multiclass imbalanced data sets.

This paper presents a method for extracting association rules by means of a multi-objective grammar guided ant programming algorithm. Solution construction is guided by a context-free grammar specifically suited for association rule mining, which defines the search space of all possible expressions or programs. Evaluation of individuals is considered from a Pareto-based point of view, measuring support and confidence of rules mined, and assigning them a ranking fitness. The proposed algorithm is verified over 10 varied data sets and compared to other association rule mining algorithms from several paradigms such as exhaustive search, genetic algorithms and genetic programming, showing that ant programming is a good technique at addressing the association task of data mining as well.

The extraction of comprehensible knowledge is one of the major challenges in many domains. In this paper, an ant programming (AP) framework, which is capable of mining classification rules easily comprehensible by humans, and, therefore, capable of supporting expert-domain decisions, is presented. The algorithm proposed, called grammar based ant programming (GBAP), is the first AP algorithm developed for the extraction of classification rules, and it is guided by a context-free grammar that ensures the creation of new valid individuals. To compute the transition probability of each available movement, this new model introduces the use of two complementary heuristic functions, instead of just one, as typical ant-based algorithms do. The selection of a consequent for each rule mined and the selection of the rules that make up the classifier are based on the use of a niching approach. The performance of GBAP is compared against other classification techniques on 18 varied data sets. Experimental results show that our approach produces comprehensible rules and competitive or better accuracy values than those achieved by the other classification algorithms compared with it.

This paper presents the G3PARM algorithm for mining representative association rules. G3PARM is an evolutionary algorithm that uses G3P (Grammar Guided Genetic Programming) and an auxiliary population made up of its best individuals who will then act as parents for the next generation. Due to the nature of G3P, the G3PARM algorithm allows us to obtain valid individuals by defining them through a context-free grammar and, furthermore, this algorithm is generic with respect to data type. We compare our algorithm to two multiobjective algorithms frequently used in literature and known as NSGA2 (Non dominated Sort Genetic Algorithm) and SPEA2 (Strength Pareto Evolutionary Algorithm) and demonstrate the efficiency of our algorithm in terms of running-time, coverage and average support, providing the user with high representative rules.

This paper focuses on the application of a new ACO-based automatic programming algorithm to the classification task of data mining. This new model, called GBAP algorithm, is based on a context-free grammar that properly guides the creation of new valid individuals. Moreover, its most differentiating factors, such as the use of two complementary heuristic measures for every transition rule, as well as the way it assigns a consequent and evaluates the extracted rules, are also discussed. These features enhance the final rule compilation from the output classifier. The performance of the proposed algorithm is evaluated and compared against other top algorithms, and the results obtained over 17 diverse data sets show that our approach reaches pretty competitive and even better accuracy values than those resulting from the other algorithms considered in the experimentation.

This work presents a novel proposal for incremental intruder detection in collaborative recommender systems. We explore the use of rare association rule mining to reveal the existence of a suspected raid of attackers that would alter the normal behaviour of a rating-based system. In this position paper we have extended our previous G3PARM algorithm, which has already proven to serve as a solid method for extracting frequent association rules. G3PARM is an evolutionary algorithm that uses G3P (Grammar Guided Genetic Programming), which provides expressiveness and flexibility enough to adapt and apply the base context-free grammar to each specific problem or domain. We fully outline, moreover, the complete exploration and detection model, which includes some further post-analysis steps. Finally, as a proof of concept, we validate the scalability, efficiency and accuracy of our proposal showing the results obtained when different malicious intruders want to attack an on line recommender system.

Viewpoint modeling is an effective technique for specifying complex software systems in terms of a set of independent viewpoints and correspondences between them. Each viewpoint focuses on a particular aspect of the system, abstracting away from the rest of the concerns. Correspondences specify the relationships between the elements in different views, together with the constraints that guarantee the consistency among these elements. However, most Architectural Frameworks, which follow a multi-viewpoint approach, either do not consider the explicit specification of correspondences, or do it in a very simplistic way. This paper proposes a generic model-driven approach to the specification and realization of correspondences between viewpoints. In particular, we show how correspondences can be modeled both extensionally and intensionally, and propose the use of model transformations to connect these two approaches. As a proof-of-concept, we show how our proposal can be implemented in the context of the RM-ODP and UML4ODP, and present a tool to support the realization of correspondences between ODP views. This proposal can be extended to any other Architectural Framework that uses models to represent their views.

Distributed Software Engineering (DSE) concepts in Computer Science (or Engineering) Degrees are commonly introduced using a hands-on approach mainly consisting of teaching a particular distributed and component-based technology platform (such as Java Enterprise Edition or Microsoft .NET) and proposing the students to develop a small distributed software application with it. Though this approach provides the students with some relevant practical knowledge, we believe that it is not the most appropriate way of teaching all the concepts and particularities of DSE. Thus, in this paper we report on our experience of redesigning an initial DSE course following a model-based approach. By raising the level of abstraction we gained modularity, separation of concerns and technology independence, while making the course evolve according to the latest trends in software development methods.

The open distributed processing (ODP) computational viewpoint describes the functionality of a system and its environment in terms of a configuration of objects interacting at interfaces, independently of their distribution. Quality of service (QoS) contracts and service level agreements are an integral part of any computational specification, which are specified in ODP in terms of environment contracts. Up until unified modeling language (UML) version 2, both the lack of precision in the UML definition and the semantic gap between the ODP concepts and the UML constructs hindered its application for ODP computational viewpoint modeling. With the advent of UML 2 the situation has changed, because its semantics have been more precisely defined and it now incorporates a whole new set of concepts more apt for modeling the structure and behavior of distributed systems. In this paper, we explore the benefits provided by the new extension mechanisms of UML for modeling the ODP computational specifications and, in particular, we show how ODP environment contracts can be modeled with this approach.

The ODP computational viewpoint describes the functionality of a system and its environment in terms of a configuration of objects interacting at interfaces, independently of their distribution. Up until UML version 2.0, both the lack of precision in the UML definition and the semantic gap between the ODP concepts and the UML constructs hindered its application for ODP computational viewpoint modeling. With the advent of UML 2.0 the situation may have changed, because its semantics have been more precisely defined and it now incorporates a whole new set of concepts more apt for modeling the structure and behavior of distributed systems. In this paper, we explore the benefits provided by the new extension mechanisms of UML and, more specifically, we present a UML profile for modeling the ODP computational viewpoint concepts. We also show a case study that illustrates how our proposal is applied to a multimedia distributed system.