Test case prioritisation (TCP) involves ordering and selecting the most relevant test cases to verify that the current functionality of a software system remains unaffected by code changes. Recently, TCP has been addressed by machine learning (ML), predicting the failure probability of each test case. However, software engineers may struggle to identify and implement the most suitable predictive models for TCP. As new builds adapt the test suite being tested, the model performance may decline with the incorporation of these new builds. In this study, we address these challenges by applying automated workflow composition, including algorithm selection and hyperparameter optimisation. They are considered tasks within automated machine learning (AutoML). With this aim, our proposal employs grammar-guided genetic programming as the underlying mechanism for implementing the AutoML algorithm. Our experimental results demonstrate that our approach can adapt to the particularities of the system under test, selecting the most appropriate ML pipeline and hyperparameters for each build. More importantly, our approach reduces the ML knowledge required by testers—particularly regarding the selection and tuning of algorithms—while enabling them to generate pipelines suited to successive changes in SUT builds. This research showcases the potential of AutoML in software engineering, specifically for the TCP problem.

To measure and improve the strength of test suites, software projects and their developers commonly use code coverage and aim for a threshold of around 80%. But what is the 80 % of the source code that should be covered? To prepare for the development of new, more refined code coverage criteria, we introduce a novel predictive problem in software testing: whether a code line is, or should be, covered by the test suite. In this short paper, we propose the collection of coverage information, source code metrics, and abstract syntax tree data and explore whether they are relevant to predict whether a code line is exercised by the test suite or not. We present a preliminary experiment using four machine learning (ML) algorithms and an open source Java project. We observe that ML classifiers can achieve high accuracy (up to 90%) on this novel predictive problem. We also apply an explainable method to better understand the characteristics of code lines that make them more “appealing” to be covered. Our work opens a research line worth to investigate further, where the focus of the prediction is the code to be tested. Our innovative approach contrasts with most predictive problems in software testing, which aim to predict the test case failure probability.

Abstract Multi‐objective optimization problems frequently appear in many diverse research areas and application domains. Metaheuristics, as efficient techniques to solve them, need to be easily accessible to users with different expertise and programming skills. In this context, metaheuristic optimization frameworks are helpful, as they provide popular algorithms, customizable components and additional facilities to conduct experiments. Due to the broad range of available tools, this paper presents a systematic evaluation and experimental comparison of 10 frameworks, covering from multi‐purpose, consolidated tools to recent libraries specifically designed for multi‐objective optimization. The evaluation is organized around seven characteristics: search components and techniques, configuration, execution, utilities, external support and community, software implementation and performance. An analysis of code metrics and a series of experiments serves to assess the last two features. Lesson learned and open issues are also discussed as part of the comparative study. The outcomes of the evaluation process reveal a contrasted support to recent advances in multi‐objective optimization, with a lack of novel algorithms and variety of metaheuristics other than evolutionary algorithms. The experimental comparison also reports significant differences in terms of both execution time and memory usage under demanding configurations.

Most software companies have extensive test suites and re-run parts of them continuously to ensure that recent changes have no adverse effects. Since test suites are costly to execute, industry needs methods for test case prioritisation (TCP). Recently, TCP methods use machine learning (ML) to exploit the information known about the system under test and its test cases. However, the value added by ML-based TCP methods should be critically assessed with respect to the cost of collecting the information. This article analyses two decades of TCP research and presents a taxonomy of 91 information attributes that have been used. The attributes are classified with respect to their information sources and the characteristics of their extraction process. Based on this taxonomy, TCP methods validated with industrial data and those applying ML are analysed in terms of information availability, attribute combination and definition of data features suitable for ML. Relying on a high number of information attributes, assuming easy access to system under test code and simplified testing environments are identified as factors that might hamper industrial applicability of ML-based TCP. The TePIA taxonomy provides a reference framework to unify terminology and evaluate alternatives considering the cost-benefit of the information attributes.

Automated test case generation has proven to be useful to reduce the usually high expenses of software testing. However, several studies have also noted the skepticism of testers regarding the comprehension of generated test suites when compared to manually designed ones. This fact suggests that involving testers in the test generation process could be helpful to increase their acceptance of automatically-produced test suites. In this paper, we propose incorporating interactive readability assessments made by a tester into EvoSuite, a widely-known evolutionary test generation tool. Our approach,InterEvo-TR, interacts with the tester at different moments during the search and shows different test cases covering the same coverage target for their subjective evaluation. The design of such an interactive approach involves a schedule of interaction, a method to diversify the selected targets, a plan to save and handle the readability values, and some mechanisms to customize the level of engagement in the revision, among other aspects. To analyze the potential and practicability of our proposal, we conduct a controlled experiment in which 39 participants, including academics, professional developers, and student collaborators, interact withInterEvo-TR. Our results show that the strategy to select and present intermediate results is effective for the purpose of readability assessment. Furthermore, the participants’ actions and responses to a questionnaire allowed us to analyze the aspects influencing test code readability and the benefits and limitations of an interactive approach in the context of test case generation, paving the way for future developments based on interactivity.

Test generation is a costly but necessary testing activity to increase the quality of software projects. Automated testing tools based on evolutionary computation principles constitute an appealing modern approach to support testing tasks. However, these tools still find difficulties to detect certain types of plausible faults in real-world projects. Besides, recent studies have shown that, in general, automatically-generated tests do not resemble those manually written and, consequently, testers are reluctant to adopt them. We observe two key issues, namely the opacity of the process and the lack of cooperation with the tester, currently hampering the acceptance of automated results. Based on these findings, we explore in this paper how the interaction between current tools and expert testers would help address the test case generation problem. More specifically, we identify a number of interaction opportunities related to the object-oriented test case design driven to boost their readability and detection power. Using EvoSuite as base implementation, we present a proof of concept focused on the possibility to integrate readability assessment of the most promising test suites into a genetic algorithm.

Abstract Data preprocessing is known to be essential to produce accurate data from which mining methods are able to extract valuable knowledge. When data constantly arrives from one or more sources, preprocessing techniques need to be adapted to efficiently handle these data streams. To help domain experts to define and execute preprocessing tasks for data streams, this paper proposes the use of active rule‐based systems and, more specifically, complex event processing (CEP) languages and engines. The main contribution of our approach is the formulation of preprocessing procedures as event detection rules, expressed in an SQL‐like language, that provide domain experts a simple way to manipulate temporal data. This idea is materialized into a publicly available solution that integrates a CEP engine with a library for online data mining. To evaluate our approach, we present three practical scenarios in which CEP rules preprocess data streams with the aim of adding temporal information, transforming features and handling missing values. Experiments show how CEP rules provide an effective language to express preprocessing tasks in a modular and high‐level manner, without significant time and memory overheads. The resulting data streams do not only help improving the predictive accuracy of classification algorithms, but also allow reducing the complexity of the decision models and the time needed for learning in some cases.

Search-Based Software Engineering (SBSE) has been successfully applied to automate a wide range of software development activities. Nevertheless, in those software engineering problems where human evaluation and preference are crucial, such insights have proved difficult to characterize in search, and solutions might not look natural when that is the expectation. In an attempt to address this, an increasing number of researchers have reported the incorporation of the 'human-in-the-loop' during search and interactive SBSE has attracted significant attention recently. However, reported results are fragmented over different development phases, and a great variety of novel interactive approaches and algorithmic techniques have emerged. To better integrate these results, we have performed a systematic literature review of interactive SBSE. From a total of 669 papers, 26 primary studies were identified. To enable their analysis, we formulated a classification scheme focused on four crucial aspects of interactive search, i.e., the problem formulation, search technique, interactive approach, and the empirical framework. Our intention is that the classification scheme affords a methodological approach for interactive SBSE. Lastly, as well as providing a detailed cross analysis, we identify and discuss some open issues and potential future trends for the research community.

Software architects devote substantial efforts to find the most fitting architectural description for their system, which should not only specify its structure, but is also required to meet multiple, simultaneous quality criteria. Evolutionary computation has recently demonstrated to provide insightful support during the design phase by automatically deciding how to organise internal software components and how they should interact each other. Observed from a multi-objective perspective, particular care has to be taken in order to reach an appropriate trade-off among design metrics, while providing the software engineer with diverse alternatives to choose among. However, multi-objective evolutionary algorithms may find difficulties to control both aspects and, at the same time, to explore the entire search space in depth. Under these circumstances, local search can be applied to complement the evolution by scrutinising the most promising search directions. This paper proposes two different approaches that take advantage of the benefits of local search within the multi-objective evolutionary discovery of component-based software architectures. A detailed analysis and comparative study provides interesting findings like the importance of assigning a sufficient number of evaluations to the local improvement. The way in which local search explores and compares solutions for acceptance is a relevant aspect to promote diversity during the discovery process as well.

During the design of complex systems, software architects have to deal with a tangle of abstract artefacts, measures and ideas to discover the most fitting underlying architecture. A common way to structure these systems is in terms of their interacting software components, whose composition and connections need to be properly adjusted. Its abstract and highly combinatorial nature increases the complexity of the problem. In this scenario, Search-based Software Engineering (SBSE) may serve to support this decision making process from initial analysis models, since the discovery of component-based architectures can be formulated as a challenging multiple optimisation problem, where different metrics and configurations can be applied depending on the design requirements and its specific domain. Many-objective optimisation evolutionary algorithms can provide an interesting alternative to classical multi-objective approaches. This paper presents a comparative study of five different algorithms, including an empirical analysis of their behaviour in terms of quality and variety of the returned solutions. Results are also discussed considering those aspects of concern to the expert in the decision making process, like the number and type of architectures found. The analysis of many-objectives algorithms constitutes an important challenge, since some of them have never been explored before in SBSE.

This work presents a novel proposal for incremental intruder detection in collaborative recommender systems. We explore the use of rare association rule mining to reveal the existence of a suspected raid of attackers that would alter the normal behaviour of a rating-based system. In this position paper we have extended our previous G3PARM algorithm, which has already proven to serve as a solid method for extracting frequent association rules. G3PARM is an evolutionary algorithm that uses G3P (Grammar Guided Genetic Programming), which provides expressiveness and flexibility enough to adapt and apply the base context-free grammar to each specific problem or domain. We fully outline, moreover, the complete exploration and detection model, which includes some further post-analysis steps. Finally, as a proof of concept, we validate the scalability, efficiency and accuracy of our proposal showing the results obtained when different malicious intruders want to attack an on line recommender system.