Context: Simulation-based testing is a cost-efficient alternative to field testing for Autonomous Vehicles (AVs), but generating safety-critical test cases is challenging due to the vast search space. Prior work has studied static (road features) and dynamic (AV behavior) features of test scenarios separately, but their inter-dependencies are underexplored. Objective: In this paper, we describe an empirical to analyze how static and dynamic features of test scenarios, and their inter-dependencies, influence AV test scenario outcomes. Method: This study proposes an integrated approach using Instance Space Analysis (ISA) to evaluate both types of features, identify key influences on AV safety, and predict test outcomes without execution. Results: Our study identifies critical features affecting test outcomes (effective/ineffective, depending on whether it leads to a safety-critical condition). Results show that combining static and dynamic features improves prediction accuracy, confirmed by models trained on both feature types outperforming models trained with only one type of feature. Conclusion: The interplay of static and dynamic features enhances fault detection in AV testing. This research underscores the importance of integrating both types of features to create more effective testing frameworks for autonomous systems. Key contributions include: (1) a unified framework for AV safety assessment, (2) identification of influential features using ISA, and (3) efficient test outcome prediction for optimized regression testing.

Machine learning (ML) has shown promising results in detecting software vulnerabilities. However, ML detectors are not guaranteed to make predictions based on the right indicators. Studies have revealed that they can rely onirrelevantcode features, such as identifiers or function signatures, particularly those that commonly appear in vulnerable code, yet are not related to the actual vulnerabilities. As a result, the lines of code that the detectors depend on and flag as suspicious are not always genuinely vulnerable. Consequently, developers must manually review these suspicious lines, which is time-consuming and error-prone. If the suspicious lines are wrong, developers may be misled, spend unnecessary effort, or even reach incorrect patching strategies. This highlights the need for automated approaches to identify untrustworthy vulnerability predictions.In this paper, we introduce UNTRUSTVUL, a new approach for identifying untrustworthy vulnerability predictions. Specifically, we focus on cases where a model highlights suspicious lines that would not appear in reliable predictions, i.e., lines that are inherently non-vulnerable and unrelated to any vulnerabilities. To achieve this, we leverage patterns of vulnerable lines observed in historical data. UNTRUSTVUL automatically rules out as untrustworthy any predictions that highlight suspicious lines neither observed in history nor influential to those that have been observed. We refer to such lines as vulnerability-irrelevant. A line is deemed vulnerability-irrelevant if ① it does not match any known patterns of historical vulnerabilities, and ② all its successors in the data and control dependency graph are also vulnerability-irrelevant. Intuitively, a vulnerability-irrelevant line shows low similarity to known vulnerabilities and has no dependency paths to any lines outside the vulnerability-irrelevant category. Notably, these rules are designed to be conservative, as mislabeling a trustworthy prediction as untrustworthy is also undesired. We evaluate UNTRUSTVULon 115K vulnerability predictions made by four models across BigVul, MegaVul, SARD, and PrimeVul datasets, with ground-truth trustworthiness labeled based on the overlap between actual denoised vulnerable lines and model-annotated suspicious lines. UNTRUSTVULeffectively detects untrustworthy predictions with AUC of 70%–88% and F1–score of 82%–94%, outperforming existing approaches by 6%–59% in AUC and 13%–92% in F1–score.

Acceptance criteria (ACs) play a critical role in software development by clearly defining the conditions under which a software feature satisfies stakeholder expectations. However, manually creating accurate, comprehensive, and unambiguous acceptance criteria is challenging, particularly in user interface-intensive applications, due to the reliance on domain-specific knowledge and visual context that is not always captured by textual requirements alone. To address these challenges, we propose RAGcceptance_M2RE, a novel approach that leverages Retrieval-Augmented Generation (RAG) to generate acceptance criteria from multi-modal requirements data, including both textual documentation and visual UI information. We systematically evaluated our approach in an industrial case study involving an education-focused software system used by approximately 100,000 users. The results indicate that integrating multi-modal information significantly enhances the relevance, correctness, and comprehensibility of the generated ACs. Moreover, practitioner evaluations confirm that our approach effectively reduces manual effort, captures nuanced stakeholder intent, and provides valuable criteria that domain experts may overlook, demonstrating practical utility and significant potential for industry adoption. This research underscores the potential of multi-modal RAG techniques in streamlining software validation processes and improving development efficiency. We also make our implementation and a dataset available.

Requirements engineering (RE) in the space industry is inherently complex, demanding high precision, alignment with rigorous standards, and adaptability to mission-specific constraints. Smaller space organisations and new entrants often struggle to derive actionable requirements from extensive, unstructured documents such as mission briefs, interface specifications, and regulatory standards. In this innovation opportunity paper, we explore the potential of Retrieval-Augmented Generation (RAG) models to support and (semi-)automate requirements generation in the space domain. We present a modular, AI-driven approach that preprocesses raw space mission documents, classifies them into semantically meaningful categories, retrieves contextually relevant content from domain standards, and synthesises draft requirements using large language models (LLMs). We apply the approach to a real-world mission document from the space domain to demonstrate feasibility and assess early outcomes in collaboration with our industry partner, Starbound Space Solutions. Our preliminary results indicate that the approach can reduce manual effort, improve coverage of relevant requirements, and support lightweight compliance alignment. We outline a roadmap toward broader integration of AI in RE workflows, intending to lower barriers for smaller organisations to participate in large-scale, safety-critical missions.

Artificial intelligence and recent advances in deep learning architectures, including transformer networks and large language models, change the way people think and act to solve problems. Software engineering, as an increasingly complex process to design, develop, test, deploy, and maintain large-scale software systems for solving real-world challenges, is profoundly affected by many revolutionary artificial intelligence tools in general and machine learning in particular. In this roadmap for artificial intelligence in software engineering, we highlight the recent deep impact of artificial intelligence on software engineering by discussing successful stories of applications of artificial intelligence to classic and new software development challenges. We identify the new challenges that the software engineering community has to address in the coming years to successfully apply artificial intelligence in software engineering, and we share our research roadmap toward the effective use of artificial intelligence in the software engineering profession, while still protecting fundamental human values. We spotlight three main areas that challenge the research in software engineering: the use of generative artificial intelligence and large language models for engineering large software systems, the need of large and unbiased datasets and benchmarks for training and evaluating deep learning and large language models for software engineering, and the need of a new code of digital ethics to apply artificial intelligence in software engineering.

Before being deployed on roads, Autonomous Vehicles (AVs) must undergo comprehensive testing. Safety-critical situations, however, are infrequent in usual driving conditions, so simulated scenarios are used to create them. A test scenario comprises static and dynamic features related to the AV and the test environment; the representation of these features is complex and makes testing a heavy process. A test scenario is effective if it identifies incorrect behaviors of the AV. In this article, we present a technique for identifying key features of test scenarios associated with their effectiveness using Instance Space Analysis (ISA). ISA generates a ( \(2D\) ) representation of test scenarios and their features. This visualization helps to identify combinations of features that make a test scenario effective. We present a graphical representation of each feature that helps identify how well each testing technique explores the search space. While identifying key features is a primary goal, this study specifically seeks to determine the critical features that differentiate the performance of algorithms. Finally, we present metrics to assess the robustness of testing algorithms and the scenarios generated. Collecting essential features in combination with their values associated with effectiveness can be used for selection and prioritization of effective test cases.

Autonomous Vehicles (AVs) are prone to revolutionise the transportation industry. However, they must be thoroughly tested to avoid safety violations. Simulation testing plays a crucial role in finding safety violations of Automated Driving Systems (ADSs). This paper proposes PAFOT, a position-based approach testing framework, which generates adversarial driving scenarios to expose safety violations of ADSs. We introduce a 9-position grid which is virtually drawn around the Ego Vehicle (EV) and modify the driving behaviours of Non-Playable Characters (NPCs) to move within this grid. PAFOT utilises a single-objective genetic algorithm to search for adversarial test scenarios. We demonstrate PAFOT on a well-known high-fidelity simulator, CARLA. The experimental results show that PAFOT can effectively generate safety-critical scenarios to crash ADSs and is able to find collisions in a short simulation time. Furthermore, it outperforms other search-based testing techniques by finding more safety-critical scenarios under the same driving conditions within less effective simulation time.

AI-powered systems have gained widespread popularity in various domains, including Autonomous Vehicles (AVs). However, ensuring their reliability and safety is challenging due to their complex nature. Conventional test adequacy metrics, designed to evaluate the effectiveness of traditional software testing, are often insufficient or impractical for these systems. White-box metrics, which are specifically designed for these systems, leverage neuron coverage information. These coverage metrics necessitate access to the underlying AI model and training data, which may not always be available. Furthermore, the existing adequacy metrics exhibit weak correlations with the ability to detect faults in the generated test suite, creating a gap that we aim to bridge in this study.

UI automation tests play a crucial role in ensuring the quality of mobile applications. Despite the growing popularity of machine learning techniques to generate these tests, they still face several challenges, such as the mismatch of UI elements. The recent advances in Large Language Models (LLMs) have addressed these issues by leveraging their semantic understanding capabilities. However, a significant gap remains in applying these models to industrial-level app testing, particularly in terms of cost optimization and knowledge limitation. To address this, we introduce CAT to create cost-effective UI automation tests for industry apps by combining machine learning and LLMs with best practices. Given the task description, CAT employs Retrieval Augmented Generation (RAG) to source examples of industrial app usage as the few-shot learning context, assisting LLMs in generating the specific sequence of actions. CAT then employs machine learning techniques, with LLMs serving as a complementary optimizer, to map the target element on the UI screen. Our evaluations on the WeChat testing dataset demonstrate the CAT's performance and cost-effectiveness, achieving 90% UI automation with $0.34 cost, outperforming the state-of-the-art. We have also integrated our approach into the real-world WeChat testing platform, demonstrating its usefulness in detecting 141 bugs and enhancing the developers' testing process.

Abstract Previous studies have shown that Automated Program Repair ( apr ) techniques suffer from the overfitting problem. Overfitting happens when a patch is run and the test suite does not reveal any error, but the patch actually does not fix the underlying bug or it introduces a new defect that is not covered by the test suite. Therefore, the patches generated by apr tools need to be validated by human programmers, which can be very costly, and prevents apr tool adoption in practice. Our work aims to minimize the number of plausible patches that programmers have to review, thereby reducing the time required to find a correct patch. We introduce a novel light-weight test-based patch clustering approach called xTestCluster , which clusters patches based on their dynamic behavior. xTestCluster is applied after the patch generation phase in order to analyze the generated patches from one or more repair tools and to provide more information about those patches for facilitating patch assessment. The novelty of xTestCluster lies in using information from execution of newly generated test cases to cluster patches generated by multiple APR approaches. A cluster is formed of patches that fail on the same generated test cases. The output from xTestCluster gives developers a) a way of reducing the number of patches to analyze, as they can focus on analyzing a sample of patches from each cluster, b) additional information (new test cases and their results) attached to each patch. After analyzing 902 plausible patches from 21 Java apr tools, our results show that xTestCluster is able to reduce the number of patches to review and analyze with a median of 50%. xTestCluster can save a significant amount of time for developers that have to review the multitude of patches generated by apr tools, and provides them with new test cases that expose the differences in behavior between generated patches. Moreover, xTestCluster can complement other patch assessment techniques that help detect patch misclassifications.

Context: Test case generation is a critical yet challenging task in software development. Recently, AthenaTest – a Deep Learning (DL) approach for generating unit test cases has been proposed. However, our revisiting study reveals that AthenaTest can generate less than one-fifth of the test cases correctly, due to a lack of assertion knowledge and test signature verification. Objective: This paper introduces A3Test, a novel DL-based approach to the generation of test cases, enhanced with assertion knowledge and a mechanism to verify consistency of the name and signatures of the tests. A3Test aims to adapt domain knowledge from assertion generation to test case generation. Method: A3Test employs domain adaptation principles and introduces a verification approach to name consistency and test signatures. We evaluate its effectiveness using 5,278 focal methods from the Defects4j dataset. Results: Our findings indicate that A3Test outperforms AthenaTest and ChatUniTest. A3Test generates 2.16% to 395.43% more correct test cases, achieves 2.17% to 34.29% higher method coverage, and 25.64% higher line coverage. A3Test achieves 2.13% to 12.20% higher branch coverage, 2.22% to 12.20% higher mutation scores, and 2.44% to 55.56% more correct assertions compared to both ChatUniTest and AthenaTest respectively for one iteration. When generating multiple test cases per method A3Test still shows improvements and comparable efficacy to ChatUnitTest. A survey of developers reveals that the majority of the participants 70.51% agree that test cases generated by A3Test are more readable than those generated by EvoSuite. Conclusions: A3Test significantly enhances test case generation through its incorporation of assertion knowledge and test signature verification, contributing to the generation of correct test cases.

Code review is effective and widely used, yet still time-consuming. Especially, in large-scale software systems, developers may forget to change other related functions that must be changed together (aka. co-changes). This may increase the number of review iterations and reviewing time, thus delaying the code review process. Based on our analysis of 66 projects from five open-source systems, we find that there are 16%–33% of code reviews where at least one function must be co-changed, but was not initially changed. This study aims to propose an approach to recommend co-changed functions in the context of modern code review, which could reduce reviewing time and iterations and help developers identify functions that need to be changed together. We propose CoChangeFinder, a novel method that employs a Graph Neural Network (GNN) to recommend co-changed functions for newly submitted code changes. Then, we conduct a quantitative and qualitative evaluation of CoChangeFinder with 66 studied large-scale open-source software projects. Our evaluation results show that our CoChangeFinder outperforms the state-of-the-art approach, achieving 3.44% to 40.45% for top-k accuracy, 2.00% to 26.07% for Recall@k, and 0.04 to 0.21 for mean average precision better than the baseline approach. In addition, our CoChangeFinder demonstrates the capacity to pinpoint the functions related to logic changes. Our CoChangeFinder outperforms the baseline approach (i.e., TARMAQ) in recommending co-changed functions during the code review process. Based on our findings, CoChangeFinder could help developers save their time and effort, reduce review iterations, and enhance the efficiency of the code review process.

Context: Most research in software clustering and remodularisation typically concludes by recommending the refactoring operations without further insight into the practicality of the proposed technique. Developers might be hesitant to follow through with the refactoring suggestions due to the uncertainty in the effort needed. Objective: This work aims to address this gap by introducing an effoRt Estimation AppRoach foR softwAre clusteriNG-based rEmodularisation (REARRANGE) to close the loop in extant software clustering and remodularisation research by estimating the time required to carry out the suggested refactoring operations based on the history of the evolution of the software. By providing tangible estimates of refactoring effort in person-hours, we can inform developers of complex and time-consuming refactoring operations that will help prioritise refactoring efforts, allowing practitioners to weave in these activities during sprint planning. Method: REARRANGE builds a machine learning model to predict effort estimation based on past commit activity which extracts Software Features (lines of code, number of methods), Refactoring Features (refactoring type, source and destination) and Dependency Features (dependencies between classes). REARRANGE is then compared against sanity checks, baseline effort estimation models, and state-of-the-art software estimation models. We also attempt to cross-validate REARRANGE's effort estimation with software developers. Results: Experimented through 25 open-source Java-based projects, the proposed approach estimated the refactoring effort of the test subjects with a Mean Absolute Error (MAE) of 5.47 person-hours against the MAE of the next-best approach of 453.31 person-hours. Based on a survey conducted among software developers, REARRANGE consistently delivers accurate estimates in 93.6% of cases. Conclusion: The lack of a direct comparison for REARRANGE highlights the need for a refactoring effort-focused estimation model that provides tangible effort estimates in person-hours for refactoring operations. Only then can developers selectively choose relevant refactoring operations while considering the available time and budget constraints, bridging the gap between software clustering research and real-world application.

Microservice architectures and design patterns enhance the development of large-scale applications by promoting flexibility. Industrial practitioners perceive the importance of applying architectural patterns but they struggle to quantify their impact on system quality requirements. Our research aims to quantify the effect of design patterns on system performance metrics, e.g., service latency and resource utilization, even more so when the patterns operate in real-world environments subject to heterogeneous workloads. We built a cloud infrastructure to host a well-established benchmark system that represents our test bed, complemented by the implementation of three design patterns: Gateway Aggregation, Gateway Offloading, Pipe and Filters. Real performance measurements are collected and compared with model-based predictions that we derived as part of our previous research, thus further consolidating the actual impact of these patterns. Our results demonstrate that, despite the difficulty to parameterize our benchmark system, model-based predictions are in line with real experimentation, since the performance behaviors of patterns, e.g., bottleneck switches, are mostly preserved. In summary, this is the first work that experimentally demonstrates the performance behavior of microservices-based architectural patterns. Results highlight the complexity of evaluating the performance of design patterns and emphasize the need for complementing theoretical models with empirical data.

Ensuring the safety of autonomous vehicles (AVs) is of utmost importance, and testing them in simulated environments is a safer option than conducting in-field operational tests. However, generating an exhaustive test suite to identify critical test scenarios is computationally expensive, as the representation of each test is complex and contains various dynamic and static features, such as the AV under test, road participants (vehicles, pedestrians, and static obstacles), environmental factors (weather and light), and the road’s structural features (lanes, turns, road speed, etc.). In this article, we present a systematic technique that uses Instance Space Analysis (ISA) to identify the significant features of test scenarios that affect their ability to reveal the unsafe behaviour of AVs. ISA identifies the features that best differentiate safety-critical scenarios from normal driving and visualises the impact of these features on test scenario outcomes (safe/unsafe) in two dimensions. This visualisation helps to identify untested regions of the instance space and provides an indicator of the quality of the test suite in terms of the percentage of feature space covered by testing. To test the predictive ability of the identified features, we train five Machine Learning classifiers to classify test scenarios as safe or unsafe. The high precision, recall, and F1 scores indicate that our proposed approach is effective in predicting the outcome of a test scenario without executing it and can be used for test generation, selection, and prioritisation.

Defect predictors, static bug detectors, and humans inspecting the code can propose locations in the program that are more likely to be buggy before they are discovered through testing. Automated test generators such as search-based software testing (SBST) techniques can use this information to direct their search for test cases to likely buggy code, thus speeding up the process of detecting existing bugs in those locations. Often the predictions given by these tools or humans are imprecise, which can misguide the SBST technique and may deteriorate its performance. In this article, we study the impact of imprecision in defect prediction on the bug detection effectiveness of SBST. Our study finds that the recall of the defect predictor, i.e., the proportion of correctly identified buggy code, has a significant impact on bug detection effectiveness of SBST with a large effect size. More precisely, the SBST technique detects 7.5 fewer bugs on average (out of 420 bugs) for every 5% decrements of the recall. However, the effect of precision, a measure for false alarms, is not of meaningful practical significance, as indicated by a very small effect size. In the context of combining defect prediction and SBST, our recommendation is to increase the recall of defect predictors as a primary objective and precision as a secondary objective. In our experiments, we find that 75% precision is as good as 100% precision. To account for the imprecision of defect predictors, in particular low recall values, SBST techniques should be designed to search for test cases that also cover the predicted non-buggy parts of the program, while prioritising the parts that have been predicted as buggy.

Protein formulation is a critical aspect of the pharmaceutical industry which aims to improve the efficacy and the safety of the active drug ingredients during the storage, transportation and administration of the drug. Buffer screening is the first stage of this formulation process that selects the promising combinations of buffer and excipients that can help maintain both the stability and efficacy of the drug. In this paper, we propose an interactive Bayesian Optimisation approach that streamlines the buffer screening process and reduces the number of experiments needed to identify an optimal combination of buffer and excipients. Our approach employs two novel formulations of the (multi-buffer) optimisation problem: (i) one that unifies all buffers into a single Bayesian Optimisation framework, and (ii) the other that performs meta-learning to aggregate important excipient information over multiple buffers, in order to predict the most promising buffer and excipients combination to sample next. Our experimental results show that the proposed approach can identify an optimal combination of buffer and excipients while minimising the number of experiments required, and demonstrate the potential of using Bayesian Optimisation to enhance the protein formulation process.

The adoption of design patterns in the microservice architecture and cloud-native development scope was recently reviewed to investigate the industry practice. Interestingly, when considering performance-related aspects, practitioners focus on specific metrics (e.g., the time taken to handle requests) to identify sources of performance hindrance. This paper investigates a subset of seven design patterns that industrial practitioners indicate as relevant for system performance. We are interested to quantify the impact of these patterns while considering heterogeneous workloads, thus supporting software architects in understanding the root causes of performance issues. We use queuing networks to build the performance models of the seven design patterns and extract quantitative insights from model-based performance analysis. Our performance models are flexible in their input parameterization and reusable in different application contexts. We find that most design patterns confirm the expectation of practitioners, and our experimental results assess the identified performance gains and pains. One design pattern (i.e., Gateway Offloading) shows the peculiar characteristic of contributing to performance pains in some cases, leading to novel insights about the impact of design patterns in microservice systems.

Maintenance of existing software requires a large amount of time for comprehending the source code. The architecture of a software, however, may not be clear to maintainers if up-to-date documentations are not available. Software clustering is often used as a remodularisation and architecture recovery technique to help recover a semantic representation of the software design. However, due to the diverse domain and structure of software systems, the suitability of different clustering techniques for different software systems are not investigated thoroughly. Research that introduce new clustering techniques usually validate their approaches on a specific domain, which might limit its generalisability. If the chosen test subjects only represent a narrow perspective of the whole picture, researchers risk not being able to address the external validity of their findings. This work aims to fill this gap by introducing a new approach, Explaining Software Clustering for Remodularisation (E-SC4R), to evaluate the effectiveness of different software clustering approaches. This work focuses on hierarchical clustering and Bunch clustering algorithms and provides information about their suitability according to the features of the software, which, as a consequence, enables the selection of the optimum technique for a particular software system. The E-SC4R framework is able to characterise both the strengths and weaknesses of the analysed software clustering algorithms using software features extracted from the code. The proposed approach also provides a better understanding of the algorithms’ behaviour by showing a 2D representation of the effectiveness of clustering techniques on the feature space through the application of dimensionality reduction techniques.

Search-based software testing (SBST) is now a mature area, with numerous techniques developed to tackle the challenging task of software testing. SBST techniques have shown promising results and have been successfully applied in the industry to automatically generate test cases for large and complex software systems. Their effectiveness, however, has been shown to be problem dependent. In this paper, we revisit the problem of objective performance evaluation of SBST techniques in light of recent methodological advances – in the form of Instance Space Analysis (ISA) – enabling the strengths and weaknesses of SBST techniques to be visualised and assessed across the broadest possible space of problem instances (software classes) from common benchmark datasets. We identify features of SBST problems that explain why a particular instance is hard for an SBST technique, reveal areas of hard and easy problems in the instance space of existing benchmark datasets, and identify the strengths and weaknesses of state-of-the-art SBST techniques. In addition, we examine the diversity and quality of common benchmark datasets used in experimental evaluations.

Automated test generators, such as search-based software testing (SBST) techniques are primarily guided by coverage information. As a result, they are very effective at achieving high code coverage. However, is high code coverage alone sufficient to detect bugs effectively? In this paper, we propose a new SBST technique, predictive many objective sorting algorithm (PreMOSA), which augments coverage information with defect prediction information to decide where to increase the test coverage in the class under test (CUT). Through an experimental evaluation using 420 labelled bugs on the Defects4J benchmark and using theoretical defect predictors, we demonstrate the improved effectiveness and efficiency of PreMOSA in detecting bugs when using any acceptable defect predictor, i.e., a defect predictor with recall and precision$\geq$75%, compared to the state-of-the-art dynamic many objective sorting algorithm (DynaMOSA). PreMOSA detects up to 8.3% more labelled bugs on average than DynaMOSA when given a time budget of 2 minutes for test generation per CUT.

How can we automatically repair semantic bugs in string-processing programs? A semantic bug is an unexpected program state: The program does not crash (which can be easily detected). Instead, the program processes the input incorrectly. It produces an output which users identify as unexpected. We envision a fully automated debugging process for semantic bugs where a user reports the unexpected behavior for a given input and the machine negotiates the condition under which the program fails. During the negotiation, the machine learns to predict the user's response and in this process learns an automated oracle for semantic bugs.

Code review is an effective quality assurance practice, but can be labor-intensive since developers have to manually review the code and provide written feedback. Recently, a Deep Learning (DL)-based approach was introduced to automatically recommend code review comments based on changed methods. While the approach showed promising results, it requires expensive computational resource and time which limits its use in practice. To address this limitation, we propose CommentFinder – a retrieval-based approach to recommend code review comments. Through an empirical evaluation of 151,019 changed methods, we evaluate the effectiveness and efficiency of CommentFinder against the state-of-the-art approach. We find that when recommending the best-1 review comment candidate, our CommentFinder is 32% better than prior work in recommending the correct code review comment. In addition, CommentFinder is 49 times faster than the prior work. These findings highlight that our CommentFinder could help reviewers to reduce the manual efforts by recommending code review comments, while requiring less computational time.

Abstract Context Machine learning (ML) software systems are permeating many aspects of our life, such as healthcare, transportation, banking, and recruitment. These systems are trained with data that is often biased, resulting in biased behaviour. To address this issue, fairness testing approaches have been proposed to test ML systems for fairness, which predominantly focus on assessing classification-based ML systems. These methods are not applicable to regression-based systems, for example, they do not quantify the magnitude of the disparity in predicted outcomes, which we identify as important in the context of regression-based ML systems. Method: We conduct this study as design science research. We identify the problem instance in the context of emergency department (ED) wait-time prediction. In this paper, we develop an effective and efficient fairness testing approach to evaluate the fairness of regression-based ML systems. We propose fairness degree, which is a new fairness measure for regression-based ML systems, and a novel search-based fairness testing (SBFT) approach for testing regression-based machine learning systems. We apply the proposed solutions to ED wait-time prediction software. Results: We experimentally evaluate the effectiveness and efficiency of the proposed approach with ML systems trained on real observational data from the healthcare domain. We demonstrate that SBFT significantly outperforms existing fairness testing approaches, with up to 111% and 190% increase in effectiveness and efficiency of SBFT compared to the best performing existing approaches. Conclusion: These findings indicate that our novel fairness measure and the new approach for fairness testing of regression-based ML systems can identify the degree of fairness in predictions, which can help software teams to make data-informed decisions about whether such software systems are ready to deploy. The scientific knowledge gained from our work can be phrased as a technological rule; to measure the fairness of the regression-based ML systems in the context of emergency department wait-time prediction use fairness degree and search-based techniques to approximate it.

Accessibility design elements allow people to access software products and services independent of their different abilities. However, accessibility is challenging to handle and whether accessibility is widely considered in software projects is unclear. In this work, we aim to understand if accessibility is a prevalent consideration in practice, what accessibility issues are discussed in GitHub projects, what potential reasons cause accessibility issues, and what solutions (e.g., tools and standards) are applied for addressing accessibility issues. In this work, we collect 11,820 accessibility issues and their threads discussed by developers in popular GitHub projects. We manually analyzed and grouped the collected accessibility issues into seven categories. The results of our study uncover that accessibility is widely discussed in general projects, and the potential reasons that cause accessibility issues are because developers are not aware of the importance of accessibility and they lack knowledge about accessibility concerns, standards, and existing tools. Our results and findings can enhance and improve developers' knowledge and awareness when they conduct accessibility-relevant design or incorporate accessibility elements into their projects.

Today, most automated test generators, such as search-based software testing (SBST) techniques focus on achieving high code coverage. However, high code coverage is not sufficient to maximise the number of bugs found, especially when given a limited testing budget. In this paper, we propose an automated test generation technique that is also guided by the estimated degree of defectiveness of the source code. Parts of the code that are likely to be more defective receive more testing budget than the less defective parts. To measure the degree of defectiveness, we leverage Schwa, a notable defect prediction technique.

Swarm robots continue to become more prominent in solving challenging tasks in real world applications. Due to the complexity of operating in often unknown environments, centralised control of swarm robots is not ideal. Prior manual programming is also not practical under these kind of circumstances. Thus, we establish a hyper-heuristic based learning approach for swarm robot control. With this framework, robots can autonomously identify appropriate heuristics from a set of given low-level heuristics, each heuristic guiding certain behaviours. We evaluated this type of online learning on building surface cleaning and studied the effectiveness of our hyper-heuristic online learning. Nine heuristics were proposed in this study. Through the experiments it can be seen that robots can improve their cleaning performance through the online learning process. More importantly, the experiments show that appropriate heuristics can be selected even when the size of the heuristic set is changed. The study on four types of environments shows that with the same heuristic set, the robot swarm can adapt to different environments for different tasks. Hence, hyper-heuristic learning is an effective method for decentralised control of swarm robots.

Testing is technically and economically crucial for ensuring software quality. One of the most challenging testing tasks is to create test suites that will reveal potential defects in software. However, as the size and complexity of software systems increase, the task becomes more labour-intensive and manual test data generation becomes infeasible. To address this issue, researchers have proposed different approaches to automate the process of generating test data using search techniques; an area that is known as Search-Based Software Testing (SBST).

After the Big Bang, it took about 200 million years before the very first stars would form - now more than 13 billion years ago. Unfortunately, we will not be able to observe these stars directly. Instead, we can observe the 'fossil' records that these stars have left behind, preserved in the oldest stars of our own galaxy. When the first stars exploded as supernovae, their ashes were dispersed and the next generation of stars formed, incorporating some of the debris. We can now measure the chemical abundances in those old stars, which is similar to a genetic fingerprint that allows us to identify the parents.

Automated test suite generation (ATSG) is an important topic in software engineering, with a wide range of techniques and tools being used in academia and industry. While their usefulness is widely recognized, due to the labor-intensive nature of the task, the effectiveness of the different techniques in automatically generating test cases for different software systems is not thoroughly understood. Despite many studies introducing various ATSG techniques, much remains to be learned, however, about what makes a particular technique work well (or not) for a specific software system. In this paper, we seek an answer to the question: “What features of a software system impact the effectiveness of ATSG techniques?” Once these features are identified, can they be used to select the most effective ATSG technique for a particular software system? To this end, we have implemented the mapping the effectiveness of test automation (META) tool, a new framework that identifies important software features that can be used to select suitable ATSG techniques to apply to new software systems. We evaluate the framework on a large set of open-source software projects and three ATSG techniques. The evaluation indicates that the number of methods in a class, the coupling between object classes, and the response for a class are the most indicative of what makes a software system hard to test by different techniques. The decision tree for ATSG technique selection generated by the META framework has an 88% accuracy, as shown by n-fold cross validation.

Summary Private Cloud provides Cloud services with its relatively limited resources compared to public Clouds. Resources in private Clouds should be used energy efficiently. The resource utilization is determined by the assignment of virtual machines (VMs) to hosts. Because of the frequent changes in the resource requests on VMs, the system might become imbalanced, some hosts are overloaded/underloaded. Virtual machine migration is a solution to ease the imbalance problem. Virtual machine migration includes the selection of a VM for migration and decision upon where it should be taken to (mapped). In this work, VM selection and VM mapping are integrated and aimed to ease the imbalance problem for energy efficiency. Moreover, selection and mapping have become adaptive to ease imbalance, while optimizing energy consumption and adaptively responding to changes in the system. Our proposed adaptive mechanism applies Bayesian inference to estimate the likelihood of a VM migration decision, both VM selection for migration and VM mapping, optimizing energy consumption. The proposed mechanism is evaluated on CloudSim using PlanetLab workload on a heterogeneous Cloud. It is demonstrated to reduce energy consumption significantly, (on average) by 116%, while its total execution time is also, (on average) 5.39 times, shorter than the competing state‐of‐the‐art policies.

Complex combinatorial problems are most often optimised with heuristic solvers, which usually deliver acceptable results without any indication of the quality obtained. Recently, predictive diagnostic optimisation was proposed as a means of characterising the fitness landscape while optimising a combinatorial problem. The scalars produced by predictive diagnostic optimisation appear to describe the difficulty of the problem with relative reliability. In this study, we record more scalars that may be helpful in determining problem difficulty during the optimisation process and analyse these in combination with other well-known landscape descriptors by using exploratory factor analysis on four landscapes that arise from different search operators, applied to a varied set of quadratic assignment problem instances. Factors are designed to capture properties by combining the collinear variances of several variables. The extracted factors can be interpreted as the features of landscapes detected by the variables, but disappoint in their weak correlations with the result quality achieved by the optimiser, which we regard as the most reliable indicator of difficulty available. It appears that only the prediction error of predictive diagnostic optimisation has a strong correlation with the quality of the results produced, followed by a medium correlation of the fitness distance correlation of the local optima.

The automation of software architecture design is an important goal in software engineering. A plethora of automated design exploration techniques have been devised in the last decades to handle the complexity of making design decision in large scale, complex software systems. The common aim of these methods is the optimisation of quality attributes, such as reliability and safety. The majority of approaches use heuristic methods, such as local search or genetic algorithms, which use gradients in the fitness space to guide the search to the local optimum. When problems are constrained, search gradients are disrupted by infeasible regions, which may have a great impact on the difficulty of solving optimisation problems. Discovering the conditions under which a search heuristic will succeed or fail is critical for understanding the strengths and weaknesses of different software architecture optimisation methods. This paper investigates how to adequately characterize the features of constrained problem instances that have impact on difficulty in terms of algorithmic performance, and how such features can be defined and measured for the component deployment optimisation problem. We employ fitness landscape characterisation metrics that measure uniformity of the gradients in the search space, and investigate how two different constraints shape the search space, and as a result affect the performance of software architecture optimisation approaches.

All commonly used stochastic optimisation algorithms have to be parameterised to perform effectively. Adaptive parameter control (APC) is an effective method used for this purpose. APC repeatedly adjusts parameter values during the optimisation process for optimal algorithm performance. The assignment of parameter values for a given iteration is based on previously measured performance. In recent research, time series prediction has been proposed as a method of projecting the probabilities to use for parameter value selection. In this work, we examine the suitability of a variety of prediction methods for the projection of future parameter performance based on previous data. All considered prediction methods have assumptions the time series data has to conform to for the prediction method to provide accurate projections. Looking specifically at parameters of evolutionary algorithms (EAs), we find that all standard EA parameters with the exception of population size conform largely to the assumptions made by the considered prediction methods. Evaluating the performance of these prediction methods, we find that linear regression provides the best results by a very small and statistically insignificant margin. Regardless of the prediction method, predictive parameter control outperforms state of the art parameter control methods when the performance data adheres to the assumptions made by the prediction method. When a parameter's performance data does not adhere to the assumptions made by the forecasting method, the use of prediction does not have a notable adverse impact on the algorithm's performance.

The performance of an Evolutionary Algorithm (EA) is greatly affected by the settings of its strategy parameters. An effective solution to the parameterisation problem is adaptive parameter control, which applies learning methods that use feedback from the optimisation process to evaluate the effect of parameter value choices and adjust the parameter values over the iterations. At every iteration of an EA, the performance of an EA is reported and employed by the feedback mechanism as an indication of the success of the parameterisation of the algorithm instance. Many approaches to collect information about the algorithm's performance exist in single objective optimisation. In this work, we review the most recent and prominent approaches. In multiobjective optimisation, establishing a single scalar which can report the algorithm's performance as feedback for adaptive parameter control is a complex task. Existing performance measures of multiobjective optimisation are generally used as feedback for the optimisation process. We discuss the properties of these measures and present an empirical evaluation of the binary hypervolume and ϵ+-indicators as feedback for adaptive parameter control.

Stochastic optimisers such as Evolutionary Algorithms, Estimation of Distribution Algorithm are suitable methods when problems are highly complex and deterministic algorithms cannot be expected to produce acceptable results. Generally, when the search process produces the optimised solutions, there is no indication how successful the search has been. In previous work, we introduced Predictive Diagnostic Optimisation (PDO), a local-search-based solver which can predict with certain accuracy the quality of local optima and that can help decide which of the initial solutions is appropriate to optimise. The neighbourhood created by the swap operator was used in exploration of the search space and the number of predictors created is a metric for the homogeneity of the landscape. The advantage of PDO is that it provides information regarding the difficulty of the search landscape alongside the optimisation results. In this work we extend PDO by employing three more neighbourhood operators to allow a comparison between the performances of different types of local search. Each neighbourhood operator has its own group of predictors and the difficulty in predicting the local optima is quantified by a new metric, the prediction error. To provide an assessment of the characterisation ability for the algorithm, a set of landscapes with various degrees of difficulty has been designed by manipulating the matrices of the test problems instances. We show that the metric is able to identify the degree of difficulty that we expect the landscapes to pose for the employed local search operators.

Evolutionary Algorithms are equipped with a range of adjustable parameters, such as crossover and mutation rates which significantly influence the performance of the algorithm. Practitioners usually do not have the knowledge and time to investigate the ideal parameter values before the optimisation process. Furthermore, different parameter values may be optimal for different problems, and even problem instances. In this work, we present a parameter control method which adjusts parameter values during the optimisation process using the algorithm's performance as feedback. The approach is particularly effective with continuous parameter intervals, which are adapted dynamically. Successful parameter ranges are identified using an entropy-based clusterer, a method which outperforms state-of-the-art parameter control algorithms.

Due to significant industrial demands toward software systems with increasing complexity and challenging quality requirements, software architecture design has become an important development activity and the research domain is rapidly evolving. In the last decades, software architecture optimization methods, which aim to automate the search for an optimal architecture design with respect to a (set of) quality attribute(s), have proliferated. However, the reported results are fragmented over different research communities, multiple system domains, and multiple quality attributes. To integrate the existing research results, we have performed a systematic literature review and analyzed the results of 188 research papers from the different research communities. Based on this survey, a taxonomy has been created which is used to classify the existing research. Furthermore, the systematic analysis of the research literature provided in this review aims to help the research community in consolidating the existing research efforts and deriving a research agenda for future developments.

All existing stochastic optimisers such as Evolutionary Algorithms require parameterisation which has a significant influence on the algorithm's performance. In most cases, practitioners assign static values to variables after an initial tuning phase. This parameter tuning method requires experience the practitioner may not have and, when done conscientiously, is rather time-consuming. Also, the use of parameter values that remain constant over the optimisation process has been observed to achieve suboptimal results. This work presents a parameter control method which redefines variables repeatedly based on a separate optimisation process which receives its feedback from the primary optimisation algorithm. The feedback is used for a projection of the value performing well in the future. The parameter values are sampled from intervals which are adapted dynamically, a method which has proved particularly effective and outperforms all existing adaptive parameter controls significantly.

In stochastic optimisation, all currently employed algorithms have to be parameterised to perform effectively. Users have to rely on approximate guidelines or, alternatively, undertake extensive prior tuning. This study introduces a novel method of parameter control, i.e. the dynamic and automated variation of values for parameters used in approximate algorithms. The method uses an evaluation of the recent performance of previously applied parameter values and predicts how likely each of the parameter values is to produce optimal outcomes in the next cycle of the algorithm. The resulting probability distribution is used to determine the parameter values for the following cycle. The results of our experiments show a consistently superior performance of two very different EA algorithms when they are parameterised using the predictive parameter control method.

Decisions regarding the mapping of software components to hardware nodes affect the quality of the resulting system. Making these decisions is hard when considering the ever-growing complexity of the search space, as well as conflicting objectives and constraints. An automation of the solution space exploration would help not only to make better decisions but also to reduce the time of this process. In this paper, we propose to employ Ant Colony Optmisation (ACO) as a multi-objective optimisation strategy. The constructive approach is compared to an iterative optimisation procedure - a Genetic Algorithm (GA) adaptation - and was observed to perform suprisingly similar, although not quite on a par with the GA, when validated based on a series of experiments.