Test Case Prioritization (TCP) techniques are a key factor in reducing the regression testing costs even more when Continuous Integration (CI) practices are adopted. TCP approaches based on failure history have been adopted in this context because they are more suitable for CI environment constraints: test budget and test case volatility, that is, test cases may be added or removed over the CI cycles. Promising approaches are based on Reinforcement Learning (RL), which learns with past prioritization, guided by a reward function. In this work, we introduce a TCP approach for CI environments based on the sliding window method, which can be instantiated with different Machine Learning (ML) algorithms. Unlike other ML approaches, it does not require retraining the model to perform the prioritization and any code analysis. As an alternative for the RL approaches, we apply the Random Forest (RF) algorithm and a Long Short Term Memory (LSTM) deep learning network in our evaluation. We use three time budgets and eleven systems. The results show the applicability of the approach considering the prioritization time and the time between the CI cycles. Both algorithms take just a few seconds to execute. The RF algorithm obtained the best performance for more restrictive budgets compared to the RL approaches described in the literature. Considering all systems and budgets, RF reaches Normalized Average Percentage of Faults Detected (NAPFD) values that are the best or statistically equivalent to the best ones in around 72% of the cases, and the LSTM network in 55% of them. Moreover, we discuss some implications of our results for the usage of the algorithms evaluated.

Summary The use of mutation testing for mobile applications (apps for short) is still a challenge. Mobile apps are usually event‐driven and encompass graphical user interfaces (GUIs) and a complex execution environment. Then, they require mutant operators to describe specific apps faults, and the automation of the mutation process phases like execution and analysis of the mutants is not an easy task. To encourage research addressing such challenges, this paper presents results from a mapping study on mutation testing for mobile apps. Following a systematic plan, we found 16 primary studies that were analysed according to three aspects: (i) trends and statistics about the field; (ii) study characteristics such as focus, proposed operators and automated support for the mutation testing phases; and (iii) evaluation aspects. The great majority of studies (98%) have been published in the last 3 years. The most addressed language is Java, and Android is the only operating system considered. Mutant operators of GUI and configuration types are prevalent in a total of 138 operators found. Most studies implement a supporting tool, but few tools support mutant execution and analysis. The evaluation conducted by the studies includes apps mainly from the finance and utility domain. Nevertheless, there is a lack of benchmarks and more rigorous experiments. Future research should address other specific types of faults, languages, and operating systems. They should offer support for mutant execution and analysis, as well as to reduce the mutation testing cost and limitations in the mobile context.

Continuous Integration (CI) environments have been increasingly adopted in the industry to allow frequent integration of software changes, making software evolution faster and cost-effective. In such environments, Test Case Prioritization (TCP) techniques play an important role to reduce regression testing costs, establishing a test case execution order that usually maximizes early fault detection. Existing works on TCP in CI environments (TCPCI) present some limitations. Few pieces of work consider CI particularities, such as the test case volatility, that is, they do not consider the dynamic environment of the software life-cycle in which new test cases can be added or removed (discontinued), characteristic related to the Exploration versus Exploitation (EvE) dilemma. To solve such a dilemma an approach needs to balance: i) the diversity of test suite; and ii) the quantity of new test cases and test cases that are error-prone or that comprise high fault-detection capabilities. To deal with this, most approaches use, besides the failure-history, other measures that rely on code instrumentation or require additional information, such as testing coverage. However, to maintain the information updated can be difficult and time-consuming, not scalable due to the test budget of CI environments. In this context, and to properly deal with the TCPCI problem, this work presents an approach based on Multi-Armed Bandit (MAB) calledCOLEMAN(Combinatorial VOlatiLEMulti-Armed BANdit). The TCPCI problem falls into the category of volatile and combinatorial MAB, because multiple arms (test cases) need to be selected, and they are added or removed over the cycles. We conducted an evaluation considering three time budgets and eleven systems. The results show the applicability of our approach and thatCOLEMANoutperforms the most similar approach from literature in terms of early fault detection and performance.