Summary The use of mutation testing for mobile applications (apps for short) is still a challenge. Mobile apps are usually event‐driven and encompass graphical user interfaces (GUIs) and a complex execution environment. Then, they require mutant operators to describe specific apps faults, and the automation of the mutation process phases like execution and analysis of the mutants is not an easy task. To encourage research addressing such challenges, this paper presents results from a mapping study on mutation testing for mobile apps. Following a systematic plan, we found 16 primary studies that were analysed according to three aspects: (i) trends and statistics about the field; (ii) study characteristics such as focus, proposed operators and automated support for the mutation testing phases; and (iii) evaluation aspects. The great majority of studies (98%) have been published in the last 3 years. The most addressed language is Java, and Android is the only operating system considered. Mutant operators of GUI and configuration types are prevalent in a total of 138 operators found. Most studies implement a supporting tool, but few tools support mutant execution and analysis. The evaluation conducted by the studies includes apps mainly from the finance and utility domain. Nevertheless, there is a lack of benchmarks and more rigorous experiments. Future research should address other specific types of faults, languages, and operating systems. They should offer support for mutant execution and analysis, as well as to reduce the mutation testing cost and limitations in the mobile context.

Software systems have been continuously evolved and delivered with high quality due to the widespread adoption of automated tests. A recurring issue hurting this scenario is the presence of flaky tests, a test case that may pass or fail non-deterministically. A promising, but yet lacking more empirical evidence, approach is to collect static data of automated tests and use them to predict their flakiness. In this paper, we conducted an empirical study to assess the use of code identifiers to predict test flakiness. To do so, we first replicate most parts of the previous study of Pinto et al. (MSR 2020). This replication was extended by using a different ML Python platform (Scikit-learn) and adding different learning algorithms in the analyses. Then, we validated the performance of trained models using datasets with other flaky tests and from different projects. We successfully replicated the results of Pinto et al. (2020), with minor differences using Scikit-learn; different algorithms had performance similar to the ones used previously. Concerning the validation, we noticed that the recall of the trained models was smaller, and classifiers presented a varying range of decreases. This was observed in both intra-project and inter-projects test flakiness prediction.

The Node.js platform empowers a huge number of software systems programmed with JavaScript. Node.js employs an asynchronous execution model where event handlers are scheduled nondeterministically, and unexpected races between event handlers often cause malfunctions. Existing techniques for detecting such event races require complex modifications of the Node.js internals, or target only certain kinds of races. This paper presents a new approach, called NODERACER, that detects event races in Node.js applications by selectively postponing events, guided by happens-before relations. The technique is implemented entirely with code instrumentation, without modifications of the Node.js system. Our experimental results give evidence that NODERACER finds event race errors with higher probability than a state-of-the-art fuzzer, and that the use of happens-before relations helps avoiding false positives. Furthermore, we demonstrate that NODERACER produces actionable error reports, and that it can be helpful for detecting test flakiness that is caused by event races.

Cross‐platform apps stand out by their ability to run in various operating systems (OSs), such as Android, iOS, and Windows. Such apps are developed using popular frameworks for cross‐platform app development such as Apache Cordova, Xamarin, and React Native. However, the mechanisms to automate their tests are not cross‐platform and do not support multiple configurations. Hence, different test scripts have to be coded for each platform, yet there is no guarantee they will work in different configurations varying, e.g. platform, OS version, and hardware available. This study proposes mechanisms to produce automated tests for cross‐platform mobile apps. In order to set up the tests to execute in multiple configurations, the authors’ approach adopts two reference devices: one running Android and other iOS. As both platforms have their own user interface (UI) XML representation, they also investigated six individual expression types and two combined strategies to locate UI elements. They have developed a prototype tool called cross‐platform app test script recorder (x‐PATeSCO) to support the proposed approach, as well as the eight locating strategies considered. They evaluated the approach with nine cross‐platform mobile apps, comparing the locating strategies in six real devices.

Cross-platform app development has achieved meaningful results in practice with frameworks like React Native, Xamarin, and Apache Cordova.Unlike native apps, such frameworks support the development of a mobile app that can run in different platforms.Nevertheless, the literature lacks techniques to test cross-platform apps since most of the existing works focus on native Android apps.A promising strategy for native apps is to amplify test suites so that the specific characteristics of mobile apps can be tested.This paper aims to investigate the test amplification of cross-platform apps.To do so, we apply four test patterns that verify well-known characteristics of mobile computing and amplify existing test suites.The proposed approach has been implemented in a tool capable of generating Appium test scripts and was evaluated with nine cross-platform apps.The amplified test suites exercise new scenarios, uncovering 23 unique bugs in eight out of nine apps.

Summary In service‐oriented architecture testing, the behaviour of a composite service and its partner services can be represented as an event graph (model). As test cases are derived from such a model, the Chinese postman problem (CPP) algorithm is applied to cover all edges. Although CPP provides an optimal solution, there is a lack of alternative algorithms to obtain not only a faster execution but also a satisfactory solution. This paper introduces the event tree algorithm (ETA) to generate test cases from event sequence models for Web service compositions. Two strategies are also proposed to improve ETA: First, a greedy strategy to build the tree is defined based on the cost of expanding given edges. Second, redundant elements are eliminated (pruned) in the tree. Four versions of ETA are compared with CPP with respect to test suite size, generation time, number of test sequences, and test sequence length.

Software testing involves probing into the behavior of software systems to uncover faults. Most testing activities are complex and costly, so a practical strategy that has been adopted to circumvent these issues is to automate software testing. There has been a growing interest in applying machine learning (ML) to automate various software engineering activities, including testing-related ones. In this paper, we set out to review the state-of-the art of how ML has been explored to automate and streamline software testing and provide an overview of the research at the intersection of these two fields by conducting a systematic mapping study. We selected 48 primary studies. These selected studies were then categorized according to study type, testing activity, and ML algorithm employed to automate the testing activity. The results highlight the most widely used ML algorithms and identify several avenues for future research. We found that ML algorithms have been used mainly for test-case generation, refinement, and evaluation. Also, ML has been used to evaluate test oracle construction and to predict the cost of testing-related activities. The results of this paper outline the ML algorithms that are most commonly used to automate software-testing activities, helping researchers to understand the current state of research concerning ML applied to software testing. We also found that there is a need for better empirical studies examining how ML algorithms have been used to automate software-testing activities.

Context: while the mobile computing market has expanded and become critical, the amount and complexity of mobile apps have also increased. To assure reliability, these apps require software engineering methods, mainly verification, validation, and testing. However, mobile app testing is a challenging activity due to the diversity and limitations found in mobile devices. Thus, it would be interesting to characterize mobile apps in hopes of assisting in the definition of more efficient and effective testing approaches. Objective: this paper aims to identify and quantify the specific characteristics of mobile apps so that testers can draw from this knowledge and tailor software testing activities to mobile apps. We investigate the presence of automated tests, adopted frameworks, external connectivity, graphical user interface (GUI) elements, sensors, and different system configurations. Method: we developed a tool to support the automatic extraction of characteristics from Android apps. We conducted an empirical study with a sample of 663 open source mobile apps. Results: we found that one third of the projects perform automated testing. The frameworks used in these projects can be divided into three groups: unit testing, GUI testing, and mocking. There is a medium correlation between project size and test presence. Specific features of mobile apps (connectivity, GUI, sensors, and multiple configurations) are present in the projects, however, they are fully covered by tests. Conclusion: automated tests are still not developed in a systematic way. Interestingly, measures of app popularity (number of downloads and rating) do not seem to be correlated with the presence of tests. However, the results show a correlation of the project size and more critical domains with the existence of automated tests. Although challenges such as connectivity, sensors, and multiple configurations are present in the examined apps, only one tool has been identified to support the testing of these challenges.

Currently, mobile devices are available in several formats; their applications can be grouped into three categories: native applications, Web applications and hybrid applications. Hybrid applications use Web programming features, such as HTML, CSS and Javascript, and are able to access native device capabilities, such as GPS, camera and contacts. Several works automated software testing in native applications and this fact suggests a lack of testing approaches for hybrid applications. In this paper, we investigate test automation process in hybrid applications running graphical user interface (GUI) under various mobile platforms. Three applications were tested in five different mobile devices (Android and iOS). The manual inspections indicated correct functioning of the features under test. Automated executions found some differences between the platforms, such as in the user interface and test script coding.

Mobile computing has become ubiquitous, thus the amount and complexity of mobile applications have challenged software engineering practices. To overcome the challenges brought by mobile applications, the adoption of repeatable, systematic and mainly automated tests have been researched. In this paper, we look into open source projects in hopes of identifying how automated tests are applied to mobile applications developed for the Android platform. We analyzed the automated tests to identify the frameworks adopted, the relation between the production and the testing code, and how the following challenges have been dealt with: connectivity, rich GUIs, limited resources, sensors, and multiple configurations.

In software testing, a program is executed in hopes of revealing faults. Over the years, specific testing criteria have been proposed to help testers to devise test cases that cover the most relevant faulty scenarios. Symbolic execution has been used as an effective way of automatically generating test data that meet those criteria. Although this technique has been used for over three decades, several challenges remain and there is a lack of research on how often they appear in real-world applications. In this paper, we analyzed two samples of open source Java projects in order to understand the characteristics that may hinder the generation of unit test data using symbolic execution. The first sample, named SF100, is a third party corpus of classes obtained from 100 projects hosted by SourceForge. The second sample, called R47, is a set of 47 well-known and mature projects we selected from different repositories. Both samples are compared with respect to four dimensions that influence symbolic execution: path explosion, constraint complexity, dependency, and exception-dependent paths. The results provide valuable insight into how researchers and practitioners can tailor symbolic execution techniques and tools to better suit the needs of different Java applications.

Testing exception scenarios is a challenging task in the context of test data generation based on symbolic execution.In such a context, test data is generated based on constraints explicitly declared in the code.However, constraints required to activate specific exceptions may not be directly declared in the code.In such a case, implicit constraints have to be inferred from exception handling mechanisms.Given that exceptions can be raised in several situations, finding constraints to generate test data to exercise all possible faulty scenarios can significantly increase the number of paths and constraints, which can cause or aggravate path explosion issues.This paper reports on an investigation that we carried out to gauge the cost (i.e., number of path constraints) of four data generation approaches aimed at covering exception dependent paths.

Testing plays a key role in assessing the quality of a software product. During testing, a program is run in hopes of finding faults. As exhaustive testing is seldom possible, specific testing criteria have been proposed to help testers to devise test cases that cover the most relevant faulty scenarios. Manually creating test cases that satisfy these criteria is time consuming, error prone, and unwieldy. Symbolic execution has been used as an effective way of automatically generating test data that meets those criteria. Although this technique has been used for over three decades, several challenges remain, such as path explosion, precision of floating-point data, constraints with complex expressions, and dependency of external libraries. In this paper, we explore a sample of 100 open source Java projects in order to analyze characteristics that are relevant to generate test data using symbolic execution. The results provide valuable insight into how researchers and practitioners can tailor symbolic execution techniques and tools to better suit the needs of different Java applications.

SUMMARY The behavior of composed Web services depends on the results of the invoked services; unexpected behavior of one of the invoked services can threat the correct execution of an entire composition. This paper proposes an event‐based approach to black‐box testing of Web service compositions based on event sequence graphs, which are extended by facilities to deal not only with service behavior under regular circumstances (i.e., where cooperating services are working as expected) but also with their behavior in undesirable situations (i.e., where cooperating services arenotworking as expected). Furthermore, the approach can be used independently of artifacts (e.g., Business Process Execution Language) or type of composition (orchestration/choreography). A large case study, based on a commercial Web application, demonstrates the feasibility of the approach and analyzes its characteristics. Test generation and execution are supported by dedicated tools. Especially, the use of an enterprise service bus for test execution is noteworthy and differs from other approaches. The results of the case study encourage to suggest that the new approach has the power to detect faults systematically, performing properly even with complex and large compositions. Copyright © 2012 John Wiley & Sons, Ltd.

Service-oriented architectures and Web services have been widely adopted by enterprises to pervade integration among software systems. As reliable services are essential to assure that these systems work correctly, formal and systematic testing should be performed. This paper reports the application of a model-based approach to test Web services in the context of real-world applications of a multinational computer technology corporation. The employed approach is called ESG4WSC, in which an event-driven model is provided to support modeling and test case generation, as well as an environment to support the concretization and test execution.

Testing from finite state machines has been widely investigated due to its well-founded and sound theory as well as its practical application in different areas, e.g., Web-based systems and protocol testing. There has been a recurrent interest in developing methods capable of generating test suites that detect all faults in a given fault domain. However, the proposal of new methods motivates the comparison with traditional methods. In this context, we conducted a set of experiments that compares W, HSI, H, SPY, and P methods. The results have shown that H, SPY, and P methods produce smaller test suites than traditional methods (W, HSI). Although the P method presented the shortest test suite in most cases, its reduction is smaller compared with H and SPY. We have also observed that the reduction ratio in partial machines is smaller than that in complete machines.

Web Services are a technology based on the Service Oriented Architecture that enables communication between applications through the Internet. Using Web Services, it is possible to send any type of information in any form of encryption. In this context, different techniques have been used to attach binary files in SOAP messages. However, there is no consensus on which technique has the best performance. This paper presents a performance evaluation study with three techniques for Web Service attachments: Pure Binary, MTOM and SwA. A testing environment was configured to verify the influence of the network and the size of files. Also, we present a tool called WSATPerf that supports the execution of the performance evaluation.

Web services are an emerging Service-Oriented Architecture technology to integrate applications using open standards based on XML. Software Engineering tools integration is a promising area since companies adopt different software processes and need different tools on each activity. Software engineers could take advantage of software engineering tools available as web services and create their own workflow for integrating the required tools. In this paper, we propose the development of testing tools designed as web services and discuss the pros and cons of this idea. We developed a web service for structural testing of Java programs called JaBUTiService, which is based on the stand-alone tool JaBUTi. We also present an usage example of this service with the support of a desktop front-end and pre prepared scripts. A set of 62 classes of the library Apache-Commons-BeanUtils was used for this test and the results are discussed.