Mutation testing may be used to guide test case generation and as a technique to assess the quality of test suites. Despite being used frequently, mutation testing is not so commonly applied in the mobile world. One critical challenge in mutation testing is dealing with its computational cost. Generating mutants, running test cases over each mutant, and analyzing the results may require significant time and resources. This research aims to contribute to reducing Android mutation testing costs. It implements mutation testing operators (traditional and Android-specific) according to mutant schemata (implementing multiple mutants into a single code file). It also describes an Android mutation testing framework developed to execute test cases and determine mutation scores. Additional mutation operators can be implemented in JavaScript and easily integrated into the framework. The overall approach is validated through case studies showing that mutant schemata have advantages over the traditional mutation strategy (one file per mutant). The results show mutant schemata overcome traditional mutation in all evaluated aspects with no additional cost: it takes 8.50% less time for mutant generation, requires 99.78% less disk space, and runs, on average, 6.45% faster than traditional mutation. Moreover, considering sustainability metrics, mutant schemata have 8,18% less carbon footprint than traditional strategy.

With the massive adoption of mobile devices, it became more mandatory for developers to provide high-quality applications. Nowadays, mobile devices are used for different purposes: entertainment, shopping, banking, and communication. Moreover, mobile devices can communicate and exchange information with various IoT devices distributed across the city. However, mobile application testing has different challenges when compared to other types of applications (i.e., desktop and client-server applications). First, we must consider mobile devices' different characteristics and limitations, such as connectivity, screen size, density, sensors, and limited battery. Second, there is a wide range of mobile devices from diverse vendors and models. Hence, there is a need to consider different device configurations to reduce compatibility issues that may occur in a high-fragmented ecosystem. In this case, several tools and services with various features and business models aim to run tests on multiple devices. In this practical experience report, we present the initial results of implementing a testing tool/service at Von Braun Labs to support the execution of tests across multiple Android devices. The stakeholders stated the need to (i) execute the tests on physical devices; and (ii) the tool/service must support tests that interact with a specialized IoT device. We start the study by comparing different tools/services to select the most suitable one for Von Braun Labs. We propose a comparison framework to help evaluate six tools/services based on their technical, usability, and customization features. Then, we present a case study with an app from Von Braun Labs to validate the selected testing environment. Finally, we discuss the lessons learned, contributions, and future directions, pinpointing the need for a testing process since the beginning of the development project and the importance of lessening the gap between academia and industry.

The low performance of software projects generally arises from erroneous, omitted, misinterpreted, or conflicting requirements. To produce better quality specifications, the practise of requirements reuse through requirement patterns has been widely debated in the secondary literature. However, a tertiary study that provides an overview of secondary studies on the state of the art and the practise of requirement patterns does not exist. This study describes a study of secondary literature on requirement patterns under a perspective on research and practise. The identification and selection methods of secondary studies include automatic search on five sources, inclusion, and exclusion criteria, the snowballing technique, and the quality assessment of those studies. Four secondary studies are considered relevant according to the purpose of this research from a 26‐distinct‐study group. The authors’ contribution is two‐fold: the tertiary study itself and a preliminary research agenda dealing with state of the art and practise on requirement patterns.

This Innovate Practice Full Paper reports a Industry/Academia partnership to improve the qualification of both students and professionals. Consider the following scenario: a city located in a third world country, with one of the lowest countrywide educational performances, hosting an industrial pole based on Federal tax incentives aiming to develop a R&D cluster. The city has basic services issues, from health to urban violence, making the attraction of professionals from other locations unfeasible. As a consequence, the lack of professionals with R&D profile is now considered a relevant risk to the continuation of the city's industrial policies. Therefore, initiatives that shorten the qualification life-cycle of local professionals, while preparing senior students with real-world experience, are welcome. An example of such initiative, reported in this paper, was the software test residence, formed by professionals from the industry, and senior undergraduate and graduate students of a local university. The proposal consisted on defining a set of learning activities where students and professionals would be partners, but still being aware that they belong to different realities (with respect to timing, accountability and accreditation). The challenge can be summarized as follows: how to jointly qualify students and professionals, while respecting the peculiarities of both audiences. In order to address the issue in a short time period, representatives from a global company with R&D site in the city partnered with a local university to define a learning program inspired by the software residence approach. The university's course syllabus in Software Test was reviewed and customized to the on-the-job training subject of interest. Moreover, the approach was based on three nonnegotiable principles: Accountability, Excellence and Sustainability. The experience was successful, reaching 25 participants from four R&D organizations and the university. Results obtained include qualification of 23 participants (only two dropped off), the development of an open source educational material composed by a set of slides for testing automation classes, theory-practice of software testing targeting Test Maturity Model integration (TMMi) level 2, and the improvement in the collaboration between academia and industry for other projects. The paper will detail the experience, and list some lessons learned, including how to synchronize the schedules.

Fault localization has been one of the most manual and costly software debugging activities. The spectrum-based fault localization is the most studied and evaluated fault localization approach. Mutation-based fault localization is a promising approach but with a high computational cost. We propose a novel mutation execution strategy named Failed-Test Oriented Mutant Execution Strategy (FTMES) for improving the efficacy of fault localization techniques and also reduce the computational cost of these techniques. Our proposed approach and eight other baselines were evaluated against 221 real faults. The results show that FTMES outperformed others with respect to efficiency (computational cost) while maintaining similar accuracy.

Mutation Testing has been proved to be effective in detecting faults. However, it suffers from the large number of mutants generated and analyzed when we apply the operators to create mutants. To alleviate this problem, this paper proposes an approach to find a reduced set of mutation operators that reduces the generated mutants and maintains the test effectiveness. Our approach adapts the semantic characterization of faults in order to find this set of mutation operators that will reduce the cost of Mutation Testing. The experimental results indicate that our approach is able to find a set of operators that reduce the mutation cost while maintaining the test effectiveness.

The creation of a suite of unit testing is preceded by the selection of which components (code units) should be tested. This selection is a significant challenge, usually made based on the team member's experience or guided by defect prediction or fault localization models. We modeled the selection of components for unit testing with limited resources as a multi-objective problem, addressing two different objectives: maximizing benefits and minimizing testing cost. To measure the benefit of a component, we made use of metrics from static analysis (cost of future maintenance), dynamic analysis (risk of fault, and frequency of calls), and business value. We tackled gaps and challenges in the literature to formulate an effective method, the Selector of Software Components for Unit Testing (SCOUT). SCOUT provides an automated extraction of all necessary data followed by a multi-objective optimization process. SCOUT is a method able to assist testers in different domains, and the Android platform was chosen to perform our experiments, taking nine leading open-source applications as our subjects. SCOUT was compared with two of the most frequently used strategies in terms of efficacy. We also compared the effectiveness and efficiency of seven algorithms in solving a multi-objective component selection problem. Our experiments were performed under different scenarios, and reveal the potential of SCOUT in reducing the market vulnerability, compared to others approaches. To the best of our knowledge, SCOUT is the first method to assist in an automated way software testing managers in selecting components for the development of unit testing, combining static and dynamic metrics and business value.

Humans are hunters and love the chase — they hunt for food, they hunt for bugs in software. In the last decade, testing research has gone deeper and broader to help with the challenging task of catching bugs. Much of the literature approaches the problem from a theoretical-technical perspective and is often oriented to automated solutions. Yet, there is a gap between industry testing problems and research testing solutions. We take a different perspective and consider the human component as a major part of the solution for practical testing problems. Many of these human-related issues are reported in academic surveys of practitioners. We highlight the importance of human factors in testing by introducing a hunting metaphor. We also bring attention to evidence on cognitive support demands in unit test practices. An initial framework is proposed as an effort to bring understanding of cognitive support demands, and provides direction for further research on unit testing tools which support tester skill improvement.

Usability is related to software quality, improving its ability to be understood, operated and attractive to users.We proposed the Design Usability Evaluation (DUE) technologies to allow identifying usability problems earlier in the development of Web applications, through the inspection of mockups.While we found that the DUE technique and tool were effective and efficient in the identification of usability problems, we saw the need to investigate their acceptance in practitioners' work environment.This paper reports the results from a study evaluating the acceptance of the DUE technologies from the point of view of software engineers.We asked questions based on the indicators from the Technology Acceptance Model and identified that a majority of the software engineers who participated in the study: (a) found the DUE technologies useful and easy to use for supporting the usability inspection process; and (b) would regularly use the DUE technologies for future inspections in their job.Nevertheless, the practitioners indicated that the technique should be refined in order to reduce the ambiguity and repetition of some of its items, while the tool should become more intuitive.

This paper tries to combine SQL mutation testing techniques with evolutionary computation aiming to improve the test data to SQL instructions. Based on a heuristic perspective it presents an approach that uses Genetic Algorithms (GA) to select tuples from an original database trying to reduce this one in an effective data set. The goal is to find a reduced data set which is able to detect a large number of faults in the SQL instructions of a given application. During the evolutionary process, the analysis of mutants is used to assess each set of data test selected by GA. The results obtained from experiments reveal a good performance using GA metaheuristic.

One of the main problems to perform the Software Testing is to find a set of tests (subset from input domain of the problem) which is effective to detect the remaining bugs in the software. The Search-Based Software Testing (SBST) approach uses metaheuristics to find low cost set of tests with a high effectiveness to detect bugs. From several existing test criteria, Mutation Testing is considered quite promising to reveal bugs, despite its high computational cost, due to the great quantity of mutant programs generated. Therefore, this paper addresses the problem of selecting mutant programs and test cases in Mutation Testing context. To this end, it is proposed a Coevolutionary Genetic Algorithm (CGA) and the concept of Genetic Effectiveness, describing a new representation and implementing new genetic operators. The CGA is applied in five benchmarks and the results are compared to other five methods, showing a better performance of the proposed algorithm in subsets automatic selection with better mutation score and greater reduction of computational cost, specifically the amount of testing, when compared with exhaustive test.

The development of an effective and efficient method for generating test data is an extremely challenging process which directly impacts the time that could be spent on activities relevant to software testing. Therefore, various researches related to this area have been carried out. Among the techniques for automatically generating test data, we highlight the use of metaheuristics, a promising area called Search-Based Software Testing (SBST). Thus, this article proposes the use of an Elitist Genetic Algorithm (GA) as a tool for generation and selection of test data applied in Mutation Testing for different benchmarks. The results indicate a good performance of the algorithm used in the benchmarks.

Web services are an emerging Service-Oriented Architecture technology to integrate applications using open standards based on XML. Software Engineering tools integration is a promising area since companies adopt different software processes and need different tools on each activity. Software engineers could take advantage of software engineering tools available as web services and create their own workflow for integrating the required tools. In this paper, we propose the development of testing tools designed as web services and discuss the pros and cons of this idea. We developed a web service for structural testing of Java programs called JaBUTiService, which is based on the stand-alone tool JaBUTi. We also present an usage example of this service with the support of a desktop front-end and pre prepared scripts. A set of 62 classes of the library Apache-Commons-BeanUtils was used for this test and the results are discussed.

Abstract This paper describes intra‐method control‐flow and data‐flow testing criteria for the Java bytecode language. Six testing criteria are considered for the generation of testing requirements: four control‐flow and two data‐flow based. The main reason to work at a lower level is that, even when there is no source code, structural testing requirements can still be derived and used to assess the quality of a given test set. It can be used, for instance, to perform structural testing on third‐party Java components. In addition, the bytecode can be seen as an intermediate language, so the analysis performed at this level can be mapped back to the original high‐level language that generated the bytecode. To support the application of the testing criteria, we have implemented a tool named JaBUTi (Java Bytecode Understanding and Testing). JaBUTi is used to illustrate the application of the ideas developed in this paper. Copyright © 2006 John Wiley & Sons, Ltd.

Mutation testing (Mutation Analysis), although powerful in revealing faults, is considered a computationally expensive criterion, due to the high number of mutants created and the effort to determine the equivalent mutants. Using mutation-based alternative testing criteria it is possible to reduce the number of mutants but it is still necessary to determine the equivalent ones. In this paper the Bayesian Learning(one of the Artificial Intelligence techniques used in machine learning) is investigated to define the Bayesian Learning-Based Equivalent Detection Technique (BaLBEDeT), which provides guidelines to help the tester to analyze the live mutants in order to determine the equivalent ones.

Abstract Mutation testing, originally proposed for unit testing, has been extended to integration testing with the proposition of the Interface Mutation criterion. This paper presents the results of an experiment using two mutation‐based testing criteria for unit and integration testing phases: the Mutation Analysis and the Interface Mutation adequacy criteria, respectively. The aim is to investigate how they can be used in a complementary way during the testing activity, establishing an incremental testing strategy comprising the unit and integration testing phases and guidelines on how to obtain a high mutation score with respect to mutation testing with a low cost, in terms of the number of mutants generated. Copyright © 2001 John Wiley & Sons, Ltd.