Summary Model‐based test design is increasingly being applied in practice and studied in research. Model‐based testing (MBT) exploits abstract models of the software behaviour to generate abstract tests, which are then transformed into concrete tests ready to run on the code. Given that abstract tests are designed to cover models but are run on code (after transformation), the effectiveness of MBT is dependent on whether model coverage also ensures coverage of key functional code. In this article, we investigate how MBT approaches generate tests from model specifications and how the coverage of tests designed strictly based on the model translates to code coverage. We used snowballing to conduct a systematic literature review. We started with three primary studies, which we refer to as the initial seeds. At the end of our search iterations, we analysed 30 studies that helped answer our research questions. More specifically, this article characterizes how test sets generated at the model level are mapped and applied to the source code level, discusses how tests are generated from the model specifications, analyses how the test coverage of models relates to the test coverage of the code when the same test set is executed and identifies the technologies and software development tasks that are on focus in the selected studies. Finally, we identify common characteristics and limitations that impact the research and practice of MBT:(i) some studies did not fully describe how tools transform abstract tests into concrete tests,(ii) some studies overlooked the computational cost of model‐based approaches and (iii) some studies found evidence that bears out a robust correlation between decision coverage at the model level and branch coverage at the code level. We also noted that most primary studies omitted essential details about the experiments.

Refactoring is the process of changing the internal structure of software to improve its quality without modifying its external behavior. Empirical studies have repeatedly shown that refactoring has a positive impact on the understandability and maintainability of software systems. However, before carrying out refactoring activities, developers need to identify refactoring opportunities. Currently, refactoring opportunity identification heavily relies on developers’ expertise and intuition. In this paper, we investigate the effectiveness of machine learning algorithms in predicting software refactorings. More specifically, we train six different machine learning algorithms (i.e., Logistic Regression, Naive Bayes, Support Vector Machine, Decision Trees, Random Forest, and Neural Network) with a dataset comprising over two million refactorings from 11,149 real-world projects from the Apache, F-Droid, and GitHub ecosystems. The resulting models predict 20 different refactorings at class, method, and variable-levels with an accuracy often higher than 90 percent. Our results show that (i) Random Forests are the best models for predicting software refactoring, (ii) process and ownership metrics seem to play a crucial role in the creation of better models, and (iii) models generalize well in different contexts.

Software testing involves probing into the behavior of software systems to uncover faults. Most testing activities are complex and costly, so a practical strategy that has been adopted to circumvent these issues is to automate software testing. There has been a growing interest in applying machine learning (ML) to automate various software engineering activities, including testing-related ones. In this paper, we set out to review the state-of-the art of how ML has been explored to automate and streamline software testing and provide an overview of the research at the intersection of these two fields by conducting a systematic mapping study. We selected 48 primary studies. These selected studies were then categorized according to study type, testing activity, and ML algorithm employed to automate the testing activity. The results highlight the most widely used ML algorithms and identify several avenues for future research. We found that ML algorithms have been used mainly for test-case generation, refinement, and evaluation. Also, ML has been used to evaluate test oracle construction and to predict the cost of testing-related activities. The results of this paper outline the ML algorithms that are most commonly used to automate software-testing activities, helping researchers to understand the current state of research concerning ML applied to software testing. We also found that there is a need for better empirical studies examining how ML algorithms have been used to automate software-testing activities.

A common software product line strategy involves plugin-based web systems that support simple and quick incorporation of custom behaviors. As a result, they have been widely adopted to create web-based applications. Indeed, the popularity of ecosystems that support plugin-based development (e.g., WordPress) is largely due to the number of customization options available as community-contributed plugins. However, plugin-related vulnerabilities tend to be recurrent, exploitable and hard to be detected and may lead to severe consequences for the customized product. Hence, there is a need to further understand such vulnerabilities to enable preventing relevant security threats. Therefore, we conducted an exploratory study to characterize vulnerabilities caused by plugins in web-based systems. To this end, we went over WordPress vulnerability bulletins cataloged by the National Vulnerability Database as well as associated patches maintained by the WordPress plugins repository. We identified the main types of vulnerabilities caused by plugins as well as their impact and the size of the patch to fix the vulnerability. Moreover, we identified the most common security-related topics discussed among WordPress developers. We observed that, while plugin-related vulnerabilities may have severe consequences and might remain unnoticed for years before being fixed, they can commonly be mitigated with small and localized changes to the source code. The characterization helps to provide an understanding on how typical plugin-based vulnerabilities manifest themselves in practice. Such information can be helpful to steer future research on plugin-based vulnerability detection and prevention.

Context: while the mobile computing market has expanded and become critical, the amount and complexity of mobile apps have also increased. To assure reliability, these apps require software engineering methods, mainly verification, validation, and testing. However, mobile app testing is a challenging activity due to the diversity and limitations found in mobile devices. Thus, it would be interesting to characterize mobile apps in hopes of assisting in the definition of more efficient and effective testing approaches. Objective: this paper aims to identify and quantify the specific characteristics of mobile apps so that testers can draw from this knowledge and tailor software testing activities to mobile apps. We investigate the presence of automated tests, adopted frameworks, external connectivity, graphical user interface (GUI) elements, sensors, and different system configurations. Method: we developed a tool to support the automatic extraction of characteristics from Android apps. We conducted an empirical study with a sample of 663 open source mobile apps. Results: we found that one third of the projects perform automated testing. The frameworks used in these projects can be divided into three groups: unit testing, GUI testing, and mocking. There is a medium correlation between project size and test presence. Specific features of mobile apps (connectivity, GUI, sensors, and multiple configurations) are present in the projects, however, they are fully covered by tests. Conclusion: automated tests are still not developed in a systematic way. Interestingly, measures of app popularity (number of downloads and rating) do not seem to be correlated with the presence of tests. However, the results show a correlation of the project size and more critical domains with the existence of automated tests. Although challenges such as connectivity, sensors, and multiple configurations are present in the examined apps, only one tool has been identified to support the testing of these challenges.

Persuasion profiling has been shown to be a promising method to personalize persuasive messages that can influence users. In the context of learning, persuade students towards following pedagogical tasks is an important role of teachers and intelligent educational environments. To accomplish persuasion, it is necessary to measure students' susceptibility to influence principles. Susceptibility to Persuasion Scale (STPS) is an instrument that can measure users' susceptibility to influence principles. Currently, STPS is validated only for the English language. Thus, this study presents the translation, cultural adaptation, and initial validation of the Brazilian Portuguese version of the STPS questionnaire. We probed into the validity and reliability of the resulting questionnaire with 582 participants. Our results indicate that the final version of our questionnaire, which contains 19-items and 6-components, has a reasonable internal consistence (Cronbach's alpha = 0.78) and can adequately identify which influence principles have more impact on user' decisions.

Mobile computing has become ubiquitous, thus the amount and complexity of mobile applications have challenged software engineering practices. To overcome the challenges brought by mobile applications, the adoption of repeatable, systematic and mainly automated tests have been researched. In this paper, we look into open source projects in hopes of identifying how automated tests are applied to mobile applications developed for the Android platform. We analyzed the automated tests to identify the frameworks adopted, the relation between the production and the testing code, and how the following challenges have been dealt with: connectivity, rich GUIs, limited resources, sensors, and multiple configurations.

In software testing, a program is executed in hopes of revealing faults. Over the years, specific testing criteria have been proposed to help testers to devise test cases that cover the most relevant faulty scenarios. Symbolic execution has been used as an effective way of automatically generating test data that meet those criteria. Although this technique has been used for over three decades, several challenges remain and there is a lack of research on how often they appear in real-world applications. In this paper, we analyzed two samples of open source Java projects in order to understand the characteristics that may hinder the generation of unit test data using symbolic execution. The first sample, named SF100, is a third party corpus of classes obtained from 100 projects hosted by SourceForge. The second sample, called R47, is a set of 47 well-known and mature projects we selected from different repositories. Both samples are compared with respect to four dimensions that influence symbolic execution: path explosion, constraint complexity, dependency, and exception-dependent paths. The results provide valuable insight into how researchers and practitioners can tailor symbolic execution techniques and tools to better suit the needs of different Java applications.

Mutation analysis is a testing strategy that consists of using supporting tools to seed artificial faults in the original code of a software under test, generating faulty programs (“mutants”) that are supposed to produce incorrect outputs. Novice programmers suffer of a wide range of deficits due to defective training processes. We argue that the incorporation of experiences on mutation testing in programming courses adds valuable knowledge to the learning process. In this paper we evaluate the effects of using mutation testing to improve the learning process of students in programming courses. We present results of experiments and analysis involving undergraduate students. These experiments are the continuation of a previous work in which we raise empirical evidences that the adequate incorporation of mutation testing in programming courses contributes to form an effective environment that fosters learning. To do so, we provide a mutation testing tool to promote the practice of mutation testing by novice programmers. Through practical experiences and several analysis survey we measured the effects of using the mutation testing criterion to teach programming. In addition, we collected the opinion of senior students who already knew mutation testing concepts about their opinion on the usage of mutation concepts to teach novice programmers. Our findings reveal that the effective use of mutation analysis concepts contributes to the learning process, making students see the code as a product under development that is the result of a careful manual coding process which they need for measuring and predicting the effect of each command. The main contributions discussed in this paper are: (1) presenting results of an empirical analysis involving undergraduate students, thus giving us preliminary evidence on the effects of the novel practice; (2) exposing possible practices to explore mutation testing in programming classes, highlighting the limitations and strengths of such strategy; and (3) a mutation testing tool for educational purposes.

Testing exception scenarios is a challenging task in the context of test data generation based on symbolic execution.In such a context, test data is generated based on constraints explicitly declared in the code.However, constraints required to activate specific exceptions may not be directly declared in the code.In such a case, implicit constraints have to be inferred from exception handling mechanisms.Given that exceptions can be raised in several situations, finding constraints to generate test data to exercise all possible faulty scenarios can significantly increase the number of paths and constraints, which can cause or aggravate path explosion issues.This paper reports on an investigation that we carried out to gauge the cost (i.e., number of path constraints) of four data generation approaches aimed at covering exception dependent paths.

Testing plays a key role in assessing the quality of a software product. During testing, a program is run in hopes of finding faults. As exhaustive testing is seldom possible, specific testing criteria have been proposed to help testers to devise test cases that cover the most relevant faulty scenarios. Manually creating test cases that satisfy these criteria is time consuming, error prone, and unwieldy. Symbolic execution has been used as an effective way of automatically generating test data that meets those criteria. Although this technique has been used for over three decades, several challenges remain, such as path explosion, precision of floating-point data, constraints with complex expressions, and dependency of external libraries. In this paper, we explore a sample of 100 open source Java projects in order to analyze characteristics that are relevant to generate test data using symbolic execution. The results provide valuable insight into how researchers and practitioners can tailor symbolic execution techniques and tools to better suit the needs of different Java applications.

In this paper we argue that the incorporation of experiences on testing activities, in particular mutation testing, in programming courses adds valuable knowledge to the learning process. Mutation testing is centered on the idea of creating test data for uncovering seeded faults in programs that slightly differ from the original program. These faulty programs are called mutants. Through source code analysis and test case execution, testers have to identify the differences between the original program and the mutants. To do so, testers must have a sound understanding of the program's control flow and instructions. This broad understanding of programming represents a key skill for novice students in programming courses. We evaluate the effects of using mutation testing to improve the learning process of novice students in programming courses. We conclude that the introduction of mutation testing in the learning process provides students with learning experiences that go beyond traditional lectures and hands-on programming courses.

Mutation analysis modifies a program by applying syntactic rules, called mutation operators, systematically to create many versions of the program (mutants) that differ in small ways. Testers then design tests to cause the mutants to behave differently from the original program. Mutation testing is widely considered to result in very effective tests, however, it is also quite costly. Cost comes from the many mutants that are created, the number of tests that are needed to kill the mutants, and the difficulty of deciding whether mutants behave equivalently to the original program. One-op mutation theorizes that cost can be reduced by using a single, very powerful, mutation operator that leads to tests that are almost as effective as if all operators are used. Previous research proposed the statement deletion operator (SDL) and found promising results. This paper investigates the use of SDL-mutation in a new context, the language C, and poses additional empirical questions, including whether other operators can be used. We carried out a controlled experiment in which cost and effectiveness of each individual C mutation operator were collected for 39 different subject programs. Experimental data are used to define a cost-effectiveness metric to choose the best single operator for one-op mutation.

High-level language virtual machines (HLL VMs) are now widely used to implement high-level programming languages. To a certain extent, their widespread adoption is due to the software engineering benefits provided by these managed execution environments, for example, garbage collection (GC) and cross-platform portability. Although HLL VMs are widely used, most research has concentrated on high-end optimizations such as dynamic compilation and advanced GC techniques. Few efforts have focused on introducing features that automate or facilitate certain software engineering activities, including software testing. This paper suggests that HLL VMs provide a reasonable basis for building an integrated software testing environment. As a proof-of-concept, we have augmented a Java virtual machine (JVM) to support weak mutation analysis. Our mutation-aware HLL VM capitalizes on the relationship between a program execution and the underlying managed execution environment, thereby speeding up the execution of the program under test and its associated mutants. To provide some evidence of the performance of our implementation, we conducted an experiment to compare the efficiency of our VM-based implementation with a strong mutation testing tool (muJava). Experimental results show that the VM-based implementation achieves speedups of as much as 89% in some cases.

The interface is the main mechanism of communication between user and system features. In educational software, successful user interface designs minimize the cognitive load on users, thereby users can direct their efforts to maximize their understanding of the educational concepts being presented. We investigated whether a reduced interface make few cognitive demands on users in comparison to a complete interface. In this context, this research aims at analyzing a reduced and a complete interface of an interactive geometry software, and verify the educational benefits they provide. To this end, we designed the interfaces and carried out an experiment involving 69 undergraduate students. The experimental results indicate that an interface that hides advanced and extraneous features helps novice users to perform slightly better than novice users using a complete interface. After receiving proper training, however, a complete interface makes users more productive than a reduced interface.