Mutation testing may be used to guide test case generation and as a technique to assess the quality of test suites. Despite being used frequently, mutation testing is not so commonly applied in the mobile world. One critical challenge in mutation testing is dealing with its computational cost. Generating mutants, running test cases over each mutant, and analyzing the results may require significant time and resources. This research aims to contribute to reducing Android mutation testing costs. It implements mutation testing operators (traditional and Android-specific) according to mutant schemata (implementing multiple mutants into a single code file). It also describes an Android mutation testing framework developed to execute test cases and determine mutation scores. Additional mutation operators can be implemented in JavaScript and easily integrated into the framework. The overall approach is validated through case studies showing that mutant schemata have advantages over the traditional mutation strategy (one file per mutant). The results show mutant schemata overcome traditional mutation in all evaluated aspects with no additional cost: it takes 8.50% less time for mutant generation, requires 99.78% less disk space, and runs, on average, 6.45% faster than traditional mutation. Moreover, considering sustainability metrics, mutant schemata have 8,18% less carbon footprint than traditional strategy.

Software testing is an important part of the software development lifecycle. As it is a highly sought-after skill in the industry, it is not surprising that there has been a great deal of research into the teaching of software testing in higher education. Most of this research proposes or evaluates pedagogical approaches or software testing tools to assist teachers in educating the next generation of software engineers. These evaluations are often limited to measuring teachers’ opinions about the use of a novel pedagogical approach or an educational tool and students’ acceptance and performance in terms of desired software testing skills. While tools and pedagogical approaches address specific aspects of a course, to date, little attention has been paid to the opinions of the students about all the individual aspects of a software testing course. This paper aims to address this missing student perspective by taking a holistic view of software testing course designs. To address this gap, an exploratory study was performed by distributing a questionnaire to 103 students from ten different courses to gauge their opinions on a software testing course they are enrolled in. The results show that students generally have a positive perception of the different aspects of their software testing course. However, several areas for improvement were suggested based on the gathered data.

Abstract Platforms to support novices learning to program are often accompanied by automated next-step hints that guide them towards correct solutions. Many of those approaches are data-driven, building on historical data to generate higher quality hints. Formal specifications are increasingly relevant in software engineering activities, but very little support exists to help novices while learning. Alloy is a formal specification language often used in courses on formal software development methods, and a platform—Alloy4Fun—has been proposed to support autonomous learning. While non-data-driven specification repair techniques have been proposed for Alloy that could be leveraged to generate next-step hints, no data-driven hint generation approach has been proposed so far. This paper presents the first data-driven hint generation technique for Alloy and its implementation as an extension to Alloy4Fun, being based on the data collected by that platform. This historical data is processed into graphs that capture past students’ progress while solving specification challenges. Hint generation can be customized with policies that take into consideration diverse factors, such as the popularity of paths in those graphs successfully traversed by previous students. Our evaluation shows that the performance of this new technique is competitive with non-data-driven repair techniques. To assess the quality of the hints, and help select the most appropriate hint generation policy, we conducted a survey with experienced Alloy instructors.

Software testing is an indispensable component of software development, yet it often receives insufficient attention. The lack of a robust testing culture within computer science and informatics curricula contributes to a shortage of testing expertise in the software industry. Addressing this problem at its root -education- is paramount. In this paper, we conduct a comprehensive mapping review of software testing courses, elucidating their core attributes and shedding light on prevalent subjects and instructional methodologies. We mapped 117 courses offered by Computer Science (and related) degrees in 49 academic institutions from four Western European countries, namely Belgium, Italy, Portugal and Spain. The testing subjects were mapped against the conceptual framework provided by the ISO/IEC/IEEE 29119 standard on software testing. Among the results, the study showed that dedicated software testing courses are offered by only 39% of the analysed universities, whereas the basics of software testing are taught in at least one course at every university. The analysis of the software testing topics highlights the gaps that need to be filled in order to better align the current academic offerings with the real industry needs.

Mutation testing has evolved beyond academic research, is deployed in industrial and open-source settings, and is increasingly part of universities' software engineering curricula. While many mutation testing tools exist, each with different strengths and weaknesses, integrating them into educational activities and exercises remains challenging due to the tools' complexity and the need to integrate them into a development environment. Additionally, it may be desirable to use different tools so that students can explore differences, e.g., in the types or numbers of generated mutants. Asking students to install and learn multiple tools would only compound technical complexity and likely result in unwanted differences in how and what students learn. This paper presents FRAFOL, a framework for learning mutation testing. FRAFOL provides a common environment for using different mutation testing tools in an educational setting.

It is important to properly test developed software because this may contribute to fewer bugs going unreported in deployed software. Often, little attention is spent on the topic of software testing in curricula, yielding graduate students without adequate preparation to deal with the quality standards required by the industry. This problem could be tackled by introducing bite-sized software testing education capsules that allow teachers to introduce software testing to their students in a less time-consuming manner and with a hands-on component that will facilitate learning. In order to design appropriate software testing educational tools, it is necessary to consider both the software testing needs of the industry and the cognitive models of students. This work-in-progress paper proposes an experimental design to gain an understanding of the cognitive strategies used by students during test case design based on real-life cases. Ultimately, the results of the experiment will be used to develop educational support for teaching software testing.

Frequently software testing tends to be neglected at the beginning of the projects, only performed on the late stage. However, it is possible to benefit from combining requirement with testing specification activities. On one hand, acceptance tests specification will require less manual effort since they are defined or generated automatically from the requirements specification. On the other hand, the requirements specification itself will end up having higher quality due to the use of a more structured language, reducing typical problems such as ambiguity, inconsistency and incorrectness. This research proposes an approach that promotes the practice of tests specification since the very beginning of projects, and its integration with the requirements specification itself. It is a model-driven approach that contributes to maintain the requirements and tests alignment, namely between requirements, test cases, and low-level automated test scripts. To show the applicability of the approach, two complementary languages are adopted: the ITLingo RSL that is particularly designed to support both requirements and tests specification; and the Robot language, which is a low-level keyword-based language for the specification of test scripts. The approach includes model-to-model transformation techniques, such as test cases into test scripts transformations. In addition, these test scripts are executed by the Robot test automation framework.

Mobile applications are increasingly present in our daily lives. Being increasingly dependent on apps, we all want to make sure apps work as expected. One way to increase confidence and quality of software is through testing. However, the existing approaches and tools still do not provide sufficient solutions for testing mobile apps with features different from the ones found in desktop or web applications. In particular, there are guidelines that mobile developers should follow and that may be tested automatically but, as far as we know, there are no tools that are able do it. The iMPAcT tool combines exploration, reverse engineering and testing to check if mobile apps follow best practices to implement specific behavior called UI Patterns. Examples of UI Patterns within this catalog are: orientation, background-foreground, side drawer, tab-scroll, among others. For each of these behaviors (UI Patterns), the iMPAcT tool has a corresponding Test Pattern that checks if the UI Pattern implementation follows the guidelines. This paper presents an extension to iMPAcT tool. It enables to test if Android apps work properly after receiving an incoming call, i.e., if the state of the screen after the call is the same as before getting the call. It formalizes the problem, describes the overall approach, describes the architecture of the tool and reports an experiment performed over 61 public mobile apps.

This paper presents iMPAcT tool that tests recurring common behavior on Android mobile applications. The process followed combines exploration, reverse engineering and testing to automatically test Android mobile applications. The tool explores automatically the App by firing UI events. After each event fired, the tool checks if there are UI patterns present using a reverse engineering process. If a UI pattern is present, the tool runs the corresponding testing strategy (Test Pattern). During reverse engineering the tool uses a catalog of UI Patterns which describes recurring behavior (UI Patterns) to test and the corresponding test strategies (Test Patterns). This catalog may be extended in the future as needed (e.g., to deal with new interaction trends). This paper describes the implementation details of the iMPAcT tool, the catalog of patterns used, the outputs produced by the tool and the results of experiments performed in order to evaluate the overall testing approach. These results show that the overall testing approach is capable of finding failures on existing Android mobile applications.

Summary This paper investigates the failures exposed in mobile apps by the mobile‐specific event of changing the screen orientation. We focus on GUI failures resulting in unexpected GUI states that should be avoided to improve the apps quality and to ensure better user experience. We propose a classification framework that distinguishes 3 main classes of GUI failures due to orientation changes and exploit it in 2 studies that investigate the impact of such failures in Android apps. The studies involved both open‐source and apps from Google Play that were specifically tested exposing them to orientation change events. The results showed that more than 88% of these apps were affected by GUI failures, some classes of GUI failures were more common than others, and some GUI objects were more frequently involved. The app source code analysis allowed us to identify 6 classes of common faults causing specific GUI failures.

Summary Software systems with a graphical user interface (GUI) front end are typically designed using user interface (UI) Patterns, which describe generic solutions (with multiple possible implementations) for recurrent GUI design problems. However, existing testing techniques do not take advantage of this fact to test GUIs more efficiently. In this paper, we present a new pattern‐based GUI testing (PBGT) approach that formalizes the notion of UI Test Patterns, which are generic test strategies to test UI patterns over their different implementations. The PBGT approach is evaluated via 2 case studies. The first study involves 2 fielded Web application subjects; findings show that PBGT is both practical and useful, as testing teams were able to find real bugs in a reasonable time interval. The second study allows deeper analysis by studying software subjects seeded with artificial faults; the findings show that PBGT is more effective than a manual model‐based test case generation approach.

This paper presents the iMPAcT tool that tests recurring behaviour, i.e., UI Patterns, on mobile applications. This tool is implemented in Java and makes use of Android's APIs UI Automator and UiAutomation. The tool automatically explores a mobile application in order to automatically identify and test UI Patterns. Each UI Pattern has a test strategy, Test Patterns, associated, which is applied when an UI Pattern is found. The approach works on top of a catalogue of UI Patterns, which determines which UI Patterns are to be tested, and what should their correct behaviour be, and may be used for any application.

Modeling and building software systems for a given specific domain is a complex task. Domain-Specific Languages (DSLs) have been increasingly gaining attention because they are developed to cope with particularities of specific domains. However, DSL development consists in a set of tasks to be performed and some can be error-prone. Identifying the correct set of elements within a DSL and their constraints can be very demanding. Alloy is a popular lightweight intuitive formal language with a simple notation that is easy to read and write. When models of a DSL are specified using Alloy, it becomes possible to generate instances that should represent valid models. So, this paper presents a generic innovative methodology using Alloy in DSL engineering, in order to find and tune language constraints in a systematic way. It also presents an empirical study illustrating the applicability of the proposed methodology.

Today's software feature user interface (UI) patterns. Those patterns describe generic solutions for common recurrent problems. However, to the best of our knowledge, there is no specific testing methodology that is particularly suited for testing those patterns providing generic testing solutions that can be reused after minor configurations in order to test slightly different implementations. Pattern-Based Graphical User Interface Testing (PBGT) is a recent methodology that aims at systematizing and automating the GUI testing process, by sampling the input space using "UI Test Patterns" that express generic solutions to test common recurrent GUI's behaviour. This paper describes the development process of PARADIGM, a domain specific language (DSL) to be used in the context of PBGT and empirically evaluates PARADIGM to assess its diminished modeling efforts, usefulness, graphical power, and acceptability.

Pattern Based GUI Testing (PBGT) is a new methodology that aims at systematizing and automating the GUI testing process. It is supported by a Tool (PBGT Tool) which provides an integrated modeling and testing environment that supports the crafting of test models based on UI Test Patterns, using a GUI modeling DSL called PARADIGM. The tool is freely available as an Eclipse plugin, developed on top of the Eclipse Modeling Framework. This paper presents PBGT Tool, which has been successfully used in several projects, and more recently at industry level.

User Interface (UI) patterns are used extensively in the design of today's software. UI patterns embody commonly recurring solutions that solve common GUI design problems, such as “login,” “file-open,” and “search.” Yet, testing of GUIs for functional correctness has largely ignored UI patterns. This paper formalizes the notion of a Pattern-Based Graphical User Interface (GUI) Testing method (PBGT) for systematizing and automating the GUI testing process. The space of all possible interactions with a GUI is typically very large. PBGT presents a new methodology to sample the input space using “UI Test Patterns,” that embody commonly recurring solutions to test GUIs. Our empirical studies show that the PBGT methodology is effective in revealing faults in fielded GUIs.