Abstract In the digital age, the omnipresence of software drives the mass production fueled by artificial intelligence. This dependence requires rigorous methods to assess the quality of the software. Inspired by Proficiency Tests (PT) from other fields, this work proposes an innovative process to evaluate the competence of software testing teams and laboratories. Based on Action Design Research(ADR) and Business Process Management(BPM), the developed solution includes software that automates the process, provides a knowledge base, and generates performance indicators. Furthermore, we explore the use of Large Language Models (LLMs) to extract valuable insights from PT results. Quantitative experiments demonstrated the effectiveness of the proposed process, while a qualitative study validated the solution, demonstrating its feasibility and potential to transform the quality assessment of software.

The increasing use of mobile apps in daily life involves managing and sharing sensitive user information. New vulnerabilities are frequently reported in bug tracking systems, highlighting the need for effective security testing processes for these applications. This study introduces a GUI-based Metamorphic Testing technique designed to detect five common real-world vulnerabilities related to username and password authentication methods in Android applications, as identified by OWASP. We developed five Metamorphic Relationships to test for these vulnerabilities and implemented a Metamorphic Vulnerability Testing Environment to automate the technique. This environment facilitates the generation of Source test case and the automatic creation and execution of Follow-up test case . The technique was applied to 163 real-world Android applications, uncovering 159 vulnerabilities. Out of these, 108 apps exhibited at least one vulnerability. The vulnerabilities were validated through expert analysis conducted by three security professionals, who confirmed the issues by interacting directly with the app’s graphical user interfaces (GUIs). Additionally, to assess the practical relevance of our approach, we engaged with 37 companies whose applications were identified as vulnerable. Nine companies confirmed the vulnerabilities, and 26 updated their apps to address the reported issues. Our findings also indicate a weak inverse correlation between user-perceived quality and vulnerabilities; even highly rated apps can harbor significant security flaws.

Summary Despite the rapid growth and popularization of virtual reality (VR) applications, which have enabled new concepts for handling and solving existing problems through VR in various domains, practices related to software engineering have not kept up with this growth. Recent studies indicate that one of the topics that is still little explored in this area is software testing, as VR applications can be built for practically any type of purpose, making it difficult to generalize knowledge to be applied. In this paper, we present an approach that combines metamorphic testing, agent‐based testing and machine learning to test VR applications, focusing on finding collision and camera‐related faults. Our approach proposes the use of metamorphic relations to detect faults in collision and camera components in VR applications, as well as the use of intelligent agents for the automatic generation of test data. To evaluate the proposed approach, we conducted an experimental study on four VR applications, and the results showed an of the solution ranging from 93% to 69%, depending on the complexity of the application tested. We also discussed the feasibility of extending the approach to identify other types of faults in VR applications. In conclusion, we discussed important trends and opportunities that can benefit both academics and practitioners.

Abstract As software systems have increased in complexity, manual testing has become harder or even infeasible. In addition, each test phase and application domain may have its idiosyncrasies in relation to testing automation. Techniques and tools to automate test oracles in domains such as graphical user interfaces are available; nevertheless, they are scarce in the virtual reality (VR) realm. We present an approach to automate software testing in VR-based systems with haptic interfaces—interfaces that allow bidirectional communication during human–computer interaction, capturing movements and providing touch feedback. It deals with the complexity and characteristics of haptic interfaces to apply the record and playback technique. Our approach also provides inference rules to identify possible faulty modules of the system under testing. A case study was performed with three systems: a system with primitive virtual objects, a dental anesthesia simulator and a game. Faulty versions of the systems were created by seeding faults manually and by using mutation operators. The results showed that 100% of the manually seeded faults and 93% of mutants were detected. Moreover, the inference rules helped identify the faulty modules of the systems, suggesting that the approach improves the test activity in VR-based systems with haptic interfaces.

Mutation testing is well known as one of the most effective approaches to create test cases, which can detect software faults. However, its drawback is the low scalability — if no special attention is given to improve efficiency — that directly affects its application in practice. This paper shows a replication study focused on emphasizing evidence in which the use of distributed processing structures can improve mutation testing. For this purpose, an architecture that enables mutation testing concurrent execution was designed. Five load balancing algorithms responsible for controlling the distribution and execution of data while carrying out mutation testing were evaluated. Experiments were conducted in order to evaluate the scalability and performance of the architecture considering homogeneous and heterogeneous setups. A time reduction of 50% was observed when executing mutants in parallel in relation to the conventional sequential application of mutation testing. The performance gain was above 95% when there was a higher number of nodes in the distributed architecture.

This Research Full Paper reinforces that Software Testing can be a helpful practice to students while working on programming assignments. Considering Software Testing as a process, the testing activity is composed by a sequence of steps. When students write and submit their own test cases, they are responsible for the test design, automation, execution, and evaluation. Otherwise, instructors can provide ready-made test suites, needing only to execute it and evaluate it. In this scenario, we conducted an experimental study to investigate how the programming performance of students from the Computer Science area is affected when Software Testing is integrated with programming. We proposed three different approaches: (i) ad hoc programming; (ii) programming and testing by writing the test suite; and (iii) programming and testing with a readymade test suite. We assessed students' programs in terms of correctness, measured by the pass rate of the reference test suite. Results indicate that students had a lower performance with ad hoc programming in comparison with both approaches involving Software Testing. On the other hand, using ready-made test cases raised better results than when students had to write their own test cases. We also assessed students' attitudes towards testing by means of a survey.

Mutation Testing has been proved to be effective in detecting faults. However, it suffers from the large number of mutants generated and analyzed when we apply the operators to create mutants. To alleviate this problem, this paper proposes an approach to find a reduced set of mutation operators that reduces the generated mutants and maintains the test effectiveness. Our approach adapts the semantic characterization of faults in order to find this set of mutation operators that will reduce the cost of Mutation Testing. The experimental results indicate that our approach is able to find a set of operators that reduce the mutation cost while maintaining the test effectiveness.

Summary Software testing is regarded as an important method for fault revealing. Despite this advantage, it has been poorly used within the scope of virtual reality (VR) applications because they are highly complex and have peculiar features. Most testing performed of this VR applications are usability, which is conducted manually and only at final of the development process. Although some works try to propose criteria for this domain, there are no approaches that automatize the generation of test data from requirements specification in the VR domain. This paper proposes an approach called virtual reality—requirements specification and testing (VR‐ReST) to assist the requirements specification through a semiformal language and uses structural test criteria to generate test requirements and test data automatically for VR applications using scene graph concepts. The paper also examines the empirical results concerning the cost‐effectiveness of the approach for three different VR applications through two experiments. Mutation testing was used to evaluate effectiveness. We found that the approach achieved a high mutation score outperforming random testing, by 20%, on average. Our results also demonstrate that the approach is promising since it assists in writing and validating the requirements, as well as in reducing the risks of requirement specification by adopting a semiformal language.

Systems-of-Systems (SoS) refer to large, complex, and software-intensive systems, resulted from the interoperability among heterogeneous, independent constituent systems. The main purpose of SoS is to perform tasks that could not be achieved by these constituents separately; besides, unique SoS characteristics impose new challenges to their development processes. We conducted a Systematic Mapping to identify initiatives, trends, and challenges in the SoS development and, as a result, 32 initiatives were identified. We also evaluated these initiatives with regard to the adherence to the technical processes of IEEE/ISO/IEC 15288:2015 and to the software implementation essential processes of IEEE/ISO/IEC 12207:2008. In general, these initiatives emphasize artifacts in the early stages of the development processes, addressing mainly requirements and architecture. Moreover, 17 of them emphasize the testing activity. We also summarized the SoS characteristics addressed in the studies and discuss the main trends and challenges.

Various forms of selective mutation testing have long been accepted as valid approximations to full mutation testing. This paper presents counterevidence to traditional selective mutation. The recent development of dominator mutants and minimal mutation analysis lets us analyze selective mutation without the noise introduced by the redundancy inherent in traditional mutation. We then exhaustively evaluate all small sets of mutation operators for the Proteum mutation system and determine dominator mutation scores and required work for each of these sets on an empirical test bed. The results show that all possible selective mutation approaches have poor dominator mutation scores on at least some of these programs. This suggests that to achieve high performance with respect to full mutation analysis, selective approaches will have to become more sophisticated, possibly by choosing mutants based on the specifics of the artifact under test, that is, specialized selective mutation.

This paper describes a novel application of Content-Based Image Retrieval (CBIR) to search a medical database consisting of 3D models for diagnosis purposes. The 3D models, which are generated using Magnetic Resonance Imaging and include depth information, are used to search for similarity a database of 3D annotated medical cases using their pairwise feature similarity. The 3D models consist of both local and global feature descriptors that consider the surface of the 3D model and the overall geometry of the medical artifact. The models are then matched using spectral clustering that embeds the Euclidean distance for affinity and partitions the models into two groups, Congestive Heart Failure (CHF) and non-CHF. This suffices to demarcate using pairwise similarity the existence of CHF for the left ventricle. Experimental results using thirty 3D models show the utility of the new 3D method compared to existing methods. In particular, the novel method yields 83% overall accuracy.

Content-Based Image Retrieval (CBIR) systems constitute an innovative approach to store, to compare and to query images in a database. Visual aspects such as color, texture or shape are used to perform such operations. Recently, CBIR concepts were applied to build testing oracles for image processing programs, where test verdicts (approval/disapproval) are based on similarity measures between images produced by the program and reference images. However, the results of a CBIR system may vary depending on the components employed in the system (feature extractors and similarity functions), and few studies assessing this influence have been found in the literature. Our aim is to present an empirical analysis of ten similarity functions in CBIR systems within the context of software testing with graphic outputs. A case study with images obtained from a computer-aided diagnosis system in mammography indicated some variability among image test verdicts (approval/disapproval) according to the similarity function choice. The case study also indicates the existence of some clusters of similarity functions with high correlation coefficients.

Mutation analysis generates tests that distinguish variations, or mutants, of an artifact from the original. Mutation analysis is widely considered to be a powerful approach to testing, and hence is often used to evaluate other test criteria in terms of mutation score, which is the fraction of mutants that are killed by a test set. But mutation analysis is also known to provide large numbers of redundant mutants, and these mutants can inflate the mutation score. While mutation approaches broadly characterized as reduced mutation try to eliminate redundant mutants, the literature lacks a theoretical result that articulates just how many mutants are needed in any given situation. Hence, there is, at present, no way to characterize the contribution of, for example, a particular approach to reduced mutation with respect to any theoretical minimal set of mutants. This paper's contribution is to provide such a theoretical foundation for mutant set minimization. The central theoretical result of the paper shows how to minimize efficiently mutant sets with respect to a set of test cases. We evaluate our method with a widely-used benchmark.

Mutation analysis modifies a program by applying syntactic rules, called mutation operators, systematically to create many versions of the program (mutants) that differ in small ways. Testers then design tests to cause the mutants to behave differently from the original program. Mutation testing is widely considered to result in very effective tests, however, it is also quite costly. Cost comes from the many mutants that are created, the number of tests that are needed to kill the mutants, and the difficulty of deciding whether mutants behave equivalently to the original program. One-op mutation theorizes that cost can be reduced by using a single, very powerful, mutation operator that leads to tests that are almost as effective as if all operators are used. Previous research proposed the statement deletion operator (SDL) and found promising results. This paper investigates the use of SDL-mutation in a new context, the language C, and poses additional empirical questions, including whether other operators can be used. We carried out a controlled experiment in which cost and effectiveness of each individual C mutation operator were collected for 39 different subject programs. Experimental data are used to define a cost-effectiveness metric to choose the best single operator for one-op mutation.

As a test criterion, mutation analysis is known for yielding very effective tests. It is also known for creating many test requirements, each of which is represented by a "mutant" that must be "killed." In recent years, researchers have found that these test requirements have a lot of duplication, in that many test requirements yield the same tests. Put another way, hundreds of mutants can usually be killed by only a few dozen tests. If we could reduce this duplication without reducing mutation's effectiveness, mutation testing could become more cost-effective. One avenue of this research has been to use only one type of mutant, the statement deletion mutation operator. Researchers have found that statement deletion mutation has relatively few mutants, but yields tests that are almost as effective as using all mutants, with the significant benefit that fewer equivalent mutants are generated. This paper extends this idea by asking a simple question: if deleting statements is a cost-effective way to design tests, will deleting other program elements also be effective? This paper presents results from mutation operators that delete variables, operators, and constants, finding that indeed, this is an efficient and effective approach.

Time and resource constraints should be taken into account in software testing activities, and thus optimizing the test suite is fundamental in the development process. In this context, the test case selection aims to eliminate redundant or unnecessary test data, which is crucial for the definition of test strategies. This paper presents a systematic review on the test case selection conducted through a selection of 449 articles published in leading journals and conferences in Computer Science. We addressed the state-of-art by collecting and comparing existing evidence on the methods used in the different software domains and the methods used to evaluate the test case selection. Our study identified 32 papers that met the research objectives, which featured 18 different selection methods and were evaluated through 71 case studies. The most commonly reported methods are adaptive random testing, genetic algorithms and greedy algorithm. Most approaches rely on heuristics, such as diversity of test cases and code or model coverage. This paper also discusses the key concepts and approaches, areas of application and evaluation metrics inherent to the methods of test case selection available in the literature.

This paper presents a partially-automated solution for the definition of test oracles for validating Simulink models. Since these models are widely used for the design of embedded systems, their thorough validation is key. Errors and misunderstandings in these models may preclude the correctness and reliability of the final system: a manual, human-intensive validation activity is not enough, and automated solutions are fundamental to improve the current state of the art. The paper also presents Apolom, a prototype oracle generator, and the results of two experiments.

Fault diagnosis is the process of analyzing programs with the aim of identifying the code fragments that are faulty. It has been identified as one of the most expensive and time consuming tasks of software development. Even worst, this activity is usually accomplished based on manual analysis. To this end, automatic or semi-automatic fault diagnosis approaches are useful in assisting software developers. Hence, they can play an essential role in decreasing the overall development cost. This paper presents Proteum/FL, a mutation analysis tool for diagnosing previously detected faults. Given an ANSI-C program and a set of test cases, Proteum/FL returns a list of program statements ranked according to their likelihood of being faulty. The tool differs from the rest of the mutation analysis and fault diagnosis tools by employing mutation analysis as a means of diagnosing program faults. It therefore demonstrates the effective use of mutation in supporting both testing and debugging activities.

SUMMARY Automation of testing is an essential requirement to render it viable for software development. Although there are several testing techniques and criteria in many different domains, developing methods to test programs with complex outputs remains an unsolved challenge. This setting includes programs with graphical output, which produce images or interface windows. One possible approach towards automating the testing activity is the use of automatic oracles in which a reference image, taken as correct, can be used to establish a correctness measure in the tested program execution. A method that uses concepts of content‐based image retrieval to facilitate oracle automation in the domain of programs with graphics output is presented. Two case studies, one using a computer‐aided diagnostic system and one using a Web application, are presented, including some reflections and discussions that demonstrate the feasibility of the proposed approach. Copyright © 2011 John Wiley & Sons, Ltd.

High-level language virtual machines (HLL VMs) are now widely used to implement high-level programming languages. To a certain extent, their widespread adoption is due to the software engineering benefits provided by these managed execution environments, for example, garbage collection (GC) and cross-platform portability. Although HLL VMs are widely used, most research has concentrated on high-end optimizations such as dynamic compilation and advanced GC techniques. Few efforts have focused on introducing features that automate or facilitate certain software engineering activities, including software testing. This paper suggests that HLL VMs provide a reasonable basis for building an integrated software testing environment. As a proof-of-concept, we have augmented a Java virtual machine (JVM) to support weak mutation analysis. Our mutation-aware HLL VM capitalizes on the relationship between a program execution and the underlying managed execution environment, thereby speeding up the execution of the program under test and its associated mutants. To provide some evidence of the performance of our implementation, we conducted an experiment to compare the efficiency of our VM-based implementation with a strong mutation testing tool (muJava). Experimental results show that the VM-based implementation achieves speedups of as much as 89% in some cases.

Web services are an emerging Service-Oriented Architecture technology to integrate applications using open standards based on XML. Software Engineering tools integration is a promising area since companies adopt different software processes and need different tools on each activity. Software engineers could take advantage of software engineering tools available as web services and create their own workflow for integrating the required tools. In this paper, we propose the development of testing tools designed as web services and discuss the pros and cons of this idea. We developed a web service for structural testing of Java programs called JaBUTiService, which is based on the stand-alone tool JaBUTi. We also present an usage example of this service with the support of a desktop front-end and pre prepared scripts. A set of 62 classes of the library Apache-Commons-BeanUtils was used for this test and the results are discussed.

Abstract This paper describes intra‐method control‐flow and data‐flow testing criteria for the Java bytecode language. Six testing criteria are considered for the generation of testing requirements: four control‐flow and two data‐flow based. The main reason to work at a lower level is that, even when there is no source code, structural testing requirements can still be derived and used to assess the quality of a given test set. It can be used, for instance, to perform structural testing on third‐party Java components. In addition, the bytecode can be seen as an intermediate language, so the analysis performed at this level can be mapped back to the original high‐level language that generated the bytecode. To support the application of the testing criteria, we have implemented a tool named JaBUTi (Java Bytecode Understanding and Testing). JaBUTi is used to illustrate the application of the ideas developed in this paper. Copyright © 2006 John Wiley & Sons, Ltd.

Mutation testing (Mutation Analysis), although powerful in revealing faults, is considered a computationally expensive criterion, due to the high number of mutants created and the effort to determine the equivalent mutants. Using mutation-based alternative testing criteria it is possible to reduce the number of mutants but it is still necessary to determine the equivalent ones. In this paper the Bayesian Learning(one of the Artificial Intelligence techniques used in machine learning) is investigated to define the Bayesian Learning-Based Equivalent Detection Technique (BaLBEDeT), which provides guidelines to help the tester to analyze the live mutants in order to determine the equivalent ones.

Abstract Mutation testing, originally proposed for unit testing, has been extended to integration testing with the proposition of the Interface Mutation criterion. This paper presents the results of an experiment using two mutation‐based testing criteria for unit and integration testing phases: the Mutation Analysis and the Interface Mutation adequacy criteria, respectively. The aim is to investigate how they can be used in a complementary way during the testing activity, establishing an incremental testing strategy comprising the unit and integration testing phases and guidelines on how to obtain a high mutation score with respect to mutation testing with a low cost, in terms of the number of mutants generated. Copyright © 2001 John Wiley & Sons, Ltd.

The need for test adequacy criteria is widely recognized. Several criteria have been proposed for the assessment of adequacy of tests at the unit level. However, there remains a lack of criteria for the assessment of the adequacy of tests generated during integration testing. We present a mutation based interprocedural criterion, named Interface Mutation (IM), suitable for use during integration testing. A case study to evaluate the proposed criterion is reported. In the study, the UNIX sort utility was seeded with errors and Interface Mutation evaluated by measuring the cost of its application and its error revealing effectiveness. Alternative IM criteria using different sets of Interface Mutation operators were also evaluated. While comparing the error revealing effectiveness of these Interface Mutation-based test sets with same size randomly generated test sets, we observed that in most cases Interface Mutation based test sets are superior. The results suggest that Interface Mutation offers a viable test adequacy criteria for use at the integration level.

A criterion for assessing the adequacy of test sets during integration testing is proposed. The criterion is based on a testing technique named Interface Mutation. The technique itself is designed to be scalable with the size of the software under test; the size being measured in the number of subsystems integrated. Using Interface Mutation it is possible to assess the adequacy of tests incrementally while integrating various subsystems. Also reported are results from a pilot experiment conducted to study the cost and error defection effectiveness of Interface Mutation.

Proposes the application of the mutation analysis criterion in the context of specifications based on finite state machines. The main concepts of finite state machines and mutation analysis are briefly introduced. An experiment is reported which manually applies mutation analysis to a finite state machine modeling a Class 0 ISO transport protocol specification, using two test-sequence generator criteria-the W method and the TT (transition tours) method. The results obtained are presented, and evidences are given that the use of mutation analysis is effective in this context. Finally, the lines of evolution of the work presented in this paper are briefly discussed.>