ABSTRACT System‐level testing of avionics software systems requires compliance with different international safety standards such as DO‐178C. An important consideration of the avionics industry is automated test data generation according to the criteria suggested by safety standards. One of the recommended criteria by DO‐178C is the modified condition/decision coverage (MC/DC) criterion. Current model‐based test data generation approaches use constraints written in Object Constraint Language (OCL) and apply search techniques to generate test data. These approaches either do not support MC/DC criterion or suffer from performance issues while generating test data for large‐scale avionics systems. In this paper, we propose an effective way to automate MC/DC test data generation during model‐based testing. We develop a strategy that utilizes case‐based reasoning (CBR) and range reduction heuristics designed to solve MC/DC‐tailored OCL constraints. We performed an empirical study to compare our proposed strategy for MC/DC test data generation using CBR, range reduction, both CBR and range reduction, with an original search algorithm, and random search. We also empirically compared our strategy with existing constraint‐solving approaches. The results show that both CBR and range reduction for MC/DC test data generation outperform the baseline approach. Moreover, the combination of both CBR and range reduction for MC/DC test data generation is an effective approach compared to existing constraint solvers.

Summary The unmanned aircraft system (UAS) is rapidly gaining popularity in civil and military domains. A UAS consists of an application software that is responsible for defining a UAS mission and its expected behavior. A UAS during its mission experiences changes (or interruptions ) that require the unmanned aerial vehicle (UAV) in a UAS to self‐adapt, that is, to adjust both its behavior and position in real‐time, particularly for maintaining formation in the case of a UAS swarm. This adaptation is critical as the UAS operates in an open environment, interacting with humans, buildings, and neighboring UAVs. To verify if a UAS correctly makes an adaptation, it is important to test it. The current industrial practice for testing the self‐adaptive behaviors in UAS is to carry out testing activities manually. This is particularly true for existing UAS rather than newly developed ones. Manual testing is time‐consuming and allows the execution of a limited set of test cases. To address this problem, we propose an automated model‐based approach to test the self‐adaptive behavior of UAS application software. The work is conducted in collaboration with an industrial partner and demonstrated through a case study of UAS swarm formation flight application software. Further, the approach is verified on various self‐adaptive behaviors for three open‐source autopilots (i.e., Ardu‐Copter, Ardu‐Plane, and Quad‐Plane). Using the proposed model‐based testing approach we are able to test sixty unique self‐adaptive behaviors. The testing results show that around 80% of the behavior adaptations are correctly executed by UAS application software.

Abstract Model transformation testing has become crucial as model‐driven engineering has raised the abstraction level for developing software systems. Transformation is written to transform models from one level of abstraction to another, for example, model to model or model to code. A major challenge in testing the transformation is the creation of test models, such that (i) they conform to the source meta‐model (i.e., multiplicities and Object Constraint Language [OCL] constraints on meta‐model) and (ii) they provide coverage of the complete transformation (solving branch conditions for traversing all paths). Manual creation of test models requires a lot of time and effort. Still, the validity of the developed test models cannot be ensured. This paper aims to solve the above challenges using an automated search‐based strategy. The proposed approach is two‐stepped. First, valid test models are generated by solving source meta‐model constraints. Second, the generated models are evolved for achieving the structural coverage of the transformation by solving the branch conditions. A toolset model transformation testing environment (MOTTER) is developed to automate the search‐based solution. The proposed work is empirically evaluated on two case studies using four search algorithms. The result reflects that it successfully generates valid test models for achieving desired structural coverage with high performance on both the case studies.

Abstract Data science (DS) applications not only suffer from traditional software faults but may also suffer from data‐specific and model‐related faults. Fault models play an important role in evaluating and designing tests for testing DS applications. The existing fault models do not consider DS specific faults. In this study, we built a fault model DS applications. We investigate the faults by using diverse approaches: (i) a multi‐vocal literature survey of published literature, (ii) semi‐structured interviews of industry experts. The Multi‐vocal study allows us to synthesize the existing knowledge from researchers and practitioners. Qualitative data from semi‐structured interviews provide us with insights into the nature of faults encountered by practitioners. We combine the results of (i) and (ii) to derive a detailed fault model. The developed fault model is further validated through a quantitative survey of industry practitioners, and the respondents were asked to identify the faults from our proposed fault model that they have experienced and classify those faults based on their severity as perceived by practitioners and its frequency. The results show that practitioners consider prediction bias and model decay as the most severe faults while data sampling and splitting faults along with feature engineering faults are the most frequent.

Avionics are highly critical systems that require extensive testing governed by international safety standards. Cockpit Display Systems (CDS) are an essential component of modern aircraft cockpits and display information from the user application using various widgets. A significant step in the testing of avionics is to evaluate whether these CDS are displaying the correct information. A common industrial practice is to manually test the information on these CDS by taking the aircraft into different scenarios during the simulation. Given the large number of scenarios to test, manual testing of such behavior is a laborious activity. In the previous work, we proposed a model-based testing approach for the CDS. In this paper, we present a CDST (CDS Testing) toolkit that automates the process of testing CDS. We discuss the workflow and architecture of the tool and also demonstrates the tool on an industrial case study. The results show that the tool is able to generate, execute, and evaluate the test cases and identify 3 bugs in the case study.

Abstract Mobile devices have limited resources, including memory and processing speed. The performance of mobile applications is an important concern. There are a large number of mobile platforms available with varying operating systems and hardware. Native applications are usually developed and maintained separately for these platforms. The overall performance of native applications may significantly vary across platforms. The current industrial practice is to manually test the performance for each variant, which is not a scalable or efficient approach. We tackled the problem of generating native application variants in our previous work. This paper proposes an automated model‐based approach for performance test generation for native application variants at unit level. We propose a performance profile that allows modeling of domain‐specific performance parameters on UML models, which are used for automated performance test generation for each native variant. The results of applying the approach on two real‐world applications show that the approach evaluates the performance of application variants for two different versions of Android successfully and have potential to reduce the effort and time. A questionnaire‐based study is conducted to evaluate the usefulness of the approach.

Avionics are highly critical systems that require extensive testing governed by international safety standards. Cockpit Display Systems (CDS) are an essential component of modern aircraft cockpits and display information from the user application (UA) using various widgets. A significant step in the testing of avionics is to evaluate whether these CDS are displaying the correct information. A common industrial practice is to manually test the information on these CDS by taking the aircraft into different scenarios during the simulation. Such testing is required very frequently and at various changes in the avionics. Given the large number of scenarios to test, manual testing of such behavior is a laborious activity. In this paper, we propose a model-based strategy for automated testing of the information displayed on CDS. Our testing approach focuses on evaluating that the information from the user applications is being displayed correctly on the CDS. For this purpose, we develop a profile for capturing the details of different widgets of the display screens using models. The profile is based on the ARINC 661 standard for Cockpit Display Systems. The expected behavior of the CDS visible on the screens of the aircraft is captured using constraints written in Object Constraint Language. We apply our approach on an industrial case study of a Primary Flight Display (PFD) developed for an aircraft. Our results showed that the proposed approach is able to automatically identify faults in the simulation of PFD. Based on the results, it is concluded that the proposed approach is useful in finding display faults on avionics CDS.

In safety critical domains, system test cases are often derived from functional requirements in natural language (NL) and traceability between requirements and their corresponding test cases is usually mandatory. The definition of test cases is therefore time-consuming and error prone, especially so given the quickly rising complexity of embedded systems in many critical domains. Though considerable research has been devoted to automatic generation of system test cases from NL requirements, most of the proposed approaches re- quire significant manual intervention or additional, complex behavioral modelling. This significantly hinders their applicability in practice. In this paper, we propose Use Case Modelling for System Tests Generation (UMTG), an approach that automatically generates executable system test cases from use case spec- ifications and a domain model, the latter including a class diagram and constraints. Our rationale and motivation are that, in many environments, including that of our industry partner in the reported case study, both use case specifica- tions and domain modelling are common and accepted prac- tice, whereas behavioural modelling is considered a difficult and expensive exercise if it is to be complete and precise. In order to extract behavioral information from use cases and enable test automation, UMTG employs Natural Language Processing (NLP), a restricted form of use case specifica- tions, and constraint solving.

Game development has recently gained a lot of momentum and is now a major software development industry. Platform games are being revived with both their 2D and 3D versions being developed. A major challenge faced by the industry is a lack of automated system-level approaches for game testing. Currently in most game development organizations, games are tested manually or using semi-automated techniques. Such testing techniques do not scale to the industry requirements where more systematic and repeatable approaches are required. In this paper we propose a model-based testing approach for automated black box functional testing of platform games. The paper provides a detailed modeling methodology to support automated system-level game testing. As part of the methodology, we provide guidelines for modeling the platform games for testing using our proposed game test modeling profile. We use domain modeling for representing the game structure and UML state machines for behavioral modeling. We present the details related to automated test case generation, execution, and oracle generation. We demonstrate our model-based testing approach by applying it on two cases studies, a widely referenced and open source implementation of Mario brothers game and an industrial case study of an endless runner game. The proposed approach was able to identify major faults in the open source game implementation. Our results showed that the proposed approach is practical and can be applied successfully on industrial games.

We present UMTG, a toolset for automatically generating executable and traceable system test cases from use case specifications. UMTG employs Natural Language Processing (NLP), a restricted form of use case specifications, and constraint solving. Use cases are expected to follow a template with restriction rules that reduce imprecision and enable NLP. NLP is used to capture the control flow implicitly described in use case specifications. Finally, to generate test input, constraint solving is applied to OCL constraints referring to the domain model of the system. UMTG is integrated with two tools that are widely adopted in industry, IBM Doors and Rhapsody. UMTG has been successfully evaluated on an industrial case study.

Mobile application development has emerged as one of the most focused areas in software industry due to exponential growth of mobile users and applications. The focus of the current mobile industry is on direct implementation, rather than analysis and design. Model-driven engineering practices are ignored, which results in low reusability and lack of portability along with other challenges. In addition, mobile applications also have to support multiple platforms, as an application written for one platform (e.g., Android) cannot run on another platform (e.g., Windows Phone). Each of the platforms in turn suffers from fragmentation. This results in multiple versions of an application that need to be simultaneously maintained. This is a huge burden on the development team, both in terms of cost and effort, due to various platforms that an application supports. This paper proposes a model-driven approach to generate mobile applications for multiple platforms. We carefully identify a subset of UML and propose a modeling methodology for this purpose. We use real use-cases for requirement gathering, class diagram for structural modeling, while state machine are used for behavioral modeling. We focus on modeling business logic layer for the mobile application. We also propose a UML profile for modeling mobile domain specific concepts. Our approach uses Action Language for Foundational Subset of UML (ALF) to specify actions in the state machines. Our approach allows the modeler to automatically generate business logic code for multiple platforms. The approach is supported by an automated tool MAG (Mobile Application Generator). We apply our model-driven approach on an industrial case study 'Scramble' app developed by 'Next In' to demonstrate its viability.

The Object Constraint Language (OCL) is a standard language for specifying constraints on Unified Modeling Language (UML) models. The specified constraints can be used for various purposes including verification, and model-based testing (e.g., test data generation). Efficiently solving OCL constraints is one of the key requirements for the practical use of OCL. In this paper, we propose an improvement in existing heuristics to solve OCL constraints using search algorithms. We evaluate our improved heuristics using two empirical studies with three search algorithms: Alternating Variable Method (AVM), (1+1) Evolutionary Algorithm (EA), and a Genetic Algorithm (GA). We also used Random Search (RS) as a comparison baseline. The first empirical study was conducted using carefully designed artificial problems (constraints) to assess each individual heuristics. The second empirical study is based on an industrial case study provided by Cisco about model-based testing of Video Conferencing Systems. The results of both empirical evaluations reveal that the effectiveness of the search algorithms, measured in terms of time to solve the OCL constraints to generate data, is significantly improved when using the novel heuristics presented in this paper. In particular, our experiments show that (1+1) EA with the novel heuristics has the highest success rate among all the analyzed algorithms, as it requires the least number of iterations to solve constraints.

UML state machines are commonly used to model the state-based behavior of communication and control systems to support various activities such as test cases and code generation. Standard UML state machines are well suited to model functional behavior, however extra-functional behavior such as robustness and security can also be directly modeled on them, but this often results in cluttered models since extra-functional behaviors are often crosscutting. Such modeling crosscutting behavior directly on UML state machines is a common practice. Aspect-Oriented Modeling (AOM) allows systematically modeling of crosscutting behavior and has shown to provide a scalable solution in the recent years. However, due to lack of familiarity of AOM in both academic and industry, extra-functional behavior is often modeled directly on UML state machines and as a result those UML state machines are difficult to read and maintain. To improve the readability of already developed UML state machines and ease maintenance, we propose a set of heuristics, derived from two industrial cases studies, implemented in a tool to automatically identify commonly observed crosscutting behaviors in UML state machines and refactor them as Aspect State Machines. Such refactoring makes the state machines easier to maintain and comprehend. We present the results of applying our proposed heuristics to the existing UML state machines of two industrial case studies developed for model-based testing.

Model-based testing (MBT) aims at automated, scalable, and systematic testing solutions for complex industrial software systems. To increase chances of adoption in industrial contexts, software systems can be modeled using well-established standards such as the Unified Modeling Language (UML) and the Object Constraint Language (OCL). Given that test data generation is one of the major challenges to automate MBT, we focus on test data generation from OCL constraints in this paper. This endeavor is all the more challenging given the numerous OCL constructs and operations that are designed to facilitate the definition of constraints. Though search-based software testing has been applied to test data generation for white-box testing (e.g., branch coverage), its application to the MBT of industrial software systems has been limited. In this paper, we propose a set of search heuristics targeted to OCL constraints to guide test data generation and automate MBT in industrial applications. We evaluate these heuristics for three search algorithms: Genetic Algorithm, (1+1) Evolutionary Algorithm, and Alternating Variable Method. We empirically evaluate our heuristics using complex artificial problems, followed by empirical analyses of the feasibility of our approach on one industrial system in the context of robustness testing. Our approach is also compared with the most widely referenced OCL solver (UMLtoCSP) in the literature and shows to be significantly more efficient.

System testing of real-time embedded systems (RTES) is a challenging task and only a fully automated testing approach can scale up to the testing requirements of industrial RTES. One such approach, which offers the advantage for testing teams to be black-box, is to use environment models to automatically generate test cases and oracles and an environment simulator to enable earlier and more practical testing. In this paper, we propose novel heuristics for search-based, RTES system testing which are based on these environment models. We evaluate the fault detection effectiveness of two search-based algorithms, i.e., Genetic Algorithms and (1+1) Evolutionary Algorithm, when using these novel heuristics and their combinations. Preliminary experiments on 13 carefully selected, non-trivial artificial problems, show that, under certain conditions, these novel heuristics are effective at bringing the environment into a state exhibiting a system fault. The heuristic combination that showed the best overall performance on the artificial problems was applied on an industrial case study where it showed consistent results.

A substantial amount of work has shed light on whether random testing is actually a useful testing technique. Despite its simplicity, several successful real-world applications have been reported in the literature. Although it is not going to solve all possible testing problems, random testing appears to be an essential tool in the hands of software testers. In this paper, we review and analyze the debate about random testing. Its benefits and drawbacks are discussed. Novel results addressing general questions about random testing are also presented, such as how long does random testing need, on average, to achieve testing targets (e.g., coverage), how does it scale, and how likely is it to yield similar results if we rerun it on the same testing problem (predictability). Due to its simplicity that makes the mathematical analysis of random testing tractable, we provide precise and rigorous answers to these questions. Results show that there are practical situations in which random testing is a viable option. Our theorems are backed up by simulations and we show how they can be applied to most types of software and testing criteria. In light of these results, we then assess the validity of empirical analyzes reported in the literature and derive guidelines for both practitioners and scientists.

There has been a lot of work to shed light on whether random testing is actually a useful testing technique. Despite its simplicity, several successful real-world applications appear in the literature. Although it is not going to solve all possible testing problems, random testing is an essential tool in the hands of software testers. In this paper, we address general questions about random testing, such as how long random testing needs on average to achieve testing targets (e.g., coverage), how does it scale and how likely is it to yield similar results if we re-run random testing on the same testing problem. Due to its simplicity that makes the mathematical analysis of random testing tractable, we provide precise and rigorous answers to these questions. Our formal results can be applied to most types of software and testing criteria. Simulations are carried out to provide further support to our formal results. The obtained results are then used to assess the validity of empirical analyses reported in the literature. Results show that random testing is more effective and predictable than previously thought.

Web service composition is an emerging trend in the field of service oriented architecture, where a new Web service is developed using existing Web services. Developing a composite Web service is a complex task. There are a number of reported techniques for modeling Web service compositions to specify the exact requirements, identify errors, and eliminate the conflicts in the development of a composite Web service at design level. In this paper, we study and classify various UML based approaches for modeling web service compositions. We also present a detailed analysis of all such approaches.

This paper presents a methodology for identifying changes and test case selection based on the UML designs of the system. Design artifacts used for this purpose are UML class diagram and sequence diagrams, which are used to generate an extended concurrent control flow graph (ECCFG) which is further used for regression testing, i.e., change identification and test case selection. A proof- of-concept tool has been developed and used on a case study, which shows that our approach selects a precise set of test cases from an existing test suite.