ABSTRACT System‐level testing of avionics software systems requires compliance with different international safety standards such as DO‐178C. An important consideration of the avionics industry is automated test data generation according to the criteria suggested by safety standards. One of the recommended criteria by DO‐178C is the modified condition/decision coverage (MC/DC) criterion. Current model‐based test data generation approaches use constraints written in Object Constraint Language (OCL) and apply search techniques to generate test data. These approaches either do not support MC/DC criterion or suffer from performance issues while generating test data for large‐scale avionics systems. In this paper, we propose an effective way to automate MC/DC test data generation during model‐based testing. We develop a strategy that utilizes case‐based reasoning (CBR) and range reduction heuristics designed to solve MC/DC‐tailored OCL constraints. We performed an empirical study to compare our proposed strategy for MC/DC test data generation using CBR, range reduction, both CBR and range reduction, with an original search algorithm, and random search. We also empirically compared our strategy with existing constraint‐solving approaches. The results show that both CBR and range reduction for MC/DC test data generation outperform the baseline approach. Moreover, the combination of both CBR and range reduction for MC/DC test data generation is an effective approach compared to existing constraint solvers.

Summary The unmanned aircraft system (UAS) is rapidly gaining popularity in civil and military domains. A UAS consists of an application software that is responsible for defining a UAS mission and its expected behavior. A UAS during its mission experiences changes (or interruptions ) that require the unmanned aerial vehicle (UAV) in a UAS to self‐adapt, that is, to adjust both its behavior and position in real‐time, particularly for maintaining formation in the case of a UAS swarm. This adaptation is critical as the UAS operates in an open environment, interacting with humans, buildings, and neighboring UAVs. To verify if a UAS correctly makes an adaptation, it is important to test it. The current industrial practice for testing the self‐adaptive behaviors in UAS is to carry out testing activities manually. This is particularly true for existing UAS rather than newly developed ones. Manual testing is time‐consuming and allows the execution of a limited set of test cases. To address this problem, we propose an automated model‐based approach to test the self‐adaptive behavior of UAS application software. The work is conducted in collaboration with an industrial partner and demonstrated through a case study of UAS swarm formation flight application software. Further, the approach is verified on various self‐adaptive behaviors for three open‐source autopilots (i.e., Ardu‐Copter, Ardu‐Plane, and Quad‐Plane). Using the proposed model‐based testing approach we are able to test sixty unique self‐adaptive behaviors. The testing results show that around 80% of the behavior adaptations are correctly executed by UAS application software.

Abstract Model transformation testing has become crucial as model‐driven engineering has raised the abstraction level for developing software systems. Transformation is written to transform models from one level of abstraction to another, for example, model to model or model to code. A major challenge in testing the transformation is the creation of test models, such that (i) they conform to the source meta‐model (i.e., multiplicities and Object Constraint Language [OCL] constraints on meta‐model) and (ii) they provide coverage of the complete transformation (solving branch conditions for traversing all paths). Manual creation of test models requires a lot of time and effort. Still, the validity of the developed test models cannot be ensured. This paper aims to solve the above challenges using an automated search‐based strategy. The proposed approach is two‐stepped. First, valid test models are generated by solving source meta‐model constraints. Second, the generated models are evolved for achieving the structural coverage of the transformation by solving the branch conditions. A toolset model transformation testing environment (MOTTER) is developed to automate the search‐based solution. The proposed work is empirically evaluated on two case studies using four search algorithms. The result reflects that it successfully generates valid test models for achieving desired structural coverage with high performance on both the case studies.

Abstract Data science (DS) applications not only suffer from traditional software faults but may also suffer from data‐specific and model‐related faults. Fault models play an important role in evaluating and designing tests for testing DS applications. The existing fault models do not consider DS specific faults. In this study, we built a fault model DS applications. We investigate the faults by using diverse approaches: (i) a multi‐vocal literature survey of published literature, (ii) semi‐structured interviews of industry experts. The Multi‐vocal study allows us to synthesize the existing knowledge from researchers and practitioners. Qualitative data from semi‐structured interviews provide us with insights into the nature of faults encountered by practitioners. We combine the results of (i) and (ii) to derive a detailed fault model. The developed fault model is further validated through a quantitative survey of industry practitioners, and the respondents were asked to identify the faults from our proposed fault model that they have experienced and classify those faults based on their severity as perceived by practitioners and its frequency. The results show that practitioners consider prediction bias and model decay as the most severe faults while data sampling and splitting faults along with feature engineering faults are the most frequent.

Avionics are highly critical systems that require extensive testing governed by international safety standards. Cockpit Display Systems (CDS) are an essential component of modern aircraft cockpits and display information from the user application using various widgets. A significant step in the testing of avionics is to evaluate whether these CDS are displaying the correct information. A common industrial practice is to manually test the information on these CDS by taking the aircraft into different scenarios during the simulation. Given the large number of scenarios to test, manual testing of such behavior is a laborious activity. In the previous work, we proposed a model-based testing approach for the CDS. In this paper, we present a CDST (CDS Testing) toolkit that automates the process of testing CDS. We discuss the workflow and architecture of the tool and also demonstrates the tool on an industrial case study. The results show that the tool is able to generate, execute, and evaluate the test cases and identify 3 bugs in the case study.

Abstract Mobile devices have limited resources, including memory and processing speed. The performance of mobile applications is an important concern. There are a large number of mobile platforms available with varying operating systems and hardware. Native applications are usually developed and maintained separately for these platforms. The overall performance of native applications may significantly vary across platforms. The current industrial practice is to manually test the performance for each variant, which is not a scalable or efficient approach. We tackled the problem of generating native application variants in our previous work. This paper proposes an automated model‐based approach for performance test generation for native application variants at unit level. We propose a performance profile that allows modeling of domain‐specific performance parameters on UML models, which are used for automated performance test generation for each native variant. The results of applying the approach on two real‐world applications show that the approach evaluates the performance of application variants for two different versions of Android successfully and have potential to reduce the effort and time. A questionnaire‐based study is conducted to evaluate the usefulness of the approach.

Avionics are highly critical systems that require extensive testing governed by international safety standards. Cockpit Display Systems (CDS) are an essential component of modern aircraft cockpits and display information from the user application (UA) using various widgets. A significant step in the testing of avionics is to evaluate whether these CDS are displaying the correct information. A common industrial practice is to manually test the information on these CDS by taking the aircraft into different scenarios during the simulation. Such testing is required very frequently and at various changes in the avionics. Given the large number of scenarios to test, manual testing of such behavior is a laborious activity. In this paper, we propose a model-based strategy for automated testing of the information displayed on CDS. Our testing approach focuses on evaluating that the information from the user applications is being displayed correctly on the CDS. For this purpose, we develop a profile for capturing the details of different widgets of the display screens using models. The profile is based on the ARINC 661 standard for Cockpit Display Systems. The expected behavior of the CDS visible on the screens of the aircraft is captured using constraints written in Object Constraint Language. We apply our approach on an industrial case study of a Primary Flight Display (PFD) developed for an aircraft. Our results showed that the proposed approach is able to automatically identify faults in the simulation of PFD. Based on the results, it is concluded that the proposed approach is useful in finding display faults on avionics CDS.

Game development has recently gained a lot of momentum and is now a major software development industry. Platform games are being revived with both their 2D and 3D versions being developed. A major challenge faced by the industry is a lack of automated system-level approaches for game testing. Currently in most game development organizations, games are tested manually or using semi-automated techniques. Such testing techniques do not scale to the industry requirements where more systematic and repeatable approaches are required. In this paper we propose a model-based testing approach for automated black box functional testing of platform games. The paper provides a detailed modeling methodology to support automated system-level game testing. As part of the methodology, we provide guidelines for modeling the platform games for testing using our proposed game test modeling profile. We use domain modeling for representing the game structure and UML state machines for behavioral modeling. We present the details related to automated test case generation, execution, and oracle generation. We demonstrate our model-based testing approach by applying it on two cases studies, a widely referenced and open source implementation of Mario brothers game and an industrial case study of an endless runner game. The proposed approach was able to identify major faults in the open source game implementation. Our results showed that the proposed approach is practical and can be applied successfully on industrial games.

Mobile application development has emerged as one of the most focused areas in software industry due to exponential growth of mobile users and applications. The focus of the current mobile industry is on direct implementation, rather than analysis and design. Model-driven engineering practices are ignored, which results in low reusability and lack of portability along with other challenges. In addition, mobile applications also have to support multiple platforms, as an application written for one platform (e.g., Android) cannot run on another platform (e.g., Windows Phone). Each of the platforms in turn suffers from fragmentation. This results in multiple versions of an application that need to be simultaneously maintained. This is a huge burden on the development team, both in terms of cost and effort, due to various platforms that an application supports. This paper proposes a model-driven approach to generate mobile applications for multiple platforms. We carefully identify a subset of UML and propose a modeling methodology for this purpose. We use real use-cases for requirement gathering, class diagram for structural modeling, while state machine are used for behavioral modeling. We focus on modeling business logic layer for the mobile application. We also propose a UML profile for modeling mobile domain specific concepts. Our approach uses Action Language for Foundational Subset of UML (ALF) to specify actions in the state machines. Our approach allows the modeler to automatically generate business logic code for multiple platforms. The approach is supported by an automated tool MAG (Mobile Application Generator). We apply our model-driven approach on an industrial case study 'Scramble' app developed by 'Next In' to demonstrate its viability.

UML state machines are commonly used to model the state-based behavior of communication and control systems to support various activities such as test cases and code generation. Standard UML state machines are well suited to model functional behavior, however extra-functional behavior such as robustness and security can also be directly modeled on them, but this often results in cluttered models since extra-functional behaviors are often crosscutting. Such modeling crosscutting behavior directly on UML state machines is a common practice. Aspect-Oriented Modeling (AOM) allows systematically modeling of crosscutting behavior and has shown to provide a scalable solution in the recent years. However, due to lack of familiarity of AOM in both academic and industry, extra-functional behavior is often modeled directly on UML state machines and as a result those UML state machines are difficult to read and maintain. To improve the readability of already developed UML state machines and ease maintenance, we propose a set of heuristics, derived from two industrial cases studies, implemented in a tool to automatically identify commonly observed crosscutting behaviors in UML state machines and refactor them as Aspect State Machines. Such refactoring makes the state machines easier to maintain and comprehend. We present the results of applying our proposed heuristics to the existing UML state machines of two industrial case studies developed for model-based testing.