Abstract A cyber-physical system-of-systems (CPSoS) facilitates the achievement of high-level goals, such as efficient traffic management on roads, by designing and developing the collaboration of constituent CPSs. A platooning that groups autonomous vehicles in proximity is an example of collaboration. The intricate collaboration innately causes serious collaboration failures such as collisions. However, limited knowledge and complex dynamics of CPSoS cause several challenges in effectively analyzing the collaboration failures. Existing studies have applied pattern mining techniques to investigate various failures but have limitations when applied to collaboration failures: (1) absence of data model for continuous and discrete logs in CPSoS; (2) information loss problem by not considering the integrated relationship of the data; (3) dependence only on failed logs; (4) limited capability of fixed-size time windows. We propose a fuzzy clustering-based pattern mining approach that consists of a novel data model for CPSoS logs and comprehensive metrics for classifying and mining optimal collaboration failure patterns. In experiments on vehicle platooning, our approach exhibited the highest accuracy on pattern mining and clustering results. Further, we identified five collaboration failure scenarios in the empirical analysis of drone swarming results. The findings of this study can facilitate the effective analysis of CPSoS collaboration failures.

Programmable logic controllers (PLCs) are widely used for process control and automation in safety-critical systems, such as reactor protection systems (RPS) in nuclear power plants. Industrial systems are often composed of multiple PLCs and have several interfaces with other systems. It is crucial to test PLC programs for such systems effectively. As manual integration testing is costly and time-consuming, simulation- based testing is becoming more important in validating safety- critical systems. However, simulation-based testing is limited because the simulation execution results may differ from the real PLC program execution results. This study presents the PLC synchronization method and multiple PLCs execution method for the Automated Simulation-based Integrated Testing approach (ASIT). MultiPLCTester is a test tool we developed to support ASIT and it executes test inputs with a simulation model and communicates with a PLC and compares them with execution results. Based on the synchronized test execution results, it con-ducts ASIT for target systems. We performed experiments using real industrial cases to evaluate the effectiveness and efficiency of our approach. The test suite execution time using ASIT was significantly reduced compared to the manual integration testing. When considering the test execution time and the cost of building a test bed, ASIT provides effective and efficient integrated testing for PLC testers.

Programmable Logic Controllers (PLCs) are the core unit of the production system, which frequently need to implement new processes to address customer needs. These changes must be fully tested to ensure the reliability of the PLC code, which is commonly programmed through Functional Block Diagrams (FBDs). This is a tedious task that requires considerable time and effort given the manual nature of the process involved in PLC testing. Hence, we present a cost-effective test selection approach to test FBD programs in dynamic environments. The proposed method uses a search-based multi-objective test case selection algorithm as a regression technique to test recently modified FBD programs. Specifically, we derived a total of 7 fitness function combinations, by combining different cost and quality-based fitness functions. We carried out an empirical evaluation, by employing fitness metrics in the wellknown NSGA-II algorithm to determine the best configuration setup for testing FBD programs. Furthermore, we benchmarked the performance of the NSGA-II with the baseline Random Search (RS). The study was carried out with three case studies of a reactor protection system, and evaluated with two sets of mutants. The results demonstrated that the proposed approach significantly reduces time, while keeping high the overall fault detection capability.

Summary Function block diagram (FBD) is a standard programming language for programmable logic controllers (PLCs). PLCs have been widely used to develop safety‐critical systems such as nuclear reactor protection systems. It is crucial to test FBD programs for such systems effectively. This paper presents an automated test sequence generation approach using mutation testing techniques for FBD programs and the developed tool, MuFBDTester. Given an FBD program, MuFBDTester analyses the program and generates mutated programs based on mutation operators. MuFBDTester translates the given program and mutants into the input language of a satisfiability modulo theories (SMT) solver to derive a set of test sequences. The primary objective is to find the test data that can distinguish between the results of the given program and mutants. We conducted experiments with several examples including real industrial cases to evaluate the effectiveness and efficiency of our approach. With the control of test size, the results indicated that the mutation‐based test suites were statistically more effective at revealing artificial faults than structural coverage‐based test suites. Furthermore, the mutation‐based test suites detected more reproduced faults, found in industrial programs, than structural coverage‐based test suites. Compared to structural coverage‐based test generation time, the time required by MuFBDTester to generate one test sequence from industrial programs is approximately 1.3 times longer; however, it is considered to be worth paying the price for high effectiveness. Using MuFBDTester, the manual effort of creating test suites was significantly reduced from days to minutes due to automated test generation. MuFBDTester can provide highly effective test suites for FBD engineers.

System of Systems (SoS), refers to a class of systems that are comprised of independent constituent systems (CS) interacting with one another to form a larger system, thus resulting in a common goal being achieved through the set of interactions between the CSs. In conventional systems software testing, the problem of identifying correct behaviour given a set of test inputs is called the oracle problem. In terms of SoS, the oracle problem is exacerbated due to each CS in an SoS being sufficiently complex systems themselves. We address the oracle problem in the domain of the software testing in System of Systems (SoS) by applying Metamorphic Testing (MT). In order to derive concrete Metamorphic Relations (MR) for SoS, we first borrow the concept of creating MRs from abstract Metamorphic Relation Patterns (MRPs), where the represented MRPs themselves describe characteristics that are used in the modeling and analysis of SoS. They are then applied to two SoS simulator types, the Smart Grid SoS and Mass Casualty Incident (MCI) Response SoS in order to test for reliability in SoS and to also determine the viability of MT in SoS. It is shown through the experiments conducted that the derived concrete MRs are able to find faults in both the systems under test.

A System-of-Systems (SoS) is a collection of systems, which consists of independent constituent systems to achieve higher-level goals. As an SoS changes constantly due to external and internal factors, dynamic reconfiguration and evolutionary development must be performed effectively. To manage an SoS that has these characteristics, SoS managers and engineers need to model and verify an SoS that accommodates changes. While many researchers have proposed techniques for modeling and verifying an SoS, little attention has been paid to the verification of a continuously changing SoS. Since the size and complexity of SoSs are too great to apply existing verification techniques, an efficient verification method for dynamically changing SoSs is required. If we can detect the change-related goals of an SoS and the corresponding change-affected parts in an SoS model, we can slice the SoS model in our area of concern and verify the SoS model efficiently. This paper proposes SoS GaP slicer, a verification method that considers the dynamic reconfiguration and evolution of an SoS. Our method could save the cost of verifying the whole of an SoS model by an efficient statistical model checking for SoSs. The experimental results show the accuracy and efficiency of the proposed method.

The worldwide mobile software market has grown dramatically since feature phones became popular in the early 1990s. In practice, mobile usability — which can be defined for a resource-constrained device in two ways, namely, User eXperience (UX) and User Interface (UI) — has been regarded as the key to gaining superiority in terms of both market share and customer loyalty. Unfortunately, de facto standards for software design and the development process, such as Unified Modeling Language (UML) and Rational Unified Process (RUP), do not seem to promote mobile usability in a systematic manner in practice. This paper proposes a systematic and generalizable approach to modeling and evaluating the properties of mobile usability, herein treating it as a first-class software quality from the perspective of software engineering. We devise a UX evaluation framework for mobile usability, which we call UX Evaluation Framework (UEF) throughout this paper. A UX is specified by inter-scene interactions between users and terminals of software products using Extended Menu Navigation Viewpoints (EMNVs); then, a model checker, NuSMV, is adopted to observe whether the EMNV model meets a set of given UX properties. Importantly, the analysis and design of RUP is extended to support the co-design of UX and UI so that major roles, activities and artifacts in the UX and UI can be explicitly monitored and controlled by stakeholders. Through case studies, we demonstrate that UEF works properly to treat software products that prioritize mobile usability. Consequently, UEF plays a key role in filling the gap between two research disciplines to address usability: software engineering and human–computer interactions.

As Function Block Diagram (FBD) programs are used to implement safety-critical systems such as nuclear reactor protection systems, it is crucial to be able to generate effective test cases. The FBD is one of programming languages that are used for programmable logic controllers (PLCs). PLC programs are repeatedly run within a particular scan time for every execution. Among the constituents of FBD programs, function blocks and feedback variables use the inputs and outputs of the previous scan cycle on which to operate. Researchers have recently developed an automated test generation technique that satisfied several structural test coverage criteria for FBD programs using symbolic execution. However, their work could not generate test sequences for consecutive scan cycle but test inputs for one scan cycle. Test sequences are essential for testing FBD programs that have function blocks and feedback variables. This paper extends previous work in the field by generating test sequences for FBD programs with function blocks in a fully automated manner. The key technique involves explicitly unwinding FBD programs and solving test requirements using an SMT solver. We conduct experiments on increasing the coverage of test requirements by unwinding cycles and evaluating the effectiveness of the test set using mutation analysis with several subject programs, including a real-world reactor protection system. The experimental results show that the proposed approach is able to generate effective test sequences for FBD programs.

Collective adaptive systems (CAS) consist of multiple agents that adapt to changing system and environmental conditions in order to satisfy system goals and quality requirements. As more applications involve using CAS in a critical context, ensuring the correct and safe adaptive behaviors of quality-driven CAS has become more important. In this paper, we propose Collective Adaptive System Testing (CAST), a scalable and efficient approach to testing self-adaptive behaviors of CAS. We propose a selective method to instantiate and execute test cases relevant to the current adaptation context. This enables testers to focus testing on key self-adaptive behaviors while dealing with the scale and dynamicity of the system. An experimental evaluation using a traffic monitoring system is performed to validate its scalability, efficiency, and fault-detection effectiveness. The experimental results provide insights into how CAST can serve as a feasible and effective assurance technique for CAS.

SUMMARY Automated and effective testing for function block diagram (FBD) programs has become an important issue, as FBD is increasingly used in implementing safety‐critical systems. This work describes an automated test case generation technique for FBD programs and its associated tool—FBDTester. Given an FBD program and desired test coverage criteria, FBDTester generates test requirements and invokes the Satisfiability Modulo Theories solver iteratively to derive a set of test cases. An industrial case study using reactor protection system software shows that the automatically generated test suites detected at least 82% of the known faults, whereas manually generated test cases only detected approximately 35%. Mutation analysis revealed that the automatically generated test suites substantially outperformed manually generated ones. Although test sequence generation requires some manual effort in the current FBDTester, it is apparent that the proposed approach significantly improves the efficiency and the reliability of FBD testing. Copyright © 2014 John Wiley & Sons, Ltd.

Medical cyber-physical systems (MCPS) are life-critical, context-aware, networked systems of medical devices. These systems are increasingly used in hospitals to provide high-quality continuous care for patients. The need to design complex MCPS that are both safe and effective has presented numerous challenges, including achieving high assurance in system software, intoperability, context-aware intelligence, autonomy, security and privacy, and device certifiability. In this paper, we discuss these challenges in developing MCPS, some of our work in addressing them, and several open research issues.

Guaranteeing timing properties is an important issue as we develop safety-critical real-time systems such as cardiac pacemakers. We present a safety assured development approach of real-time software using a pacemaker as our case study. Following the model-driven development techniques, measurement-based timing analysis is used to guarantee timing properties in implementation as well as in the formal model. Formal specification with timed automata is checked with respect to timing properties by model checking technique and is transformed into implementation systematically. When timing properties may be violated in the implementation due to timing delay, it is suggested to measure the time deviation and reflect it to the code explicitly by modifying guards. The model is altered according to the modifications in the code. These changes of the code and the model are considered safe if all the properties are still satisfied by the modified model in re-performed model checking. We demonstrate how the suggested approach can be applied to single-threaded and multi-threaded versions of implementation. This approach can provide developers with a useful time-guaranteeing technique applicable to several code generation schemes without imposing many restrictions.

Formal verification of Function Block Diagram (FBD) based software is an essential task when replacing traditional relay-based analog system with PLC-based software in nuclear reactor protection system (RPS). FBD programs are developed manually and revised frequently in process of development. There are a set of properties to be verified formally, which all FBD releases should satisfy. Whenever FBDs are modified, there is also a need to verify behavioral equivalence of subsequently modified FBDs. This paper proposes a software verification framework for FBD software in nuclear power plants. It uses SMV model checker for verifying whether an FBD meets its required properties, and VIS verification system for checking behavioral equivalence between modified FBDs. A case study, conducted using a nuclear power plant shutdown system being developed in Korea, demonstrated that the proposed verification framework is effective and useful.

Testing for time-related behaviors of PLC software is important and should be performed carefully. We propose a structural testing technique on function block diagram (FBD) networks including timer function blocks. In order to test FBD networks including timer function blocks, we generate templates for timer function blocks and transform a unit FBD into a flow-graph using the proposed templates. We apply existing testing techniques to the generated flowgraph and describe how the characteristics of timer function blocks are reflected in the testing process. By the proposed method, FBD networks including timer function blocks can be tested thoroughly without the intermediate code which was essential in the previous FBD testing. To demonstrate the effectiveness of the proposed method, we use a trip logic of bistable processor of digital plant protection systems which is being developed in Korea.