In practice, integration testing typically focuses on a small selection of components or subsystems to integrate and test. This reduces the effort required to create test cases and test environments. However, many defects are only detected when performing integration testing on all possible integrations. These defects are typically only detected later in the development process and lead to increased testing and fault localization efforts. By describing and operationalizing knowledge of such defects, we are able to (semi-)automatically detect them in integration testing. Our OUTFIT tool targets superfluous or missing functionality and untested exception/fault handling in Matlab Simulink models and generated code. It re-uses existing, or automatically generates, high coverage test cases to measure coverage in an opportunistically assembled integration of components or subsystems. A manual inspection of the coverage results then reveals missing or potentially superfluous behavior and thus reveals defects of the targeted kind. Used in a bottom-up integration testing strategy, OUTFIT front loads the detection of such defects, and reduces the fault localization effort. We evaluate OUTFIT using three components of a real-world electrical engine control system of a hybrid car. We find that the results are reproducible, effective and efficiently produced. The achieved coverage of the results is reproducible for 10 executions within a small standard deviation. OUTFIT is effective in finding a potential defect, and efficiently analyzes all evaluated components within a worst case execution time of 110 minutes.

Ranging from temperature control to safety-critical applications, continuous controllers are used in a plethora of applications becoming increasingly complex. In turn, testing continuous control systems also is more complex. Particularly, application-specific manual formal analysis or testing the complete input range becomes infeasible. We present a comprehensive failure-based testing methodology and a respective automated tool for continuous controllers. Our methodology is based on an existing automated approach, testing stability, liveness, smoothness and responsiveness in a single value-response scenario only. We performed a practitioner survey and literature review in the domain revealing the quality criteria steadiness and reliability to be vital for meaningful testing of continuous controllers. In addition, we identified 4 further scenarios including disturbance response for comprehensive testing. We contribute a library of failure models and quality criteria for the automated testing of continuous control systems more complete than in previous approaches. On the grounds of our comprehensive experiments on 9 real-world control systems, our results demonstrate our failure-based testing methodology to provide better worst cases than manual testing (effectiveness) within an adequate time frame (efficiency) for any configuration used in our experiments (reproducibility).

Matlab Simulink models, mainly used for the specification of continuous embedded systems, employ a data flow-driven notation well understood by engineers. This notation abstracts from the underlying computational model, hiding run time failures such as over-/underflows and divisions by zero. They are often detected late in the development process by the use of static analysis tools on the completely developed system. The responsible underlying faults are sometimes attributable to a single operation in a model. 8Cage is an automated test case generator for the early detection of such single operation related faults. It is configurable to detect these faults and runs automatically in the background. It tries to find potentially failure-causing operations and generates a test case to gather evidence for an actual fault. 8Cage is usable by developing/testing engineers with knowledge of Matlab. It does not require an expert to perform result validation or fault localization.