Summary In this paper, we report on applying combinatorial testing to Internet of Things (IoT) home automation hub systems. We detail how to create a dedicated input parameter model of an IoT home automation hub system for use with combinatorial test case generation strategies. Further, we developed an automated test execution framework and two test oracles for evaluation purposes. We applied and evaluated our proposed methodological approach to a real‐world IoT system and analysed the obtained results of various combinatorial test sets with different properties generated based on the derived input model. Additionally, we compare these results to a random testing approach. Our empirical testing evaluations revealed multiple errors in the tested devices and also showed that all considered approaches performed nearly equally well.

Summary This work presents an extended and enhanced gray‐box combinatorial security testing methodology for SQL injection vulnerabilities in web applications. We propose multiple new attack grammars modelling SQLi attacks against MySQL‐compatible databases, each one targeting a different injection context. Additionally, these grammars are also dynamically refined at the beginning of each attack against an endpoint of a web application, as a further optimization of the used attack model by taking into account the specifics of the generated query of that endpoint. Our goal is to enhance existing combinatorial approaches for detecting SQL injection vulnerabilities. The newly developed methodology is implemented in a prototype security testing tool called SQLInjector+, which is an extension of an earlier prototype developed by us in prior work. This improved tool can attack (i.e. test) any web application that uses a MySQL‐compatible database management system. We evaluate our revised approach and improved prototype tool in a case study comprising of different kinds of web applications to which SQLi is a potential security threat. The case study contains the well‐known verification framework WAVSEP among other five real‐world web applications and one web application firewall. Our generated attack vectors, constructed via combinatorial methods applied to our improved and dynamically optimized attack grammars, are capable of injecting every known vulnerable endpoint in WAVSEP and also of finding new vulnerable parameters in some of the real‐world applications investigated in this paper. Our approach performs equally well or better when compared with existing state‐of‐art of SQL injection security testing tools (sqlmap, w3af, wapiti and fuzzdb) across all tested web applications in the case study.

Security testing of web applications remains a major problem of software engineering. In order to reveal vulnerabilities, manual and automatic testing approaches use different strategies for detection of certain kinds of inputs that might lead to a security breach. In this paper we compared a state-of-the-art manual testing tool with an automated one that is based on model-based testing. The first tool requires user input from the tester whereas the second one reduces the necessary amount of manual manipulation. Both approaches depend on the corresponding test case generation technique and its produced inputs are executed against the system under test (SUT). For this case we enhance a novel technique, which combines a combinatorial testing technique for input generation and a model-based technique for test execution. In this work the input parameter modelling is improved by adding constraints to generate more comprehensive and sophisticated testing inputs. The evaluated results indicate that both techniques succeed in detecting security leaks in web applications with different results, depending on the background logic of the testing approach. Last but not least, we claim that attack pattern-based combinatorial testing with constraints can be an alternative method for web application security testing, especially when we compare our method to other test generation techniques like fuzz testing.