Techniques like deep reinforcement learning (DRL) enable autonomous agents to solve tasks in complex environments automatically through learning. Despite their potential, neural-network-based decision-making policies are hard to understand and test. To ease the adoption of such techniques, we learn automata models of environmental behavior under the control of an agent. These models provide insights into the decisions faced by agents and a basis for testing. To scale automata learning to environments with complex and continuous dynamics, we compute an abstract state-space representation through dimensionality reduction and clustering of observed environmental states. The stochastic transitions are learned via passive automata learning from agent-environment interactions. Furthermore, we iteratively sample additional tra-jectories to enhance the learned model's accuracy. We demonstrate the potential of our automata learning frame-work by (1) solving popular RL benchmark problems and (2) applying it for differential testing of DRL agents. Our results show that the learned models are sufficiently precise to compute policies that solve the respective control tasks. Yet the models are sufficiently general for coverage-guided testing, where we reveal significant differences in the functional failure frequency of pairs of DRL agents.

Abstract Black-box systems are inherently hard to verify. Many verification techniques, like model checking, require formal models as a basis. However, such models often do not exist, or they might be outdated. Active automata learning helps to address this issue by offering to automatically infer formal models from system interactions. Hence, automata learning has been receiving much attention in the verification community in recent years. This led to various efficiency improvements, paving the way toward industrial applications. Most research, however, has been focusing on deterministic systems. In this article, we present an approach to efficiently learn models of stochastic reactive systems. Our approach adapts $$L^*$$ L ∗ -based learning for Markov decision processes, which we improve and extend to stochastic Mealy machines. When compared with previous work, our evaluation demonstrates that the proposed optimizations and adaptations to stochastic Mealy machines can reduce learning costs by an order of magnitude while improving the accuracy of learned models.

The number of devices in the Internet of Things (IoT) immensely grew in recent years. A frequent challenge in the assurance of the dependability of IoT systems is that components of the system appear as a black box. This paper presents a semi-automatic testing methodology for black-box systems that combines automata learning and fuzz testing. Our testing technique uses stateful fuzzing based on a model that is automatically inferred by automata learning. Applying this technique, we can simultaneously test multiple implementations for unexpected behavior and possible security vulnerabilities.We show the effectiveness of our learning-based fuzzing technique in a case study on the MQTT protocol. MQTT is a widely used publish/subscribe protocol in the IoT. Our case study reveals several inconsistencies between five different MQTT brokers. The found inconsistencies expose possible security vulnerabilities and violations of the MQTT specification.