Abstract Message Queuing Telemetry Transport (MQTT) is the most widely used protocol within the communication layer of the Internet of Things (IoT). Message brokers are a key component of the MQTT protocol and a single point of failure. Incompatibilities between different MQTT brokers or broker versions with their clients can cause critical failures and become a source of security risks. Thus, every MQTT broker change or update needs to be accompanied by an evaluation of the compatibility between the new and the previous broker. In this work, we develop an automated framework for compatibility evaluation of MQTT brokers, which can be easily generalized to other similar IoT components. We apply this framework to perform a comprehensive experiment conducted with 16 different versions of 6 popular MQTT brokers. We report inconsistencies in the behavior of different MQTT brokers and broker versions. Based on the experiment results, we calculate and provide a visualization of compatibility among the evaluated brokers in terms of their distance, which indicates the risk of incompatibilities when replacing a broker with another one. The calculation of distance measures can be adjusted by giving higher weights to important features. We use this method to show security‐related differences between the brokers.

The dramatic increase of attacks and malicious activities has made security a major concern in the development of interconnected cyber-physical systems and raised the need to address this concern also in testing. The goal of security testing is to discover vulnerabilities in the system under test so that they can be fixed before an attacker finds and abuses them. However, testing for security issues faces the challenge of systematically exploring a potentially non-tractable number of interaction scenarios that have to include also invalid inputs and possible harmful interaction attempts. In this paper, we describe an approach for automated generation of test cases for security testing, which are based on attack patterns. These patterns are blueprints that can be used for exploiting common vulnerabilities. The approach combines random test case generation with attack patterns implemented for the Message Queuing Telemetry Transport (MQTT) protocol. We have applied the proposed testing approach to five popular and widely available MQTT brokers, generating 1,804 interaction sequences in form of executable test cases which resulted in numerous test failures, unhandled exceptions and crashes. A detailed manual analysis of these cases have revealed 28 security-relevant issues and critical shortcomings in the tested MQTT broker implementations.