Red teams simulate adversaries and conduct sophisticated attacks against defenders without informing them about used tactics in advance. These interactive cyber exercises are highly beneficial to assess and improve the security posture of organizations, detect vulnerabilities, and train employees. Unfortunately, they are also time-consuming and expensive, which often limits their scale or prevents them entirely. To address this situation, adversary emulation tools partially automate attacker behavior and enable fast, continuous, and repeatable security testing even when involved personnel lacks red teaming experience. Currently, a wide range of tools designed for specific use-cases and requirements exist. To obtain an overview of these solutions, we conduct a review and structured comparison of nine open-source adversary emulation tools. To this end, we assemble a questionnaire with 80 questions addressing relevant aspects, including setup, support, documentation, usability, and technical features. In addition, we conduct a user study with domain experts to investigate the importance of these aspects for distinct user roles. Based on the evaluation and user feedback, we rank the tools and find MITRE Caldera, Metasploit, and Atomic Red Team on top.

Many modern cyber attack techniques cannot be prevented. Logging and monitoring, however, offer a means to at least detect these techniques early, and therefore become increasingly important for defense. Many companies are unfortunately reluctant to invest more in cyber security logging and monitoring or hire additional security staff to operate detective solutions. There is a need for a methodology to pick appropriate cyber security solutions from the vast pool of available products. Our model takes requirements mandated by common standards from ISO, NIST, BSI and the like into account. While standards and guidelines remain at a high abstraction level and are applicable to different organizations over a long period of time, guidance on implementation becomes outdated comparatively quickly. We propose a novel logging maturity and decision model for the selection of the best fitting cyber security solutions for an organization. The novelty is that this model accounts for constraints in the selection process, such as cost, complexity, compliance, and relevance to the organization's assets. We validate the model with MITRE ATT&CK framework data and apply it to illustrative use cases based on our survey.