Specifying reward functions without causing side effects is still a challenge to be solved in Reinforcement Learning. Attainable Utility Preservation (AUP) seems promising to preserve the ability to optimize for a correct reward function in order to minimize negative side-effects. Current approaches however assume the existence of a no-op action in the environment’s action space, which limits AUP to solve tasks where doing nothing for a single time-step is a valuable option. Depending on the environment, this cannot always be guaranteed. We introduce four different baselines that do not build on such actions and therefore extend the concept of AUP to a broader class of environments. We evaluate all introduced variants on different AI safety gridworlds and show that this approach generalizes AUP to a broader range of tasks, with only little performance losses.

Today’s IT and OT infrastructure is threatened by a plethora of cyber-attacks conducted by actors with different motivations and means. Furthermore, the complexity of these exposed systems as well as the adversaries’ sophisticated technical arsenal makes it increasingly difficult to plan and implement an organization’s defense. Understanding the link between specific attacks and effective mitigating measures is particularly challenging – as is understanding the underlying information security concepts. To support the training of current, and more importantly, nascent security engineers, we propose PenQuest, a digital attack and defense game where an attacker attempts to compromise an abstracted IT infrastructure and the defender works to prevent or mitigate the threat. The game is based on MITRE ATT&CK, D3FEND, and the NIST SP 800-53 security standard and incorporates a multitude of concepts such as cyber kill chains, attack vectors, network segmentation, and more. PenQuest is built to support security education and risk assessment and was evaluated with a class of engineering students as well as independent security experts. Initial results show a significant increase in knowledge retention and attest to the game’s feasibility for educational use.