In the past years, a growing number of highly-automated systems has build on Artificial-Intelligence (AI) capabilities, for example, self-driving vehicles or predictive health-state diagnoses. As for any software system, there is a risk that misbehavior occurs (e.g., system failure due to bugs) or that malicious actors aim to misuse the system (e.g., generating attack scripts), which can lead to safety and security incidents. While software safety and security incidents have been studied in the past, we are not aware of research focusing on the specifics of AI incidents. With this paper, we aim to shed light on this gap through a case survey of 240 incidents that we elicited from four datasets comprising safety and security incidents involving AI from 2014 to 2023. Using manual data analyses and automated topic modeling, we derived relevant topics as well as the major issues and contexts in which the incidents occurred. We find that the topic of AI incidents is, not surprisingly, becoming more and more relevant, particularly in the contexts of autonomous driving and process-automation robotics. Regarding security and its intersection with safety, most incidents connect to generative AI (i.e., large-language models, deep fakes) and computer-vision systems (i.e., facial recognition). This emphasizes the importance of security to also ensure safety in the context of AI systems, with our results further revealing a high number of serious consequences (system compromise, human injuries) and major violations of confidentiality, integrity, availability, as well as authorization. We hope to support practitioners and researchers in understanding major safety and security issues to support the development of more secure, safe, and trustworthy AI systems.

The COVID-19 virus has caused a global pandemic that has heavily impacted daily life. Rapid advances in testing and vaccinating led to an additional use case besides the well-known contact-tracing apps: certificate-verification systems. Verification systems are often commissioned by local authorities to enable more public life, and are often developed by smaller organizations or startups. So, the development of verification systems differs from other software projects, featuring interesting and unique properties. In this article, we present an experience report on the development of one verification system by a German startup, focusing on three properties: working in a pandemic, developing a product for handling a pandemic, and the startup context. To this end, we surveyed nine startup developers and analyzed the results with two experts from the startup. We found that the developers focused on fast delivery to cope with the time pressure of releasing the verification system, which is why some phases of typical development processes were hardly carried out. As a result, while the verification system is successful, we also identified negative effects of the properties (e.g., programming mistakes, well-being). We discuss our findings to guide researchers and practitioners in preparing for software engineering in future emergencies. Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.