Large Language Models (LLMs) are currently being integrated into industrial software applications to help users perform more complex tasks in less time. However, these LLM-Integrated Applications (LIA) expand the attack surface and introduce new kinds of threats. Threat modeling is commonly used to identify these threats and suggest mitigations. However, it is a time-consuming practice that requires the involvement of a security practitioner. Our goals are to 1) provide a method for performing threat modeling for LIAs early in their lifecycle, (2) develop a threat modeling tool that integrates existing threat models, and (3) ensure high-quality threat modeling. To achieve the goals, we work in collaboration with our industry partner. Our proposed way of performing threat modeling will benefit industry by requiring fewer security experts’ participation and reducing the time spent on this activity. Our proposed tool combines LLMs and Retrieval Augmented Generation (RAG) and uses sources such as existing threat models and application architecture repositories to continuously create and update threat models. We propose to evaluate the tool offline—i.e., using benchmarking—and online with practitioners in the field. We conducted an early evaluation using ChatGPT on a simple LIA and obtained results that encouraged us to proceed with our research efforts.

Background: As Large Language Models (LLMs) reshape software development across industries, they also reshape the associated threat landscape. Traditional threat modeling methods, which assume predictable system behavior, struggle to accommodate the inherent nondeterminism of LLMs. Paradoxically, LLMs themselves offer capabilities, such as pattern recognition, natural language understanding, and semi-structured reasoning, that can support the automation of threat elicitation and mitigation. Aims: This research project, ThreMoLIA, aims to design, develop, and empirically evaluate a threat modeling tool that leverages LLMs to assist practitioners in identifying and analyzing security threats in LLM-integrated applications (LIAs). Method: To this end, we apply a mixed-methods exploratory case study to define and validate threat modeling metrics, and a comparative case study to evaluate the ThreMoLIA tool against existing threat modeling practices. Results: The current prototype of the ThreMoLIA tool uses cloud or local models. We have established, and partiallyvalidated, a measurement framework and a benchmark for the tool evaluation. Conclusions: The project is conducted in close collaboration with industry and contributes to the ESEM community by advancing Security-by-Design practices and sharing reproducible artifacts such as metrics, benchmarks, and threat models.

Context: The rise of Artificial Intelligence (AI) and Machine Learning (ML) applied in many software-intensive products and services introduces new opportunities but also new security challenges. Motivation: AI and ML will gain even more attention from industry in the future, but threats caused by already discovered attacks specifically targeting ML models are either overseen, ignored, or mishandled. Problem Statement: Current Software Engineering security practices and tools are insufficient to detect and mitigate ML Threats systematically. Contribution: We will develop and evaluate a threat modeling technique for non-security experts assessing ML-intensive systems in close collaboration with industry and academia.

Adversarial Machine Learning (AML) discusses the act of attacking and defending Machine Learning (ML) Models, an essential building block of Artificial Intelligence (AI). ML is applied in many software-intensive products and services and introduces new opportunities and security challenges. AI and ML will gain even more attention from the industry in the future, but threats caused by already-discovered attacks specifically targeting ML models are either overseen, ignored, or mishandled. Current AML research investigates attack and defense scenarios for ML in different industrial settings with a varying degree of maturity with regard to academic rigor and practical relevance. However, to the best of our knowledge, a synthesis of the state of academic rigor and practical relevance is missing. This literature study reviews studies in the area of AML in the context of industry, measuring and analyzing each study’s rigor and relevance scores. Overall, all studies scored a high rigor score and a low relevance score, indicating that the studies are thoroughly designed and documented but miss the opportunity to include touch points relatable for practitioners.