Large Language Models (LLMs) are currently being integrated into industrial software applications to help users perform more complex tasks in less time. However, these LLM-Integrated Applications (LIA) expand the attack surface and introduce new kinds of threats. Threat modeling is commonly used to identify these threats and suggest mitigations. However, it is a time-consuming practice that requires the involvement of a security practitioner. Our goals are to 1) provide a method for performing threat modeling for LIAs early in their lifecycle, (2) develop a threat modeling tool that integrates existing threat models, and (3) ensure high-quality threat modeling. To achieve the goals, we work in collaboration with our industry partner. Our proposed way of performing threat modeling will benefit industry by requiring fewer security experts’ participation and reducing the time spent on this activity. Our proposed tool combines LLMs and Retrieval Augmented Generation (RAG) and uses sources such as existing threat models and application architecture repositories to continuously create and update threat models. We propose to evaluate the tool offline—i.e., using benchmarking—and online with practitioners in the field. We conducted an early evaluation using ChatGPT on a simple LIA and obtained results that encouraged us to proceed with our research efforts.

Background: As Large Language Models (LLMs) reshape software development across industries, they also reshape the associated threat landscape. Traditional threat modeling methods, which assume predictable system behavior, struggle to accommodate the inherent nondeterminism of LLMs. Paradoxically, LLMs themselves offer capabilities, such as pattern recognition, natural language understanding, and semi-structured reasoning, that can support the automation of threat elicitation and mitigation. Aims: This research project, ThreMoLIA, aims to design, develop, and empirically evaluate a threat modeling tool that leverages LLMs to assist practitioners in identifying and analyzing security threats in LLM-integrated applications (LIAs). Method: To this end, we apply a mixed-methods exploratory case study to define and validate threat modeling metrics, and a comparative case study to evaluate the ThreMoLIA tool against existing threat modeling practices. Results: The current prototype of the ThreMoLIA tool uses cloud or local models. We have established, and partiallyvalidated, a measurement framework and a benchmark for the tool evaluation. Conclusions: The project is conducted in close collaboration with industry and contributes to the ESEM community by advancing Security-by-Design practices and sharing reproducible artifacts such as metrics, benchmarks, and threat models.