Program slicing is a critical technique in software engineering, enabling developers to isolate relevant portions of code for tasks such as bug detection, code comprehension, and debugging. In this study, we investigate the application of large language models (LLMs) to both static and dynamic program slicing, with a focus on Java programs. We evaluate the performance of four state-of-the-art LLMs, i.e., GPT-4o, GPT-3.5 Turbo, Llama-2, and Gemma-7B, by leveraging advanced prompting techniques, including few-shot learning and chain-of-thought reasoning. Using a dataset of 100 Java programs derived from LeetCode problems, our experiments reveal that GPT-4o performs the best in both static and dynamic slicing across other LLMs, achieving an accuracy of 60.84% and 59.69%, respectively. Our results also show that the LLMs we experimented with are yet to achieve reasonable performance for either static slicing or dynamic slicing. Through a rigorous manual analysis, we developed a taxonomy of root causes and failure locations to explore the unsuccessful cases in more depth. We identified Complex Control Flow as the most frequent root cause of failures, with the majority of issues occurring in Variable Declarations and Assignments locations. To improve the performance of LLMs, we further examined a widely-used strategy for prompting guided by our taxonomy, i.e., prompt crafting, which involved refining the prompts to better guide the LLM through the slicing process. Our evaluation shows that prompt crafting can improve accuracy by 4%.

In the ever-evolving field of software engineering, the advent of large language models and conversational interfaces, exemplified by ChatGPT, represents a significant revolution. While their potential is evident in various domains, this paper expands upon our previous research, where we experimented with GPT –4, on its ability to create safety cases. A safety case is a structured argument supported by a body of evidence to demonstrate that a given system is safe to operate in a given environment. In this paper, we first determine GPT –4’s comprehension of the Goal Structuring Notation (GSN), a well-established notation for visually representing safety cases. Additionally, we conduct four distinct experiments using GPT –4 to evaluate its ability to generate safety cases within a specified system and application domain. To assess GPT –4’s performance in this context, we compare the results it produces with the ground-truth safety cases developed for an X-ray system, a machine learning-enabled component for tire noise recognition in a vehicle, and a lane management system from the automotive domain. This comparison enables us to gain valuable insights into the model’s generative capabilities. Our findings indicate that GPT –4 is able to generate moderately accurate and reasonable safety cases.

An assurance case is a structured hierarchy of claims aiming at demonstrating that a mission-critical system supports specific requirements (e.g., safety, security, privacy). The presence of assurance weakeners (i.e., assurance deficits, logical fallacies) in assurance cases reflects insufficient evidence, knowledge, or gaps in reasoning. These weakeners can undermine confidence in assurance arguments, potentially hindering the verification of mission-critical system capabilities which could result in catastrophic outcomes (e.g., loss of lives). Given the growing interest in employing assurance cases to ensure that systems are developed to meet their requirements, exploring the management of assurance weakeners becomes beneficial. As a stepping stone for future research on assurance weakeners, we aim to initiate the first comprehensive systematic mapping study on this subject. We followed the well-established PRISMA 2020 and SEGRESS guidelines to conduct our systematic mapping study. We searched for primary studies in five digital libraries and focused on the 2012–2023 publication year range. Our selection criteria focused on studies addressing assurance weakeners from a qualitative standpoint, resulting in the inclusion of 39 primary studies in our systematic review. Our systematic mapping study reports a taxonomy (map) that provides a uniform categorization of assurance weakeners and approaches proposed to manage them from a qualitative perspective. The taxonomy classifies weakeners in four categories: aleatory, epistemic, ontological, and argument uncertainty. Additionally, it classifies approaches supporting the management of weakeners in three main categories: representation, identification and mitigation approaches. Our study findings suggest that the SACM (Structured Assurance Case Metamodel) – a standard specified by the OMG (Object Management Group) – offers a comprehensive range of capabilities to capture structured arguments and reason about their potential assurance weakeners. Our findings also suggest novel assurance weakener management approaches should be proposed to better assure mission-critical systems.