Efficient debugging is essential for software maintenance, as identifying, reproducing, and correcting bugs are critical to ensuring software reliability. However, debugging remains a time-consuming and cost-sensitive activity due to the complexities involved in analyzing and resolving bugs throughout the software development lifecycle. Recent advancements in Artificial Intelligence, Machine Learning, and particularly Natural Language Processing (NLP), and Large Language Models (LLM) offer promising opportunities to enhance debugging process. Since bug reports are typically written in natural language, NLP techniques can streamline debugging tasks, such as bug categorization, localization, and resolution. To assess the potential of NLP in debugging, we conduct a Systematic Literature Review (SLR) of 87 research papers published between 2011 and 2025. Our study presents a comprehensive taxonomy of research efforts, evaluates the effectiveness and limitations of various NLP methods, and highlights best practices across key software debugging tasks, including bug reproduction, bug localization, and bug report analysis. Furthermore, we identify significant challenges in real-world applications, including issues with scalability, accuracy, and adaptability to diverse software environments. Finally, we derive seven findings from our result analysis and propose future research directions corresponding to the findings, to advance NLP-driven debugging practices. This study provides a holistic overview of NLP in debugging for researchers, practitioners, and tool developers, helping to uncover trends, address gaps, and inspire new approaches to improving software maintenance workflows.

Malicious Python packages make software supply chains vulnerable by exploiting trust in open-source repositories like Python Package Index (PyPI). Lack of real-time behavioral monitoring makes metadata inspection and static code analysis inadequate against advanced attack strategies such as typosquatting, covert remote access activation, and dynamic payload generation. To address these challenges, we introduce DySec, a machine learning (ML)-based dynamic analysis framework for PyPI that uses eBPF kernel and user-level probes to monitor behaviors during package installation. By capturing 36 real-time features–including system calls, network traffic, resource usage, directory access, and installation patterns–DySec detects threats like typosquatting, covert remote access activation, dynamic payload generation, and multiphase attack malware. We developed a comprehensive dataset of 14,271 Python packages, including 7,127 malicious sample traces, by executing them in a controlled isolated environment. Experimental results demonstrate that DySec achieves 96% detection accuracy with an ML inference latency of <0.5s after dynamic feature extraction, reducing false negatives by 78.65% compared to static analysis and 82.24% compared to metadata analysis. During the evaluation, DySec flagged eleven packages that PyPI classified as benign. A manual analysis, including installation behavior inspection, confirmed six of them as malicious. These findings were reported to PyPI maintainers, resulting in the removal of four packages. DySec bridges the gap between reactive traditional methods and proactive, scalable threat mitigation in open-source ecosystems by uniquely detecting malicious install-time behaviors.

Securing software supply chains is a growing challenge due to the inadequacy of existing datasets in capturing the complexity of next-gen attacks, such as multiphase malware execution, remote access activation, and dynamic payload generation. Existing datasets, which rely on metadata inspection and static code analysis, are inadequate for detecting such attacks. This creates a critical gap because these datasets do not capture what happens during and after a package is installed. To address this gap, we present QUT-DV25, a dynamic analysis dataset specifically designed to support and advance research on detecting and mitigating supply chain attacks within the Python Package Index (PyPI) ecosystem. This dataset captures install and post-install-time traces from 14,271 Python packages, of which 7,127 are malicious. The packages are executed in an isolated sandbox environment using an extended Berkeley Packet Filter (eBPF) kernel and user-level probes. It captures 36 real-time features, that includes system calls, network traffic, resource usages, directory access patterns, dependency logs, and installation behaviors, enabling the study of next-gen attack vectors. ML analysis using the QUT-DV25 dataset identified four malicious PyPI packages previously labeled as benign, each with thousands of downloads. These packages deployed covert remote access and multi-phase payloads, were reported to PyPI maintainers, and subsequently removed. This highlights the practical value of QUT-DV25, as it outperforms reactive, metadata, and static datasets, offering a robust foundation for developing and benchmarking advanced threat detection within the evolving software supply chain ecosystem.

The increasing trend of using Large Language Models (LLMs) for code generation raises the question of their capability to generate trustworthy code. While many researchers are exploring the utility of code generation for uncovering software vulnerabilities, one crucial but often overlooked aspect is the security Application Programming Interfaces (APIs). APIs play an integral role in upholding software security, yet effectively integrating security APIs presents substantial challenges. This leads to inadvertent misuse by developers, thereby exposing software to vulnerabilities. To overcome these challenges, developers may seek assistance from LLMs. In this paper, we systematically assess ChatGPT's trustworthiness in code generation for security API use cases in Java. To conduct a thorough evaluation, we compile an extensive collection of 48 programming tasks for 5 widely used security APIs. We employ both automated and manual approaches to effectively detect security API misuse in the code generated by ChatGPT for these tasks. Our findings are concerning: around 70% of the code instances across 30 attempts per task contain security API misuse, with 20 distinct misuse types identified. Moreover, for roughly half of the tasks, this rate reaches 100%, indicating that there is a long way to go before developers can rely on ChatGPT to securely implement security API code.

The emerging threat landscape causes continuous change in the Incident Response Process (IRP) and security tools of security orchestration platforms (SOAR). Users of such platforms often struggle to adapt to these changes because they are addressed in an ad-hoc manner through a complex architecture. The complex design of the SOAR can be hidden behind an easy-to-use user interface. This article introduces a Declarative API (DAPI)-driven Orchestration approach, DecOr, that alleviates the need for security teams’ detailed understanding of the libraries and plugins to address the changes of a SOAR. DecOr comprises 1) three sets of dAPIs to encapsulate the activities of security orchestration and 2) a semantic framework to support the design and generation of dAPIs from task descriptions, leveraging natural language processing techniques. The dAPIs are mapped with an ontological knowledge base to execute IRPs. We experimentally evaluate the effectiveness and efficiency of DecOr based on 147 task and dAPI pairs, curated from real-world playbooks. We show the end-to-end process from identifying dAPIs to executing 48 IRPs with seven security tools. The evaluation results show, DecOr accurately generates dAPIs in near real-time, with precision and recall values over 80% and successfully executes changing IRPs 93% of the time.

Security Orchestration, Automation, and Response (SOAR) platforms integrate and orchestrate a wide variety of security tools to accelerate the operational activities of Security Operation Center (SOC). Integration of security tools in a SOAR platform is mostly done manually using APIs, plugins, and scripts. SOC teams need to navigate through API calls of different security tools to find a suitable API to define or update an incident response action. Analyzing various types of API documentation with diverse API format and presentation structure involves significant challenges such as data availability, data heterogeneity, and semantic variation for automatic identification of security tool APIs specific to a particular task. Given these challenges can have negative impact on SOC team’s ability to handle security incident effectively and efficiently, we consider it important to devise suitable automated support solutions to address these challenges. We propose a novel learning-based framework for automated security tool API R ecommendation for security O rchestration, automation, and response, APIRO . To mitigate data availability constraint, APIRO enriches security tool API description by applying a wide variety of data augmentation techniques. To learn data heterogeneity of the security tools and semantic variation in API descriptions, APIRO consists of an API-specific word embedding model and a Convolutional Neural Network (CNN) model that are used for prediction of top three relevant APIs for a task. We experimentally demonstrate the effectiveness of APIRO in recommending APIs for different tasks using three security tools and 36 augmentation techniques. Our experimental results demonstrate the feasibility of APIRO for achieving 91.9% Top-1 Accuracy. Compared to the state-of-the-art baseline, APIRO is 26.93%, 23.03%, and 20.87% improved in terms of Top-1, Top-2, and Top-3 Accuracy and outperforms the baseline by 23.7% in terms of Mean Reciprocal Rank (MRR).

A wide variety of security software systems need to be integrated into a Security Orchestration Platform (SecOrP) to streamline the processes of defending against and responding to cybersecurity attacks. Lack of interpretability and interoperability among security systems are considered the key challenges to fully leverage the potential of the collective capabilities of different security systems. The processes of integrating security systems are repetitive, time-consuming and error-prone; these processes are carried out manually by human experts or using ad-hoc methods. To help automate security systems integration processes, we propose an Ontology-driven approach for Security OrchestrAtion Platform (OnSOAP). The developed solution enables interpretability, and interoperability among security systems, which may exist in operational silos. We demonstrate OnSOAP's support for automated integration of security systems to execute the incident response process with three security systems (Splunk, Limacharlie, and Snort) for a Distributed Denial of Service (DDoS) attack. The evaluation results show that OnSOAP enables SecOrP to interpret the input and output of different security systems, produce error-free integration details, and make security systems interoperable with each other to automate and accelerate an incident response process.