GitHub workflows or GitHub CI is a popular continuous integration platform that enables developers to automate various software engineering tasks by specifying them as workflows, i.e., YAML files with a list of jobs. However, engineering valid workflows is tedious. They are also prone to severe security issues, which can result in supply chain vulnerabilities. Recent advancements in Large Language Models (LLMs) have demonstrated their effectiveness in various software development tasks. However, GitHub workflows differ from regular programs in both structure and semantics. We perform the first comprehensive study to understand the effectiveness of Large Language Models (LLMs) on five workflow-related tasks with different levels of prompts. We curated a set of ∼ 400K workflows and generated prompts with varying detail. We also fine-tuned LLMs on GitHub workflow tasks. Our evaluation of three state-of-the-art LLMs and their fine-tuned variants revealed various interesting findings on the current effectiveness and drawbacks of LLMs.

Incorrect handling of Software Application Programming Interfaces (APIs) errors results in bugs or security vulnerabilities that are hard to trigger during regular testing. Most of the existing techniques to detect such errors are based on static analysis and fail to identify certain cases where API return values are incorrectly handled. Furthermore, most of these techniques suffer from a very high false positive rate (≥50%), raising concerns regarding their practical use. We propose a dynamic analysis approach to detect API error handling bugs based on coverage-guided software fault injection. Specifically, we inject faults into APIs and observe how a program handles them. Our fault injection mechanism is generic and targeted to explore a given program's error handling behavior effectively. We avoid false positives by proactively filtering out crashes caused by infeasible faults. We implemented our technique in an automated pipeline called FuzzERR and applied it to 20 different programs spanning 444 APIs. Our evaluation shows that FuzzERR found 31 new and previously unknown bugs resulting from incorrect handling of API errors. Moreover, a comparative evaluation showed that FuzzERR significantly outperformed the state-of-the-art tools.