The npm software package ecosystem is a notable target for adversarial actors, who seek to compromise software dependencies to exploit software developers and the end-users of their software. One especially dangerous form of attack involves the compromise of a package update. By sneaking malicious code into a package update, adversaries can trick package users into unknowingly installing malware. Detecting malicious package updates is an active research problem, as prospective solutions need to keep pace with the near-constant stream of new package updates, while also maintaining high detection accuracy. In this context, one potentially interesting and emergent approach involves utilizing large language models (LLMs) to identify malicious behaviors from the text of package code. However, practical use of LLMs also poses unique first-order challenges, as models are expensive to run and are known to struggle with task performance as input size increases. This work provides a critical exploration into the practicality and effectiveness of LLMs for detecting malicious package updates. We overcome the immediate challenges for LLM-based applications by preprocessing inputs for analysis and post-processing outputs for malware classification. We find this approach to be practical at repository scale and effective at detecting historical malware incidents, with our best-performing model correctly flagging 209 out of 209 malicious samples across a collection of historical attacks, while only flagging 8 out of 2,000 benign samples across a dataset of typical package updates. With first-order obstacles overcome, we then conduct a deeper investigation into the reasoning capabilities of LLMs–demonstrating specific mild code obfuscations that uniquely challenge tested LLMs and enable adaptive adversaries to subvert detection. Ultimately, our findings demonstrate nuanced potential for employing LLMs as a part of a larger security tool-belt for detecting package malware.

The npm software ecosystem allows developers to easily import code written by others. However, manual vetting of every individual installed component is made difficult in many cases by the number of transitive dependencies brought in by installing popular packages. This has enabled attackers to propagate malicious code by hiding it deep into the dependency chains of popular packages. A particularly dangerous form of attack comes from malicious code embedded into package install scripts.

This work presents findings and mitigations on an understudied issue, which we term shrinkwrapped clones, that is endemic to the npm software package ecosystem. A shrink-wrapped clone is a package which duplicates, or near-duplicates, the code of another package without any indication or reference to the original package. This phenomenon represents a challenge to the hygiene of package ecosystems, as a clone package may siphon interest from the package being cloned, or create hidden duplicates of vulnerable, insecure code which can fly under the radar of audit processes.