Abstract The choice of one’s programming language and relative ecosystem of libraries can affect the likelihood of encountering a critical vulnerability. Simply counting the vulnerabilities by mining a software repository is not enough, and case-control studies are a well-accepted methodology to determine relative risk. Yet, they require the ability to compare ‘equals with equals’ as a library for text processing is likely subject to less security scrutiny than a library for web applications. To compare libraries, we implemented a human-guided protocol to transfer classification categories from an ecosystem to libraries of another ecosystem. By building of this categorization, we performed a case-control study with the vulnerabilities available on Snyk and with status ’reviewed’ in the Github security Advisories till 2024. We mapped 76 Java/Maven libraries and 221 Python/PyPI packages as ’cases’ (libraries with vulnerabilities with a CVSS critical score) compared them against 58 Java/Maven and 166 Python/PyPI ’controls’ (Only with a high CVSS score). We found and overall the odds ratio of ending with a critical vulnerability is slightly higher when using a Java/Maven library in comparison to using a Python/PyPi package (1.13x). We refine the analysis to understand possible reasons for our result by using the CVSS vector metric. A possible explanation is that a vulnerability with low attack complexity has disproportionately higher chances to be critical in Java/Maven (38.9x) than in Python/PyPI (5.9x). Such results might be explained by the lack of past security interest in the Python ecosystem. By using the introduction of the OWASP dependency checker in 2023 for Python as possible indication of community interest, we found a risk reversal: after 2023 the risk of ending with a critical vulnerability (as opposed to just a high severity one) is significantly higher (2.4x) for a Python/PyPI package than for a Java/Maven library. To allow replication and updates, we make the dataset and the protocol individual steps available as open data.

Reusable software libraries, frameworks, and components, such as those provided by open source ecosystems and third-party suppliers, accelerate digital innovation. However, recent years have shown almost exponential growth in attackers leveraging these software artifacts to launch software supply chain attacks. Past well-known software supply chain attacks include the SolarWinds, log4j, and xz utils incidents. Supply chain attacks are considered to have three major attack vectors: through vulnerabilities and malware accidentally or intentionally injected into open source and third-party dependencies/components/containers ; by infiltrating the build infrastructure during the build and deployment processes; and through targeted techniques aimed at the humans involved in software development, such as through social engineering. Plummeting trust in the software supply chain could decelerate digital innovation if the software industry reduces its use of open source and third-party artifacts to reduce risks. This article contains perspectives and knowledge obtained from intentional outreach with practitioners to understand their practical challenges and from extensive research efforts. We then provide an overview of current research efforts to secure the software supply chain. Finally, we propose a future research agenda to close software supply chain attack vectors and support the software industry.

[Context:] Patch commits are useful to complete vulnerability datasets for training ML models and for developers to find a safe version for their dependencies. [Objective:] However, there is a gap in the state-of-the-art (SOTA) for a lightweight low False Positive patch commit finder. [Method:] We implemented Hash4Patch, a new tool to be used along with a current SOTA patch finder. We then validated it with a dataset of 160 CVEs. [Results:] Our approach significantly reduced the False Positives produced by a state-of-the-art tool with only 1 minute of additional running time on average. [Conclusions:] Our tool is able to effectively and efficiently reduce the number of alerts found by other patch commit finders, thus minimizing the manual effort needed by developers.

Abstract Security vulnerability fixes could be a promising research avenue for Automated Program Repair (APR) techniques. In recent years, APR tools have been thoroughly developed for fixing generic bugs. However, the area is still relatively unexplored when it comes to fixing security bugs or vulnerabilities. In this paper, we evaluate nine state-of-the-art APR tools and one vulnerability-specific repair tool. In particular, we investigate their ability to generate patches for 79 real-world Java vulnerabilities in the Vul4J dataset, as well as the level of trustworthiness of these patches. We evaluate the tools with respect to their ability to generate security patches that are (i) testable, (ii) having the positive effect of closing the vulnerability, and (iii) not having side effects from a functional point of view. Our results show that the evaluated APR tools were able to generate testable patches for around 20% of the considered vulnerabilities. On average, nearly 73% of the testable patches indeed eliminate the vulnerabilities, but only 44% of them could actually fix security bugs while maintaining the functionalities. To understand the root cause of this phenomenon, we conduct a detailed comparative study of the general bug fix patterns in Defect4J and the vulnerability fix patterns in ExtraVul (which we extend from Vul4J). Our investigation shows that, although security patches are short in terms of lines of code, they contain unique characteristics in their fix patterns compared to general bugs. For example, many security fixes require adding method calls. These method calls contain specific input validation-related keywords, such asencode,normalize, andtrim. In this regard, our study suggests that additional repair patterns should be implemented for existing APR tools to fix more types of security vulnerabilities.

Abstract Objective We investigated whether (possibly wrong) security patches suggested by Automated Program Repairs (APR) for real world projects are recognized by human reviewers. We also investigated whether knowing that a patch was produced by an allegedly specialized tool does change the decision of human reviewers. Method We perform an experiment with $$n= 72$$ n = 72 Master students in Computer Science. In the first phase, using a balanced design, we propose to human reviewers a combination of patches proposed by APR tools for different vulnerabilities and ask reviewers to adopt or reject the proposed patches. In the second phase, we tell participants that some of the proposed patches were generated by security-specialized tools (even if the tool was actually a ‘normal’ APR tool) and measure whether the human reviewers would change their decision to adopt or reject a patch. Results It is easier to identify wrong patches than correct patches, and correct patches are not confused with partially correct patches. Also patches from APR Security tools are adopted more often than patches suggested by generic APR tools but there is not enough evidence to verify if ‘bogus’ security claims are distinguishable from ‘true security’ claims. Finally, the number of switches to the patches suggested by security tool is significantly higher after the security information is revealed irrespective of correctness. Limitations The experiment was conducted in an academic setting, and focused on a limited sample of popular APR tools and popular vulnerability types.

Context: Experimentation in Software and Security Engineering is a common research practice, in particular with human subjects.Problem: The combinatorial nature of software configurations and the difficulty of recruiting experienced subjects or running complex and expensive experiments make the use of full factorial experiments unfeasible to obtain statistically significant results.Contribution: Provide comprehensive alternative Designs of Experiments (DoE) based on orthogonal designs or crossover designs that provably meet desired requirements such as balanced pair-wise configurations or balanced ordering of scenarios to mitigate bias or learning effects.We also discuss and formalize the statistical implications of these design choices, in particular for crossover designs.Artifact: We made available the algorithmic construction of the design for 𝓁 = 2, 3, 4, 5 levels for arbitrary 𝐾 factors and illustrated their use with examples from security and software engineering research.

Abstract Context: Technical leverage is the ratio between dependencies (other people’s code) and own codes of a software package. It has been shown to be useful to characterize the Java ecosystem and there are also studies on the NPM ecosystem available. Objective: By using this metric we aim to analyze the Python ecosystem, how it evolves, and how secure it is, as a developer would perceive it when deciding to adopt or update (or not) a library. Method: We collect a dataset of the top 600 Python packages (corresponding to 21,205 versions) and used a number of innovative approaches for its analysis including the use of a two-part statistical model to deal with excess zeros, a mathematical closed formulation to estimate vulnerabilities that we confirm with bootstrapping on the actual dataset. Results: Small Python package versions have a median technical leverage of 6.9x their own code, while bigger package versions rely on dependencies code a tenth of their own (median leverage of 0.1). In terms of evolution, Python packages tend to have stable technical leverage through their evolution (once highly leveraged, always leveraged). On security, the chance of getting a safe package version when choosing a package is actually better than previous research has shown based on the ratio of safe package versions in the ecosystem. Coclusions: Python packages ship a lot of other people’s code and tend to keep doing so. However, developers will have a good chance to choose a safe package version.

Program slicing has been used to semi- or fully-automatically help developers find errors and vulnerabilities in their programs. For example, Dashevskyi et al. (IEEE TSE 2018) introduced a lightweight slicer for Java that can be used for vulnerability analysis. However, a similar lightweight slicer for C/C++ is still missing. In this work we propose a comparison method for parsers, evaluate it on two commonly-used parsers, and develop a lightweight slicer for C/C++ using the “better” parser from our comparison. From our evaluation, the Joern parsing method (island grammar) could parse non-standard C/C++ code but its resulting structure may contain semantic errors that can affect subsequent analysis. ANTLR4 is faster in returning a result, and when manually cleared of non-standard C/C++ codes, it is more accurate than Joern. We then built our C/C++ thin slicer extension using ANTLR4, and we observed that it is promising from both precision and performance perspectives. As a future work, we plan to improve the logic behind processing pointers. In particular, we consider doing deeper pointer analysis.

Open-source packages typically have their source code available on a source code repository (e.g., on GitHub), but developers prefer to use pre-built artifacts directly from the package repositories (such as npm for JavaScript). Between the source code and the distributed artifacts, there could be differences that pose security risks (e.g., attackers deploy malicious code during package installation) in the software supply chain. Existing package scanners focus on the entire artifact of a package to detect this kind of attacks. These procedures are not only time consuming, but also generate high irrelevant alerts (FPs). An approach called LastPyMile by Vu et al. (ESEC/FSE’21) has been shown to be effective in detecting discrepancies and reducing false alerts in vetting Python packages on PyPI by focusing only on the differences between the source and the package. In this work, we propose to port that approach to scan JavaScript packages in the npm ecosystem. We presented a preliminary evaluation of our implementation on a set of real malicious npm packages and the top popular packages. The results show that while being 20.7x faster than git-log approach, our approach managed to reduce the percentage of false alerts produced by package scanner by 69%.