As Large Language Models (LLMs) evolve in understanding and generating code, accurately evaluating their reliability in analyzing source code vulnerabilities becomes in-creasingly vital. While studies have examined LLM capabilities in tasks like vulnerability detection and repair, they often over-look the importance of both structure and semantic reasoning crucial for trustworthy vulnerability analysis. To address this gap, we introduce SV-TRUSTEVAL-C, a benchmark designed to evaluate LLMs' abilities for vulnerability analysis of code written in the C programming language through two key di-mensions: structure reasoning-assessing how models identify relationships between code elements under varying data and control flow complexities; and semantic reasoning-examining their logical consistency in scenarios where code is structurally and semantically perturbed. Our results show that current LLMs are far from satisfactory in understanding complex code relationships and that their vulnerability analyses rely more on pattern matching than on robust logical reasoning. These findings underscore the effectiveness of the SV-TRUSTEVAL-C benchmark and highlight critical areas for enhancing the reasoning capabilities and trustworthiness of LLMs in real-world vulnerability analysis tasks. Our ini-tial benchmark dataset is available at https://huggingface.co/datasets/LLMs4CodeSecurity/SV-TrustEval-C-1.0

The Software Assurance Metrics and Tool Evaluation (SAMATE) project at the National Institute of Standards and Technology (NIST) has created the Software Assurance Reference Dataset (SARD) to provide researchers and software security assurance tool developers with a set of known security flaws. As part of an empirical evaluation of a runtime monitoring framework, two test suites were executed and monitored, revealing deficiencies which led to a collaboration with the NIST SAMATE team to provide replacements. Test Suites 45 and 46 are analyzed, discussed, and updated to improve accuracy, consistency, preciseness, and automation. Empirical results show metrics such as recall, precision, and F-Measure are all impacted by invalid base assumptions regarding the test suites.

Monitors have been used for real-time systems to ensure proper behavior; however, most approaches do not allow for the addition of relevant fields required to identify and react to security vulnerabilities. Contracts can provide a useful mechanism for identifying and tracking vulnerabilities. Currently, contracts have been proposed for reliability and formal verification; yet, their use in security is limited. Static analysis methods are able to identify many known vulnerabilities; however, they suffer from a high rate of false-positives. The creation of a mechanism that can verify identified vulnerabilities is therefore warranted. We propose a contract-based security assertion monitoring framework (CB SAMF) for reducing the number of security vulnerabilities that are exploitable. CB SAMF will span multiple software layers and be used in an enhanced systems development life cycle (SDLC) including service-oriented analysis and design (SOAD).

Software systems today often run in malicious environments in which attacks or intrusions are quite common. This situation has brought security concerns into the development of software systems. Generally, software services are expected not only to satisfy functional requirements but also to be resistant to malicious attacks. Software attackability is defined as the likelihood that an attack on a software system will succeed. In this paper, we present a service-oriented framework to analyze attackability of software systems. More specifically, we propose a User System Interaction Effect (USIE) model that can be used systematically to derive and analyze security concerns from service-oriented software architectures. Many aspects of the model derivation and analysis can be automated, which limit the amount of user involvement, and thereby reduce the subjectivity underlying typical security risk analysis process. The model can be used as a foundation for quantitative analysis of software services from different security perspectives.