Background . Software vulnerabilities are increasing in complexity and scale, posing great security risks to many software systems. Extracting information about software vulnerabilities is a critical area of research that aims to identify and create a structured representation of vulnerability-related information. These structured data help software systems better understand vulnerabilities and provide security professionals with timely information to mitigate the impact of rapidly growing vulnerabilities while guiding future research to develop more secure systems. However, this process relies on the effectiveness of information extraction to transform manual vulnerability analysis from security experts to digital solutions. Despite its importance, the unique nature of vulnerability information and the fast pace at which machine learning-based extraction methods and techniques have evolved make it challenging to assess the current successes, failures, challenges, and opportunities within this research area. This study presents a systematic literature review aimed at clarifying this complex landscape. Methods . In this study, we conduct a systematic literature review (SLR) to explore existing research focusing on extracting information about software security vulnerabilities. We search for 829 primary studies on security vulnerability information extraction from seven widely used online digital libraries, focusing on top peer-reviewed journals and conferences published between 2001 and 2024. After applying our inclusion and exclusion criteria and the snowballing technique, we narrowed our selection to 87 studies for in-depth analysis and addressed four main research questions. We collect qualitative and quantitative data from each study, identifying 34 components such as research problems, methods, contributions, evaluation metrics, results, types of extracted vulnerability information, challenges, and limitations. We use meta-analysis, statistical machine learning, and text-mining techniques to identify themes, patterns, and trends across the primary studies and visualize findings. Result : The study provides an overview of the security vulnerability data landscape, identifies key resources, and guides efforts to improve vulnerability information extraction and analysis. The study finds a diverse landscape of learning algorithms used in security vulnerability information extraction, with Bidirectional Encoder Representations from Transformers (BERT), Long Short-term Memory (LSTM), and Support Vector Machine (SVM) being the most dominant. The study identifies key challenges, including feature engineering complexity, lack of a gold-standard corpus, preprocessing errors, generating accurate training data, addressing imbalanced data, multimodality fusion, and graph sparsity in security knowledge graphs. Insights for Future Research Directions . The study underscores the need for advanced extraction approaches, robust datasets, automated annotation methods, and advanced machine learning algorithms to improve the extraction of security vulnerability information. This study also suggests using large language models (LLMs) and transformer models to facilitate the automatic extraction of security-related words, terms, concepts, and phrases and introduce new filtering parameters for user requirements. We provide all our implementations; it can be found at https://bitbucket.org/slr-svie/vulnerability-information-extraction/src/master/ .

Large language models (LLMs) have emerged as a promising tool for detecting code vulnerabilities, potentially offering advantages over traditional rule-based methods. This paper proposes an enhanced framework for vulnerability detection using LLMs, incorporating various prompt engineering strategies to improve performance. We evaluate several techniques, including role-based prompting, zero-shot chain-of-thought, and structured prompting approaches, on the DiverseVul dataset of C/C++ vulnerabilities. Our experiments assess the framework’s performance across different code structures, contextual information levels, and LLM capabilities. Our results show that using our dynamic prompt engineering technique, you can improve the F1 score by up to 100% with GPT-3.5, a widely used LLM model. We also observe that GPT-4o, Gemini 2.0 Flash, and Meta Llama 3.1 generally outperform GPT-3.5, and all models are very poor when it comes to correctly identifying the type of vulnerability in the code, with the best F1 score of 0.16 observed. However, our follow-up experiments on LLM-based vulnerability correction (i.e., patching) show a 45.77% success rate using GPT-4o, demonstrating promising results in leveraging LLMs for enhancing software security and providing insights into optimizing prompt engineering for vulnerability detection tasks.

Utilizing code snippets on Stack Overflow (SO) is a common practice among developers for problem-solving. Although SO code snippets serve as valuable resources, it is important to acknowledge their imperfections, reusing problematic code snippets can lead to the introduction of suboptimal or buggy code into software projects. SO comments often point out weaknesses of a post and provide valuable insights to improve the quality of answers, while SO comments are usually missed and/or ignored, leaving these problematic code snippets untouched. In this work, we first investigate the task of automatic SO posts updating based on their associated comments. We introduce a novel framework, named SOUP (Stack Overflow Updator for Post) for this task. SOUP addresses two key tasks: Valid Comment-Edit Prediction (VCP) and Automatic Post Updating (APU). We fine-tuned a large language model, CodeLlama, using low-rank adaptation techniques to complete the VCP task, and constructed a dataset containing 78k valid comment-edit pairs for the APU task. Subsequently, we tested the performance of multiple large language models on the APU task. Extensive experimental results show the promising performance of our model over a set of benchmarks. Moreover, we also perform an in-the-wild evaluation on Stack Overflow, we submitted 50 edits generated by our approach to Stack Overflow posts and 21 of them have been verified and accepted by SO maintainers, further proving the practical value of SOUP.

Rumor suppression is targeted at diminishing the impact of false and negative information within social networks by decreasing the prevalence of belief in such rumors among individuals, utilizing diverse strategies. Previous studies have broadly delineated rumor suppression strategies into two primary categories: targeting key nodes or edges for obstruction, and enlisting high-influence nodes to disseminate truth-related accurate information. Traditionally, employing a singular strategy involves utilizing a static algorithm throughout the rumor suppression endeavor. This method, however, encounters difficulties in adapting to fluctuating external conditions, rendering it less efficacious in the management of rumor proliferation. In response to these challenges, we introduce the concept of Adaptive Rumor Suppression (ARS), which aims to dynamically counter rumors by taking into account the nuances of propagation dynamics and the surrounding environmental context. We propose a multi-label state transition linear threshold model to more closely mirror the complex process of information diffusion across social networks. Furthermore, we advocate for a multi-round hybrid strategy that amalgamates blocking and clarification tactics to address the ARS problem within the confines of limited resource allocations. To navigate the complexities of ARS, we introduce the Hybrid Strategy of Each Round (HS-R) algorithm, which synergizes multiple strategies to effectively counter the spread of rumors. In extension, we present the Multi-Round Multi-Label (MRML) algorithm, designed to augment the efficiency of the HS-R algorithm. Experimental evaluations conducted on authentic social network datasets illustrate that our methodologies significantly outshine baseline algorithms, offering a more effective and adaptable solution to curb rumor propagation across varied environments.

With the development of blockchain technology, smart contracts have become an important component of blockchain applications. Despite their crucial role, the development of smart contracts may introduce vulnerabilities and potentially lead to severe consequences, such as financial losses. Meanwhile, large language models, represented by ChatGPT, have gained great attention, showcasing great capabilities in code analysis tasks. In this article, we presented an empirical study to investigate the performance of ChatGPT in identifying smart contract vulnerabilities. Initially, we evaluated ChatGPT’s effectiveness using a publicly available smart contract dataset. Our findings discover that while ChatGPT achieves a high recall rate, its precision in pinpointing smart contract vulnerabilities is limited. Furthermore, ChatGPT’s performance varies when detecting different vulnerability types. We delved into the root causes for the false positives generated by ChatGPT, and categorized them into four groups. Second, by comparing ChatGPT with other state-of-the-art smart contract vulnerability detection tools, we found that ChatGPT’s F-score is lower than others for 3 out of the 7 vulnerabilities. In the case of the remaining 4 vulnerabilities, ChatGPT exhibits a slight advantage over these tools. Finally, we analyzed the limitation of ChatGPT in smart contract vulnerability detection, revealing that the robustness of ChatGPT in this field needs to be improved from two aspects: its uncertainty in answering questions; and the limited length of the detected code. In general, our research provides insights into the strengths and weaknesses of employing large language models, specifically ChatGPT, for the detection of smart contract vulnerabilities.

Web3, the next generation of the Internet, represents a decentralized and democratized web. Although it has garnered significant public interest and found numerous real-world applications, there is a limited understanding of people's perceptions and experiences with Web3. In this study, we conducted an empirical study to investigate the categories of Web3 applications and their popularity, as well as the potential challenges and opportunities within this emerging landscape. Our research was carried out in two phases. In the first phase, we analyzed 200 popular Web3 projects associated with 10 leading Web3 venture capital firms. In the second phase, we collected and examined code-related data from GitHub and market-related data from blockchain browsers (e.g., Etherscan) for these projects. Our analysis revealed that the Web3 ecosystem can be categorized into two groups, i.e., Web3 infrastructure and Web3 applications, with each consisting of several subcategories or subdomains. We also gained insights into the popularity of these Web3 projects at both the code and market levels and pointed out the challenges in the Web3 ecosystem at the system, developer, and user levels, as well as the opportunities it presents. Our findings contribute to a better understanding of Web3 for researchers and developers, which in turn promotes further exploration and advancement in this innovative field.

Web3 is leading a wave of the next generation of web services that even many Web2 applications are keen to ride. However, the lack of Web3 background for Web2 developers hinders easy and effective access and transition. On the other hand, Web3 applications desire encouragement and advertisement from conventional Web2 companies and projects due to their low market shares. In this article, we propose a seamless transition framework that transits Web2 to Web3, named WEBTTCOM [WEBTTCOM stands for Web2 (two)–Web3 (three) Communicator], after exploring the connotation of Web3 and the key differences betweenWeb2 andWeb3 applications.We also provide a full-stack implementation as a use case to support the proposed framework, followed by performance evaluation and surveys with ~1000 participants that show ~80% positive and ~20% neutral responses. We confirm that the proposed framework WEBTTCOM addresses the defined research question, and the implementation well satisfies the framework WEBTTCOM in terms of strong necessity,usability, andcompletenessbased on the survey results.

The increase of software supply chain threats has underscored the necessity for robust security mechanisms, among which the Software Bill of Materials (SBOM) stands out as a promising solution. SBOMs, by providing a machine-readable inventory of software composition details, play a crucial role in enhancing transparency and traceability within software supply chains. This empirical study delves into the practical challenges and solutions associated with the adoption of SBOMs through an analysis of 4,786 GitHub discussions across 510 SBOM-related projects. Through repository mining and analysis, this research delineates key topics, challenges, and solutions intrinsic to the effective utilization of SBOMs. Furthermore, we shed light on commonly used tools and frameworks for SBOM generation, exploring their respective strengths and limitations. This study underscores a set of findings, for example, there are four phases of the SBOM life cycle, and each phase has a set of SBOM development activities and issues; in addition, this study emphasizes the role SBOM play in ensuring resilient software development practices and the imperative of their widespread adoption and integration to bolster supply chain security. The insights of our study provide vital input for future work and practical advancements in this topic.

Software development is a collaborative process that involves various interactions among individuals and teams. TODO comments in source code play a critical role in managing and coordinating diverse tasks during this process. However, this study finds that a large proportion of open-source project TODO comments are left unresolved or take a long time to be resolved. About 46.7% of TODO comments in open-source repositories are of low-quality (e.g., TODOs that are ambiguous, lack information, or are useless to developers). This highlights the need for better TODO practices. In this study, we investigate four aspects regarding the quality of TODO comments in open-source projects: (1) the prevalence of low-quality TODO comments; (2) the key characteristics of high-quality TODO comments; (3) how are TODO comments of different quality managed in practice; and (4) the feasibility of automatically assessing TODO comment quality. Examining 2,863 TODO comments from Top100 GitHub Java repositories, we propose criteria to identify high-quality TODO comments and provide insights into their optimal composition. We discuss the lifecycle of TODO comments with varying quality. To assist developers, we construct deep learning-based methods that show promising performance in identifying the quality of TODO comments, potentially enhancing development efficiency and code quality.

Architectural design contexts contain a set of factors that greatly influence software application development. Among them, organizational design contexts consist of high-level company concerns and how it is structured, for example, stakeholders and development schedules heavily impacting design considerations. The Decentralized Autonomous Organization (DAO), as a vital concept in the Web3 space, represents an organization constructed by automatically executed rules, such as via smart contracts, holding features of the permissionless committee, transparent proposals, and fair contribution by participated stakeholders. In this work, we conduct a systematic literature review of existing DAO literature to summarize its structural features, benefits and challenges, and potential development directions in the context of Web3 applications.

The rapid growth of software supply chain attacks has attracted considerable attention to software bill of materials (SBOM). SBOMs are a crucial building block to ensure the transparency of software supply chains that helps improve software supply chain security. Although there are significant efforts from academia and industry to facilitate SBOM development, it is still unclear how practitioners perceive SBOMs and what are the challenges of adopting SBOMs in practice. Furthermore, existing SBOM-related studies tend to be ad-hoc and lack software engineering focuses. To bridge this gap, we conducted the first empirical study to interview and survey SBOM practitioners. We applied a mixed qualitative and quantitative method for gathering data from 17 interviewees and 65 survey respondents from 15 countries across five continents to understand how practitioners perceive the SBOM field. We summarized 26 statements and grouped them into three topics on SBOM's states of practice. Based on the study results, we derived a goal model and highlighted future directions where practitioners can put in their effort.

Being able to access software in daily life is vital for everyone, and thus accessibility is a fundamental challenge for software development. However, given the number of accessibility issues reported by many users, e.g., in app reviews, it is not clear if accessibility is widely integrated into current software projects and how software projects address accessibility issues. In this article, we report a study of the critical challenges and benefits of incorporating accessibility into software development and design. We applied a mixed qualitative and quantitative approach for gathering data from 15 interviews and 365 survey respondents from 26 countries across five continents to understand how practitioners perceive accessibility development and design in practice. We got 44 statements grouped into eight topics on accessibility from practitioners’ viewpoints and different software development stages. Our statistical analysis reveals substantial gaps between groups, e.g., practitioners have Direct vs. Indirect accessibility relevant work experience when they reviewed the summarized statements. These gaps might hinder the quality of accessibility development and design, and we use our findings to establish a set of guidelines to help practitioners be aware of accessibility challenges and benefit factors. We suggest development teams put accessibility as a first-class consideration throughout the software development process, and we also propose some remedies to resolve the gaps between groups and to highlight key future research directions to incorporate accessibility into software design and development.

Predictive models are one of the most important techniques that are widely applied in many areas of software engineering. There have been a large number of primary studies that apply predictive models and that present well-performed studies in various research domains, including software requirements, software design and development, testing and debugging, and software maintenance. This article is a first attempt to systematically organize knowledge in this area by surveying a body of 421 papers on predictive models published between 2009 and 2020. We describe the key models and approaches used, classify the different models, summarize the range of key application areas, and analyze research results. Based on our findings, we also propose a set of current challenges that still need to be addressed in future work and provide a proposed research road map for these opportunities.

The release note is one of the most important software artifacts that serves as a communication bridge between development teams and users. Release notes contain a set of crucial information, such as descriptions of enhancements, improvements, potential issues, development, evolution, testing, and maintenance of projects throughout the whole development life cycle. A comprehensive understanding of the characteristics of release notes and how to best document one for different targeted users would be highly beneficial. However, the release note is often neglected and has not to date been systematically investigated by researchers. In this paper, we conducted a descriptive case study to investigate release note production and usage in practice. We first performed a large scale empirical study of 32,425 release notes in 1,000 GitHub projects to understand the characteristics of real-world release notes, and eight categories of information identified that are normally documented in release notes. We then conducted interviews with 15 professionals and an online survey with 314 respondents to investigate their opinions on release notes in practice. Our results show that both release note producers and users consider that well-formed release notes impact software activities (e.g., software evolution) positively. We summarised 27 statements about release notes grouped into eight topics based on participants’ opinions. Our study uncovers significant discrepancies between release note producers and users in perceiving release notes. Based on these findings, we provide a set of release note production and usage guidelines for practitioners and highlight future research directions.

Accessibility design elements allow people to access software products and services independent of their different abilities. However, accessibility is challenging to handle and whether accessibility is widely considered in software projects is unclear. In this work, we aim to understand if accessibility is a prevalent consideration in practice, what accessibility issues are discussed in GitHub projects, what potential reasons cause accessibility issues, and what solutions (e.g., tools and standards) are applied for addressing accessibility issues. In this work, we collect 11,820 accessibility issues and their threads discussed by developers in popular GitHub projects. We manually analyzed and grouped the collected accessibility issues into seven categories. The results of our study uncover that accessibility is widely discussed in general projects, and the potential reasons that cause accessibility issues are because developers are not aware of the importance of accessibility and they lack knowledge about accessibility concerns, standards, and existing tools. Our results and findings can enhance and improve developers' knowledge and awareness when they conduct accessibility-relevant design or incorporate accessibility elements into their projects.

The use of architecture and design patterns have impacts on the quality attributes of a system, and the application of patterns depend on design contexts. There are complex interdependent relationships between them. In this study, we explore how developers use architecture and design patterns with respect to quality attributes concerns and design contexts. We extracted pattern related posts from Stack Overflow and analyzed the architecture discussions. Our study reveals what contexts and quality attributes developers consider when using architecture patterns, and we have identified new and previously unknown relationships between these design elements. These findings can enhance developers' knowledge when they design with architecture patterns, quality attributes, and design contexts.