Architectural knowledge (AK) of existing systems is essential for software engineers to make design decisions. Recently, Large Language Models (LLMs) trained on large-scale datasets, including software repositories, have shown promise in embedding knowledge and answering questions. However, LLMs have not been evaluated for their abilities to answer questions about AK, leaving doubts about their accuracy. This paper assesses GPT, a leading LLM, by evaluating its responses’ accuracy, quality, and trustworthiness on the AK of the large-scale open-source system HDFS. We conducted an exploratory case study with 14 software engineers who posed questions to GPT and compared its responses to a predefined ground truth. The engineers rated GPT’s answers with moderate quality and trustworthiness. Our findings on GPT’s accuracy indicates moderate recall but lower precision, especially in identifying quality attribute solutions and design rationales. These results suggest that while GPT and similar models can provide initial insights into AK, expert validation remains necessary for reliability. This study underscores LLMs’ potential and limitations to discover software AK.

Context: A growing number of researchers are investigating how Mining Software Repositories (MSR) approaches can support software architecture activities, such as architecture recovery, tactics identification, architectural smell detection, and others. However, as of today, it is difficult to have a clear view of existing research on MSR for software architecture. Objectives: The objective of this study is to identify, classify, and summarize the state-of-the-art MSR approaches applied to software architecture (MSR4SA). Methods: This study is designed according to the systematic mapping study research method. Specifically, out of 2442 potentially relevant studies, we systematically identify 151 primary studies where MSR approaches are applied to perform software architecture activities. Then, we rigorously extract relevant data from each primary study and synthesize the obtained results to produce a clear map of reasons for adopting MSR approaches to support architecting activities, used data sources, applied MSR techniques, and captured architectural information. Results: The major reasons to adopt MSR4SA techniques are about addressing industrial concerns like achieving quality attributes and minimizing practitioners’ efforts. Most MSR4SA studies support architectural analysis, while architectural synthesis and evaluation are not commonly supported in MSR4SA studies. The most frequently mined data sources are source code repositories and issue trackers, which are also commonly mined together. Most of the MSR4SA studies apply more than one mining technique, where the most common MSR techniques are: (source code analysis, model analysis, statistical analysis), (machine learning, NLP). Architectural quality issues and components are the mostly mined type of information. Conclusion: Our results give a solid foundation for researchers and practitioners towards future research and applications of MSR approaches for software architecture.

Software engineers share their architectural knowledge (AK) in different places on the Web. Recent studies show that architectural blogs contain the most relevant AK, which can help software engineers to make design steps. Nevertheless, we know little about blogs, and specifically architectural blogs, where software engineers share their AK. In this paper, we conduct an exploratory study on architectural blogs to explore their types, topics, and their AK. Moreover, we determine the relevance of architectural blogs to make design steps. Our results support researchers and practitioners to find and re-use AK from blogs.

Technical debt, specifically Self-Admitted Technical Debt (SATD), remains a significant challenge for software developers and managers due to its potential to adversely affect long-term software maintainability. Although various approaches exist to identify SATD, tools for its comprehensive management are notably lacking. This paper presents DebtViz, an innovative SATD tool designed to automatically detect, classify, visualize and monitor various types of SATD in source code comments and issue tracking systems. DebtViz employs a Convolutional Neural Network-based approach for detection and a deconvolution technique for keyword extraction. The tool is structured into a back-end service for data collection and pre-processing, a SATD classifier for data categorization, and a front-end module for user interaction. DebtViz not only makes the management of SATD more efficient but also provides in-depth insights into the state of SATD within software systems, fostering informed decision-making on managing it. The scalability and deployability of DebtViz also make it a practical tool for both developers and managers in diverse software development environments. The source code of DebtViz is available at https://github.com/yikun-li/visdom-satd-management-system and the demo of DebtViz is at https://youtu.be/QXH6Bj0HQew.

Self-Admitted Technical Debt or SATD can be found in various sources, such as source code comments, commit messages, issue tracking systems, and pull requests. Previous research has established the existence of relations between SATD items in different sources; such relations can be useful for investigating and improving SATD management. However, there is currently a lack of approaches for automatically detecting these SATD relations. To address this, we proposed and evaluated approaches for automatically identifying SATD relations across different sources. Our findings show that our approach outperforms baseline approaches by a large margin, achieving an average F1-score of 0.829 in identifying relations between SATD items. Moreover, we explored the characteristics of SATD relations in 103 open-source projects and describe nine major cases in which related SATD is documented in a second source, and give a quantitative overview of 26 kinds of relations.

Abstract Technical debt refers to taking shortcuts to achieve short-term goals while sacrificing the long-term maintainability and evolvability of software systems. A large part of technical debt is explicitly reported by the developers themselves; this is commonly referred to as Self-Admitted Technical Debt or SATD. Previous work has focused on identifying SATD from source code comments and issue trackers. However, there are no approaches available for automatically identifying SATD from other sources such as commit messages and pull requests, or by combining multiple sources. Therefore, we propose and evaluate an approach for automated SATD identification that integrates four sources: source code comments, commit messages, pull requests, and issue tracking systems. Our findings show that our approach outperforms baseline approaches and achieves an average F1-score of 0.611 when detecting four types of SATD (i.e., code/design debt, requirement debt, documentation debt, and test debt) from the four aforementioned sources. Thereafter, we analyze 23.6M code comments, 1.3M commit messages, 3.7M issue sections, and 1.7M pull request sections to characterize SATD in 103 open-source projects. Furthermore, we investigate the SATD keywords and relations between SATD in different sources. The findings indicate, among others, that: 1) SATD is evenly spread among all sources; 2) issues and pull requests are the two most similar sources regarding the number of shared SATD keywords, followed by commit messages, and then followed by code comments; 3) there are four kinds of relations between SATD items in the different sources.

Technical debt denotes shortcuts taken during software development, mostly for the sake of expedience. When such shortcuts are admitted explicitly by developers (e.g., writing a TODO/Fixme comment), they are termed asSelf-Admitted Technical DebtorSATD. There has been a fair amount of work studying SATD management in Open Source projects, but SATD in industry is relatively unexplored. At the same time, there is no work focusing on developers’ perspectives towards SATD and its management. To address this, we conducted an exploratory case study in cooperation with an industrial partner to study how they think of SATD and how they manage it. Specifically, we collected data by identifying and characterizing SATD in different sources (issues, source code comments, and commits) and carried out a series of interviews with 12 software practitioners. The results show: 1) the core characteristics of SATD in industrial projects; 2) developers’ attitudes towards identified SATD and statistics; 3) triggers for practitioners to introduce and repay SATD; 4) relations between SATD in different sources; 5) practices used to manage SATD; 6) challenges and tooling ideas for SATD management.

The phenomenon of architecture erosion can negatively impact the maintenance and evolution of software systems, and manifest in a variety of symptoms during software development. While erosion is often considered rather late, its symptoms can act as early warnings to software developers, if detected in time. In addition to static source code analysis, code reviews can be a source of detecting erosion symptoms and subsequently taking action. In this study, we investigate the erosion symptoms discussed in code reviews, as well as their trends, and the actions taken by developers. Specifically, we conducted an empirical study with the two most active Open Source Software (OSS) projects in the OpenStack community (i.e., Nova and Neutron). We manually checked 21,274 code review comments retrieved by keyword search and random selection, and identified 502 code review comments (from 472 discussion threads) that discuss erosion. Our findings show that (1) the proportion of erosion symptoms is rather low, yet notable in code reviews and the most frequently identified erosion symptoms are architectural violation, duplicate functionality, and cyclic dependency; (2) the declining trend of the identified erosion symptoms in the two OSS projects indicates that the architecture tends to stabilize over time; and (3) most code reviews that identify erosion symptoms have a positive impact on removing erosion symptoms, but a few symptoms still remain and are ignored by developers. The results suggest that (1) code review provides a practical way to reduce erosion symptoms; and (2) analyzing the trend of erosion symptoms can help get an insight about the erosion status of software systems, and subsequently avoid the potential risk of architecture erosion.

Abstract Technical debt is a metaphor indicating sub-optimal solutions implemented for short-term benefits by sacrificing the long-term maintainability and evolvability of software. A special type of technical debt is explicitly admitted by software engineers (e.g. using a TODO comment); this is called Self-Admitted Technical Debt or SATD. Most work on automatically identifying SATD focuses on source code comments. In addition to source code comments, issue tracking systems have shown to be another rich source of SATD, but there are no approaches specifically for automatically identifying SATD in issues. In this paper, we first create a training dataset by collecting and manually analyzing 4,200 issues (that break down to 23,180 sections of issues) from seven open-source projects (i.e., Camel, Chromium, Gerrit, Hadoop, HBase, Impala, and Thrift) using two popular issue tracking systems (i.e., Jira and Google Monorail). We then propose and optimize an approach for automatically identifying SATD in issue tracking systems using machine learning. Our findings indicate that: 1) our approach outperforms baseline approaches by a wide margin with regard to the F1-score; 2) transferring knowledge from suitable datasets can improve the predictive performance of our approach; 3) extracted SATD keywords are intuitive and potentially indicating types and indicators of SATD; 4) projects using different issue tracking systems have less common SATD keywords compared to projects using the same issue tracking system; 5) a small amount of training data is needed to achieve good accuracy.

Abstract Architecture erosion (AEr) can adversely affect software development and has received significant attention in the last decade. However, there is an absence of a comprehensive understanding of the state of research about the reasons and consequences of AEr, and the countermeasures to address AEr. This work aims at systematically investigating, identifying, and analyzing the reasons, consequences, and ways of detecting and handling AEr. With 73 studies included, the main results are as follows: (1) AEr manifests not only through architectural violations and structural issues but also causing problems in software quality and during software evolution; (2) non‐technical reasons that cause AEr should receive the same attention as technical reasons, and practitioners should raise awareness of the grave consequences of AEr, thereby taking actions to tackle AEr‐related issues; (3) a spectrum of approaches, tools, and measures has been proposed and employed to detect and tackle AEr; and (4) three categories of difficulties and five categories of lessons learned on tackling AEr were identified. The results can provide researchers a comprehensive understanding of AEr and help practitioners handle AEr and improve the sustainability of their architecture. More empirical studies are required to investigate the practices of detecting and addressing AEr in industrial settings.

Software engineers need relevant and up-to-date architectural knowledge (AK), in order to make well-founded design decisions. However, finding such AK is quite challenging. One pragmatic approach is to search for AK on the web using traditional search engines (e.g. Google); this is common practice among software engineers. Still, we know very little about what AK is retrieved, from where, and how useful it is. In this paper, we conduct an empirical study with 53 software engineers, who used Google to make design decisions using the Attribute-Driven-Design method. Based on how the subjects assessed the nature and relevance of the retrieved results, we determined how effective web search engines are to find relevant architectural information. Moreover, we identified the different sources of AK on the web and their associated AK concepts.

As software systems evolve, their architecture is meant to adapt accordingly by following the changes in requirements, the environment, and the implementation. However, in practice, the evolving system often deviates from the architecture, causing severe consequences to system maintenance and evolution. This phenomenon of architecture erosion has been studied extensively in research, but not yet been examined from the point of view of developers. In this exploratory study, we look into how developers perceive the notion of architecture erosion, its causes and consequences, as well as tools and practices to identify and control architecture erosion. To this end, we searched through several popular online developer communities for collecting data of discussions related to architecture erosion. Besides, we identified developers involved in these discussions and conducted a survey with 10 participants and held interviews with 4 participants. Our findings show that: (1) developers either focus on the structural manifestation of architecture erosion or on its effect on run-time qualities, maintenance and evolution; (2) alongside technical factors, architecture erosion is caused to a large extent by non-technical factors; (3) despite the lack of dedicated tools for detecting architecture erosion, developers usually identify erosion through a number of symptoms; and (4) there are effective measures that can help to alleviate the impact of architecture erosion.

During software development, some architectural design decisions incur technical debt, either deliberately or inadvertently. These have serious impact on the quality of a software system, and can cost significant time and effort to be changed. While current research efforts have explored general concepts of architectural design decisions and technical debt separately, debt-incurring architectural design decisions have not been specifically explored in practice. In this case study, we explore debt-incurring architectural design decisions (DADDs) in practice. Specifically, we explore the main types of DADDs, why and how they are incurred in a software system, and how practitioners deal with these types of design decisions. We performed interviews and a focus group with practitioners working in embedded and enterprise software companies, discussing their concrete experience with such architectural design decisions. We provide the following contributions: 1) A categorization for the types of DADDs, which extend a current ontology on architectural design decisions. 2) A process on how deliberate DADDs are made in practice. 3) A conceptual model which shows the relationships between the causes and triggers of inadvertent DADDs. 4) The main factors that influence the way of dealing with DADDs. The results can support the development of new approaches and tools for Architecture Technical Debt management from the perspective of Design Decisions. Moreover, they support future research to capture architecture knowledge related to DADDs.

Technical debt refers to taking shortcuts to achieve short-term goals, which might negatively influence software maintenance in the long-term. There is increasing attention on technical debt that is admitted by developers in source code comments (termed as self-admitted technical debt or SATD). But SATD in issue trackers is relatively unexplored. We performed a case study, where we manually examined 500 issues from two open source projects (i.e. Hadoop and Camel), which contained 152 SATD items. We found that: 1) eight types of technical debt are identified in issues, namely architecture, build, code, defect, design, documentation, requirement, and test debt; 2) developers identify technical debt in issues in three different points in time, and a small part is identified by its creators; 3) the majority of technical debt is paid off, 4) mostly by those who identified it or created it; 5) the median time and average time to repay technical debt are 872.3 and 25.0 hours respectively.

To make good architecture design decisions, developers need to know suitable architecture solution alternatives. However, with the rapid and continuous increase of solution alternatives (e.g. technologies, patterns, tactics), it is challenging to acquire architecture knowledge and to ensure that this knowledge is up to date. Our goal is to improve how architects search for architecturally relevant information in online developer communities. We developed a new search approach for architecturally relevant information using Stack Overflow as an example of an online developer community. Our search approach differs from a conventional keyword-based search in that it considers semantic information of architecturally relevant concepts in Stack Overflow. We also implemented the search approach as a web-based search engine. To show the effectiveness of the search approach compared to a conventional keyword-based search, we conducted an experiment with 16 practitioners. To ensure realism of the experiment, tasks given to practitioners are based on real scenarios identified in interviews with a different set of practitioners. The experiment showed that the new search approach outperforms a conventional keyword based search.

Software architecting is a knowledge-intensive activity. However, obtaining and evaluating the quality of relevant and reusable knowledge (and ensuring that this knowledge is up-to-date) requires significant effort. In this paper, we explore how online developer communities (e.g., Stack Overflow), traditionally used by developers to solve coding problems, can help solve architectural problems. We develop an ontology that covers architectural knowledge concepts in Stack Overflow. The ontology provides a description of architecture-relevant information to represent and structure architectural knowledge in Stack Overflow. The ontology is empirically grounded through qualitative analyses of different Stack Overflow posts, as well as inter-coder reliability tests. Our results show that the architecture knowledge ontology in Stack Overflow captures architecture-relevant information and supports achieving practitioners' requirements and concerns.

Architectural decisions have a big influence on basic properties of a software system, and are difficult to change once implemented. Technology decisions deserve special attention because they are one of the most frequently occurring types of architectural decisions. Architectural knowledge is crucial for good decisions. Current architecture knowledge management approaches try to support architects by offering a rich base of architectural solutions and design decision rules. However, they mostly depend on manually capturing and maintaining the architectural knowledge. In this paper, we utilize the most popular online software development community (StackOverflow) as a source of knowledge for technology decisions to support architecture knowledge management approaches with a more efficient methods for knowledge capturing. We conducted an exploratory study, and followed a qualitative and quantitative content analysis approach. We analysed the posts in this community to identify architecture-relevant and technology-related knowledge, and to classify the posts into different types for the purpose of knowledge structuring. In addition, we evaluated our findings through feedback from practitioners.

Decision-making is at the core of software architecture design. However, in order for the architect to take the right design decisions, assistance is required for exploring the architectural knowledge, which encompasses the various architectural solutions, their relationships and distinctions. In the past decades, the number of available technology options has increased significantly, while existing architecture knowledge approaches support technology decisions by representing relations between the different technology solutions, as well as design problems. However, they do not differentiate the candidate technologies according to their offered qualities and drawbacks. Our main goal in this exploratory study is to understand how technology solutions are being considered by the architects during the design process, and how can we enhance existing architecture knowledge concepts to support technology decision making. Our contribution in this paper is differentiating the different technology solutions' features based on a set of architecturally significant aspects, to facilitate considering technologies during the architecture design decisions. In addition, we proposed an extension for existing architecture knowledge models, which characterise the technology design decisions, and their reasoning. We evaluated our results through real examples from practitioners. Moreover, we conducted interviews with experts to validate our proposed concepts.