Abstract Architectural uncertainties arising from incomplete or unclear information pose significant challenges when making architectural decisions in Agile teams. Based on a limited number of case studies that employed a technique called ArchHypo, four patterns were identified that propose small adjustments in the development process to handle architectural uncertainties: Protective Guideline , Bring the Specialist , Plan for Preparation , and Quality Checkpoint . Although the patterns derived from these experiences can be useful in real projects, their applicability and consequences were based on limited evidence and specific scenarios. To address this issue, this paper presents an interview study with experienced software architects and engineers to gather further information on the application of these patterns. The research method employed semi-structured interviews to gather the experiences of professionals with the target practices, and thematic analysis was used to assess their recurrence, applicability, and consequences. The findings confirmed that most professionals recognized those practices in real projects and their suitability as actions in uncertainty management. Moreover, new positive and negative consequences, not previously documented in the patterns, were identified. As a result, this work contributes to the field by providing guidance to professionals on how to better evaluate the trade-offs of those patterns when applied to architecture uncertainty management.

Multi-task learning is a paradigm that leverages information from related tasks to improve the performance of machine learning. Self-Admitted Technical Debt (SATD) are comments in the code that indicate not-quite-right code introduced for short-term needs, i.e., technical debt (TD). Previous research has provided evidence of a possible relationship between SATD and the existence of vulnerabilities in the code. In this work, we investigate if multi-task learning could leverage the information shared between SATD and vulnerabilities to improve the automatic detection of these issues. To this aim, we implemented VulSATD, a deep learner that detects vulnerable and SATD code based on CodeBERT, a pre-trained transformers model. We evaluated VulSATD on MADE-WIC, a fused dataset of functions annotated for TD (through SATD) and vulnerability. We compared the results using single and multi-task approaches, obtaining no significant differences even after employing a weighted loss. Our findings indicate the need for further investigation into the relationship between these two aspects of low-quality code. Specifically, it is possible that only a subset of technical debt is directly associated with security concerns. Therefore, the relationship between different types of technical debt and software vulnerabilities deserves future exploration and a deeper understanding.

Abstract Companies adopt agile methodologies for various reasons, primarily due to their adaptability to change and evolving business demands. In this context, addressing non-functional requirements (NFRs) may not always be a priority and can present challenges for agile teams. The focus on User Stories present in agile methods and tools often does not offer explicit alternatives for documenting NFRs. In this research, we perform a survey to explore five different strategies for documenting NFRs, to identify which fits better for different types of quality attributes and to understand the strengths and drawbacks of each one. As a result, the participants considered certain strategies as being more or less suitable for specifying different types of quality attributes. For instance, while Story Labeling was rarely recommended for security requirements, using Story Sub-sections or Verification Rules were highly recommended for this kind of quality attribute. Our results also evaluated the strategies considering several factors, such as the level of detail and requirement duplication. As a practical implication, the results of this work can provide guidance to agile development teams in choosing the most suitable alternative for each NFR documentation.

Abstract Generative AI has been rapidly adopted by the software development industry in various ways, offering innovative approaches to transforming requirements into working software. Combining Generative AI with Test-Driven Development (TDD) presents a creative method to accelerate this transformation. However, questions remain about ChatGPT’s readiness for this challenge, including the techniques and best practices required for success and the scenarios where this approach can consistently deliver results. To explore these questions, we designed a study where a group of master’s students performed programming assignments using TDD, first independently and then with the support of ChatGPT. The three assignments represent distinct scenarios: mathematical calculations (function), text processing (class), and system integration (class with dependencies). We performed a qualitative analysis of the submitted code and reports identifying key strategies that significantly influence success rates, such as providing contextual information, separating instructions in prompts following an iterative process, and assisting AI in fixing errors. Among the scenarios, the integration task achieved the highest performance. This study highlights the potential of leveraging Generative AI in TDD for software development and presents a list of effective strategies to maximize its impact. By applying these positive strategies and avoiding identified pitfalls, this research marks a step toward establishing best practices for integrating Generative AI with TDD in software engineering.

ABSTRACT Context Generative artificial intelligence (GenAI) tools have become increasingly prevalent in software development, offering assistance to various managerial and technical project activities. Notable examples of these tools include OpenAI's ChatGPT, GitHub Copilot, and Amazon CodeWhisperer. Objective Although many recent publications have explored and evaluated the application of GenAI, a comprehensive understanding of the current development, applications, limitations, and open challenges remains unclear to many. Particularly, we do not have an overall picture of the current state of GenAI technology in practical software engineering usage scenarios. Method We conducted a literature review and focus groups for a duration of five months to develop a research agenda on GenAI for software engineering. Results We identified 78 open research questions (RQs) in 11 areas of software engineering. Our results show that it is possible to explore the adoption of GenAI in partial automation and support decision‐making in all software development activities. While the current literature is skewed toward software implementation, quality assurance and software maintenance, other areas, such as requirements engineering, software design, and software engineering education, would need further research attention. Common considerations when implementing GenAI include industry‐level assessment, dependability and accuracy, data accessibility, transparency, and sustainability aspects associated with the technology. Conclusions GenAI is bringing significant changes to the field of software engineering. Nevertheless, the state of research on the topic still remains immature. We believe that this research agenda holds significance and practical value for informing both researchers and practitioners about current applications and guiding future research.

Uncertainty is present in software architecture decisions due to a lack of knowledge about the requirements and the solutions involved. However, this uncertainty is usually not made explicit, and decisions can be made based on unproven premises or false assumptions. This paper focuses on a technique called ArchHypo that uses hypotheses engineering to manage uncertainties related to software architecture. It proposes formulating a technical plan based on each hypothesis’ assessment, incorporating measures able to mitigate its impact and reduce uncertainty. To evaluate the proposed technique, this paper reports an application of the technique in a mission-critical project that faced several technical challenges. Conclusions were based on data extracted from the project documentation and a questionnaire answered by all team members. As a result, the application of ArchHypo provided a structured approach to dividing the architectural work through iterations, which facilitated architectural decision-making. However, further research is needed to fully understand its impact across different contexts. On the other hand, the team identified the learning curve and process adjustments required for ArchHypo's adoption as significant challenges that could hinder its widespread adoption. In conclusion, the evidence found in this study indicates that the technique has the potential to provide a suitable way to manage the uncertainties related to software architecture, facilitating the strategic postponement of decisions while addressing their potential impact.

In this paper, we present MADE-WIC, a large dataset of functions and their comments with multiple annotations for technical debt and code weaknesses leveraging different state-of-the-art approaches. It contains about 860K code functions and more than 2.7M related comments from 12 open-source projects. To the best of our knowledge, no such dataset is publicly available. MADE-WIC aims to provide researchers with a curated dataset on which to test and compare tools designed for the detection of code weaknesses and technical debt. As we have fused existing datasets, researchers have the possibility to evaluate the performance of their tools by also controlling the bias related to the annotation definition and dataset construction. The demonstration video can be retrieved at https://www.youtube.com/watch?v=GaQodPrcb6E.

Defining and designing a software product is not merely a technical endeavor, but also a socio-technical journey. As such, its success is associated with human-related aspects, such as the value users perceive. To handle this issue, the product manager role has become more evident in software-intensive companies. A unique, challenging context for these professionals is constituted by software startups, emerging companies developing novel solutions looking for sustainable and scalable business models. This study aims to describe the role of product managers in the context of software startups. We performed a Socio-Technical Grounded Theory study using data from blog posts and interviews. The results describe the product manager as a multidisciplinary, general role, not only guiding the product by developing its vision but also as a connector that emerges in a growing company, enabling communication of software development with other areas, mainly business and user experience. The professional performing this role has a background in one of these areas but a broad knowledge and understanding of key concepts of the other areas is needed. We also describe how differences of this role to other lead roles are perceived in practice. Our findings represent several implications for research, such as better understanding of the role transformation in growing software startups, practice, e.g., identifying the points to which a professional migrating to this role should pay attention, and the education of future software developers, by suggesting the inclusion of related topics in the education and training of future software engineers.

Context The COVID‐19 pandemic has had a disruptive impact on how people work and collaborate across all global economic sectors, including software business. While remote working is not new for software engineers, forced WFH situations come with both limitations and opportunities. As the ‘new normal’ for working might be based on the current state of Work‐from‐home (WFH), it is useful to understand what has happened and learn from that. Objective This study aims to gain insights into how their WFH arrangement impacts project management and software engineering. We are also interested in exploring these impacts in different contexts, such as startups and established companies. Method We conducted a global‐scale, cross‐sectional survey during the spring and summer 2021. Our results are based on quantitative and qualitative analysis of 297 valid responses. Results We characterize the profile of WFH in both spatial and temporal aspects, together with a set of common collaborative tools and coordination and control mechanisms. We revealed some areas of project management that are relatively more challenging during WFH situations, such as coordination, communication and project planning. We also revealed a mixed picture of the perceived impact of WFH on different software engineering activities. Conclusion WFH is a situational phenomenon which can have both negative and positive impact on software teams. For practitioners, we suggest a unified approach to consider the context of WFH, collaborative tools, associated coordination and control approaches and a process that resolve those aspects that are sensitive to physical interaction.

Qualitative surveys are emerging as a popular research method in software engineering (SE), particularly as many aspects of the field are increasingly socio-technical and thus concerned with the subtle, social, and often ambiguous issues that are not amenable to a simple quantitative survey. While many argue that qualitative surveys play a vital role amongst the diverse range of methods employed in SE there are a number of shortcomings that inhibits its use and value. First there is a lack of clarity as to what defines a qualitative survey and what features differentiate it from other methods. There is an absence of a clear set of principles and guidelines for its execution, and what does exist is very inconsistent and sometimes contradictory. These issues undermine the perceived reliability and rigour of this method. Researchers are unsure about how to ensure reliability and rigour when designing qualitative surveys and reviewers are unsure how these should be evaluated. In this paper, we present a systematic mapping study to identify how qualitative surveys have been employed in SE research to date. This paper proposes a set of principles, based on a multidisciplinary review of qualitative surveys and capturing some of the commonalities of the diffuse approaches found. These principles can be used by researchers when choosing whether to do a qualitative survey or not. They can then be used to design their study. The principles can also be used by editors and reviewers to judge the quality and rigour of qualitative surveys. It is hoped that this will result in more widespread use of the method and also more effective and evidence-based reviews of studies that use these methods in the future.

Abstract Managing non-functional requirements (NFRs) has been a challenge in software development for many years. These requirements are typically used to make important architectural decisions early in the project, which can be problematic if they are uncertain or unstable. When this uncertainty is not considered when designing the software architecture, changes are often costly and sometimes even unfeasible. Some empirical studies on the subject have already been carried out, but few have focused on the perspective of professionals with extensive experience on the changes and uncertainties of NFRs. This work aims to expand the understanding about the management, clarity and validation of NFRs to fill this gap in the literature. To achieve this goal, a survey was carried out with professionals to find out how NFRs were managed and validated. For the research design, instead of generic questions, the questionnaire focused on some specific types of NFRs to induce participants to recall and report concrete situations. As a result, 40 valid responses were obtained, most from professionals with more than 10 years of experience. The results reveal that a significant number of NFRs were defined after the delivery of software increments (more than 30%) and that revision and change occurred in about a third of the NFRs. Hence, this study presents evidence that NFRs, as the functional ones, can also be uncertain and change frequently, requiring agile approaches and techniques to evolve the software architecture to consider this uncertainty.

The amount of data collected from software use facilitated systematic inquiries about users' expectations and reactions to software systems. The most well-known practice for continuous experimentation is A/B tests, in which two or more versions of a feature are compared based on users' metrics when exposed to these options. The literature shows that A/B tests are spread in the industry, but companies usually rely on in-house software platforms based on traffic split. However, these approaches are better suitable for feature tests and not implementation options, besides increasing technical debt and maintenance effort. We aim to develop a model as a guide to frameworks for introducing experiments at class and component levels without coupling this concern to business rules. To reach this goal, we followed an iterative approach based on Design Science Research. In this process, we developed a reference im-plementation of this model by creating a framework for the Java language. We evaluated the experimentation framework usage in a web project that simulates a realistic scenario for applying A/B tests. The evaluation study showed that the framework provides the functional features needed to perform A/B tests at component and class level, collecting relevant information for decision-making. Moreover, it was possible to instantiate the framework without any coupling to code related to business rules. The proposed model could guide the development of experimentation frameworks that perform A/B tests at class and component levels, even for other languages and platforms. It captured the main elements of the experimentation domain, proposing appropriate extension points and a decoupled approach to be plugged into the application.

Background: Researchers in software engineering have increasingly added gray literature (GL) to primary and, especially, secondary studies. Several reasons explain this decision, such as grasping practitioners’ view on the topic under study. However, the use of GL in research poses several challenges like the amount and unstructured nature of data. The lack of automated tools and approaches to aid this task creates a bottleneck in selecting documents for inclusion. Aims: We investigate how summaries generated by PositionRank, an unsupervised text summarization approach, could support the inclusion analysis of documents in a GL study. Method: We performed an evaluation of using PositionRank to summarize documents analyzed on an ongoing study on software engineering. We compared the rating among two raters in a cross-over setup using summaries and full-text documents. We calculated their agreement, the precision and miss-rate using summaries against the full-text. The raters also discussed the documents on which they had conflicted answers and reached categories of reasons to explain the disagreements. Results: The results indicate that some inclusion criteria, which might be positively determined by few sentences, is susceptible to be misclassified when using summaries. Conclusions: Our study presents an analysis of the use of automatic summarization to support the inclusion assessment in gray literature studies discussing when this solution is viable. Our results could guide further studies in this direction.

Software startups develop innovative, software-intensive products or services. Such innovativeness translates into uncertainty regarding a matching need for a product from potential customers, representing a possible determinant reason for startup failure. Research has shown that experimentation, an approach based on the use of experiments to guide several aspects of software development, could improve these companies’ success rate by fostering the evaluation of assumptions about customers’ needs before developing a full-fledged product. Nevertheless, software startups are not using experimentation as expected. In this study, we investigated the reasons behind such a mismatch between theory and practice. To achieve it, we performed a qualitative survey study of 106 failed software startups. We built the eXperimentation Progression model (XPro), demonstrating that the effective adoption and implementation of experimentation is a staged process: first, teams should be aware of experimentation, then they need to develop an intention to experiment, perform the experiments, analyze the results, and finally act based on the obtained learning. Based on the XPro model, we further identified 25 inhibitors that prevent a team from progressing along the stages properly. Our findings inform researchers of how to develop practices and techniques to improve experimentation adoption in software startups. Practitioners could learn various factors that could lead to their startup failure so they could take action to avoid them.

[Context] Software startups are engines of innovation and economy, yet building software startups is challenging and subject to a high failure rate. They need to act and respond fast in highly uncertain business environments. To do so, they need to identify crucial and actionable information that supports them in making correct decisions and reduce uncertainty. So far, the software startup literature focused predominantly on what information to measure from a metrics perspective. Thus, there is a lack of research investigating how to deal with information from an analytics perspective.

Developing soft skills as well as other non-technical issues is essential for a successful career in software engineering. Educators, practitioners and researchers are paying more attention to this matter as they understand its importance to a software development context. Even the IEEE/ACM software engineering guidelines has already pointed out the importance of working with real-world projects in order to develop such skills. Being technically competent is not enough; students should have opportunities to go beyond coding and experience interactions with real users in order to better prepare themselves for their future. In this sense, this paper presents a software engineering undergraduate program that connects students with real projects throughout its curriculum. In order to evaluate whether this program helps students into understanding the importance of connecting and interacting with real stakeholders, we performed a survey with 111 students from this program. Our results indicate that providing a structure throughout the program in which students actually work on real projects is beneficial for their soft skills development.

Background: Given the social aspects of Software Engineering (SE), in the last twenty years, researchers from the field started using research methods common in social sciences such as case study, ethnography, and grounded theory. More recently, case survey, another imported research method, has seen its increasing use in SE studies. It is based on existing case studies reported in the literature and intends to harness the generalizability of survey and the depth of case study. However, little is known on how case survey has been applied in SE research, let alone guidelines on how to employ it properly. Aims: This article aims to provide a better understanding of how case survey has been applied in Software Engineering research. Method: To address this knowledge gap, we performed a systematic mapping study and analyzed 12 Software Engineering studies that used the case survey method. Results: Our findings show that these studies presented a heterogeneous understanding of the approach ranging from secondary studies to primary inquiries focused on a large number of instances of a research phenomenon. They have not applied the case survey method consistently as defined in the seminal methodological papers. Conclusions: We conclude that a set of clearly defined guidelines are needed on how to use case survey in SE research, to ensure the quality of the studies employing this approach and to provide a set of clearly defined criteria to evaluate such work.

Business Model Canvas (BMC) is a tool widely used to describe startup business models. Despite the various business aspects described, BMC pays a little emphasis on team- related factors. The importance of team-related factors in software development has been acknowledged widely in literature. While not as extensively studied, the importance of teams in software startups is also known in both literature and among practitioners. In this paper, we propose potential changes to BMC to have the tool better reflect the importance of the team, especially in a software startup environment. Based on a literature review, we identify various components related to the team, which we then further support with empirical data. We do so by means of a qualitative case study of five startups.

The Lean Startup methodology disseminated the concept of MVP. Since then, the term has been used in several contexts aside startups with a blur definition. Practitioners name several artifacts as MVPs, such as prototypes or initial versions of a new product, rather than an instrument for experimentation. Given the importance of experimentation to the success rate of software startups, it is essential to understand if the experimentation element is still present in practitioners' understanding of the term and how they applied MVPs. To achieve this objective, we performed a survey with practitioners and coded their answers according to aspects found in a systematic mapping study on MVP. Our results indicate that MVP is mostly associated with the ideas of a product version and customer value rather than hypothesis testing and learning processes. Additionally, those respondents that focused on the first related group of terms did not give experiments as examples of MVP. In contrast, the opposite happened to those that used the second group.

Software startups develop innovative products for which there are typically no customers to refer to elicit requirements. Often, these companies develop a set of features without a better understanding of customer needs. An experiment-based approach to validate hypotheses about the customer and market could increase their chance of success or, at least, accelerate their realization of the product worthlessness. The first step of an experiment-based approach is to elicit hypotheses to guide experiments. Software startups base their products on business assumptions, but there is a lack of understanding of how these assumptions are formed and how teams could elicit hypotheses systematically. To fill this gap, we performed an empirical study consisted of two steps. First, we explored based on which assumptions startups define their products using a multiple case study. The results indicate that these companies developed their products based on founders’ assumptions derived from their previous experience. Second, we investigated cognitive mapping as a tool to elicit hypotheses systematically with two software startups. The results indicate that this approach can serve as the basis of a method to elicit hypotheses in early-stage software startups.

The need of skills other than technical from software developers is becoming evident. The DevOps movement is an example of that applied to operational tasks. Startup development methodologies focus on business activities in innovative organizations. Several universities offer courses based on these methodologies to software engineering students, mainly to improve their creativity, problem solving, and business skills. This paper investigates how software engineering students learned startup development methodologies and discusses what are the challenges and benefits in their learning process. We conducted a multi-method study in three different universities. The data was collected in two phases and analyzed using thematic analysis. Our study reveals that students realized the importance of collaboration with other courses and the importance of user involvement in development. However, students tend to over-simplify concepts, trying to adapt them to what they are familiar with. The results indicate the necessity of business education for technical students and directions for improvements.

The importance of startups to economic development is indisputable. Software startups are startups that develop an innovative software-intensive product or service. In spite of the rising of several methodologies to improve their efficiency, most of software startups still fail. There are several possible reasons to failure including under or over-engineering the product because of not-suitable engineering practices, wasted resources, and missed market opportunities. The literature argues that experimentation is essential to innovation and entrepreneurship. Even though well-known startup development methodologies employ it, studies revealed that practitioners still do not use it. Given that requirements engineering is in between software engineering and business, in this study, I aim to improve these practices to foster experimentation in software startups. To achieve that, first I investigated how requirements engineering activities are performed in software startups. Then, my goal is to propose new requirements engineering practices to foster experimentation in this context.

Context: Over the past 20 years, software startups have created many products that have changed human life. Since these companies are creating brand-new products or services, requirements are difficult to gather and highly volatile. Although scientific interest in software development in this context has increased, the studies on requirements engineering in software startups are still scarce and mostly focused on elicitation activities. Objective: This study overcomes this gap by answering how requirements engineering practices are performed in this context. Method: We conducted a grounded theory study based on 17 interviews with software startups practitioners. Results: We constructed a model to show that software startups do not follow a single set of practices but, instead, build a custom process, changed throughout the development of the company, combining different practices according to a set of influences (Founders, Software Development Manager, Developers, Market, Business Model and Startup Ecosystem). Conclusion: Our findings show that requirements engineering activities in software startups are similar to those in agile teams, but some steps vary as a consequence of the lack of an accessible customer.