In recent years, large language models have been widely integrated into software engineering workflows, supporting tasks like code generation. While prior evaluations focus on functional correctness, there is still a limited understanding of the non-functional quality characteristics of generated code. Guided by the ISO/IEC 25010 quality model, this study adopts a multi-methods approach comprising three complementary elements: a literature review of 109 papers, two industry workshops with practitioners from multiple organizations, and an empirical analysis of patching real-world software issues using three LLMs. Motivated by insights from both the literature and practitioners, the empirical study examined the quality of generated patches regarding security, maintainability, and performance efficiency, which were identified as critical code-level quality attributes. Our results indicate that existing research primarily emphasizes security, performance efficiency, and maintainability, while other quality attributes are understudied. In contrast, practitioners prioritize maintainability and readability, warning that generated code may accelerate the accumulation of technical debt. The empirical evaluation demonstrates the instability of optimizing NFQCs through prompts in practical software engineering settings. Overall, our findings expose a misalignment between academic focus, industry priorities, and observed model behavior, highlighting the need to integrate quality assurance mechanisms into LLM code generation pipelines to ensure that future generated code not only passes tests but truly passes with quality.

Summary The importance of the team, its internal dynamics, and its performance are widely recognized within the software engineering community. While popular frameworks identify wholeness, stability over time, and smallness as important factors, they offer little guidance on how to form teams that achieve these three characteristics. The objective of this study is to investigate how these team characteristics interact in large‐scale software development contexts, particularly focusing on the impact of stable and dynamic teaming approaches. This was done through a multivocal study of literature, followed by individual semi‐structured interviews with 19 engineers from two companies and validation workshops with an additional two companies from unrelated industry segments. The study results show that the question of stable versus dynamic approaches to forming software engineering teams is largely unaddressed in industry, with stable teams representing a habitual default option. Meanwhile, both stable and dynamic teams clearly have respective strengths and weaknesses, calling for careful consideration of the most suitable approach in any given situation. To support such consideration, this paper presents a model of how team stability, wholeness, and smallness interact. This model is found relevant, accurate, generalizable, and useful by practitioners.

Abstract The characteristics of the test environment are of vital importance to its ability to support the organizations testing objectives. This paper seeks to address the need for a structured and reliable approach, which can be used by companies and other organizations to optimize their test environments in each individual case. The reported study included a series of interviews with 30 individuals, a series of focus groups with in total 31 individuals and a cross‐company workshop with 30 participants from five large‐scale companies, operating in different industry segments. The study resulted in a list of success factors, including not only characteristics and capabilities existing within a test environment (intrinsic success factors) but also properties not inherent to the test environment, but still vital for a successfully implemented test environment (extrinsic success factors). This distinction is important, as the root causes differ and as addressing them requires distinct approaches—not only of technology but also of organization, communication and collaboration. We find that successful implementations of test environments for large‐scale software systems depend primarily on how they support the company's business strategy, test organization and product testability (extrinsic success factors). Based on this, test environments can then be optimized to improve test environment capabilities, usability and stability (intrinsic success factors). The list of intrinsic and extrinsic success factors was well received by all five companies included in the study, supporting that the intrinsic and extrinsic success factors for test environments can be applied to a large segment of the software industry.

Based on multiple series of interviews and workshops, this paper presents the MaLET model – a representation of the typical evolution path for companies who successfully adopt exploratory testing. The model provides a step-by-step approach to systematically improve exploratory testing over time, and shows how and why capabilities may regress. The MaLET model was validated through a series of interviews with 20 interviewees from eight case study companies in separate industry segments. The interviews also revealed examples on improvement initiatives that had failed in the companies, showing that improvement initiatives tend to fail if they are not planned in an order corresponding to the maturity levels in the model. The MaLET model was well received by the interviewees during the validation, describing the model as sound, relevant and useful in practice.

Continuous integration and delivery are well-established paradigms in the software development community. With these continuous practices come many challenges; while some of these challenges are immediate and well documented in literature, others may be revealed only after sustained application of these practices in large-scale and complex contexts. Based on researcher observations and interviews with 22 senior professionals from four companies, all with significant but varied experiences of continuous practices, we present a set of underreported challenges with continuous practices observed in multiple industry settings. Through a grounded theory approach we construct the Tapco model — Test Automation Progression in Continuous Practices — identifying two distinct ways in which companies progress towards continuous delivery, onto which the studied cases are mapped. This model is then validated by presenting it to the four studied companies, operating in disparate industry segments, and to three additional industry cases, letting them evaluate its relevancy, accuracy and novelty. We find that the model provides industry professionals with essential guidance on how to avoid common pitfalls, as well as an understanding of their causes and possible remediation.

This paper identifies three improvement areas related to system design and architecture, where an organization can change to better support continuous integration and continuous delivery: “The product's architecture”, “Ways to work with system design and architecture”, and “The role of the architect”. The three improvement areas are based on a literature review, two series of interviews and a cross-company workshop with three case-study companies. Furthermore, the paper proposes three actionable strategies corresponding to the three identified improvement areas: “Systems with a modular and loosely coupled architecture”, “A balanced approach where system design and architecture is focused on the system's most important characteristics”, and “Architects shifting perspective from control to facilitation”.

Abstract Based on 25 interviews with participants from four case study companies that develop large‐scale software embedded systems, this paper presents the Test Activity Stakeholders (TAS) model. The TAS model shows how the continuous integration and delivery pipeline can be designed to include test activities that support four stakeholder interests: “Check changes,” “Secure stability,” “Measure progress,” and “Verify compliance.” The model is developed to show how each of the stakeholder interests are best supported by unit/component tests or system tests, by automated testing or manual testing and by tests executed in simulated environments or on real hardware. The TAS model may serve as a starting point for companies when evaluating and designing their continuous integration and delivery pipeline. The validation of the TAS model included twelve individuals from three case study companies. The validation showed that the TAS model is actionable and useful in practice, enabling the identification of stakeholders and showing where improvement efforts should be focused in order to support all stakeholder interests in the continuous integration and delivery pipeline.

This extended abstract summarizes an article, which has been published in Information and Software Technology and was selected for the Journal-First presentations at the International Conference on Software and System Process (ICSSP 2018).

The EMFIS model allows companies to explicate a representation of the organization's current situation regarding continuous integration impediments, and visualizes what the organization must focus on in order to enable more frequent integration of software. The model is used to perform an assessment of twelve factors, where the ratings from participants representing the developers are summarized separately from ratings from participants representing the enablers (responsible for processes, development tools, test environments etc.). The EMFIS model is based on semi-structured interviews with 20 developers from two companies which develop large-scale software systems, and a literature review that included 74 research papers and four books. The model has been validated in workshops and interviews, which in total included 28 individuals in five different companies. The model was well received during the validation, and was appreciated for its simplicity and its ability to show differences in rating between developers and enablers.

The importance of traceability in software development has long been recognized, not only for reasons of legality and certification, but also to enable the development itself. At the same time, organizations are known to struggle to live up to traceability requirements, and there is an identified lack of studies on traceability practices in the industry, not least in the area of tooling and infrastructure. This paper identifies traceability as a key challenge in achieving continuous integration and delivery and documents an industry developed framework - Eiffel - designed to provide real time traceability in continuous integration and delivery. The efficacy of the Eiffel framework is then investigated through comparison with previous traceability methods.

DevOps and continuous practices are attracting steadily growing attention by both practitioners and researchers in the software engineering community. The terms are often used inconsistently, interchangeably and with unclear meaning, however. Taking the position that this ambiguity and miscommunication renders the community great harm, in that it impedes our ability to critically appraise these practices, their effects and interplay between them, we analyze how published literature on both continuous practices and DevOps treat the terms. Based on this analysis, along with statements by often cited sources in the community as well as personal experience from researching and practicing these concepts, we propose guidelines to help authors reduce ambiguity in their publications. Additionally, definitions designed to reflect mainstream interpretation while disentangling the terms from one another are presented.

Based on interviews with 20 developers from two case study companies that develop large-scale software-intensive embedded systems, this paper presents the main factors that affect how often developers deliver software to the mainline. Further on, the paper describes the continuous integration behaviors in projects where up to 1,000 developers commit to the same mainline. The main factors that could enable more frequent integration of software are: "Activity planning and execution", "System thinking", "Speed" and "Confidence through test activities". Behind these main themes we also present a wide range of sub-categories ("Modular and loosely coupled architecture", "Test selection" etc) which summarizes what the developers themselves see as the continuous integration impediments in large-scale industry projects.

The importance of traceability in software development has long been recognized, not only for reasons of legality and certification, but also to enable the development itself. At the same time, organizations are known to struggle to live up to traceability requirements, and there is an identified lack of studies on traceability practices in the industry, not least in the area of tooling and infrastructure. This paper identifies traceability as a key challenge in achieving continuous integration and delivery and documents an industry developed framework -- Eiffel -- designed to provide real time traceability in continuous integration and delivery. The efficacy of the Eiffel framework is then investigated through comparison with previous traceability methods.