In recent years, large language models have been widely integrated into software engineering workflows, supporting tasks like code generation. While prior evaluations focus on functional correctness, there is still a limited understanding of the non-functional quality characteristics of generated code. Guided by the ISO/IEC 25010 quality model, this study adopts a multi-methods approach comprising three complementary elements: a literature review of 109 papers, two industry workshops with practitioners from multiple organizations, and an empirical analysis of patching real-world software issues using three LLMs. Motivated by insights from both the literature and practitioners, the empirical study examined the quality of generated patches regarding security, maintainability, and performance efficiency, which were identified as critical code-level quality attributes. Our results indicate that existing research primarily emphasizes security, performance efficiency, and maintainability, while other quality attributes are understudied. In contrast, practitioners prioritize maintainability and readability, warning that generated code may accelerate the accumulation of technical debt. The empirical evaluation demonstrates the instability of optimizing NFQCs through prompts in practical software engineering settings. Overall, our findings expose a misalignment between academic focus, industry priorities, and observed model behavior, highlighting the need to integrate quality assurance mechanisms into LLM code generation pipelines to ensure that future generated code not only passes tests but truly passes with quality.

Continuous integration and delivery (CI/CD) have transformed software development by reducing delivery time, improving product quality, and giving enterprises a competitive advantage. However, large-scale projects confront difficulties in giving fast feedback to developers due to large test suites, resulting in longer testing cycles and lower productivity. Traditional regression testing methods struggle to find a balance between efficacy and efficiency, demanding advanced approaches. This study investigates the use of machine learning (ML), specifically Neural Network and Random Forest models, to choose test cases based on source code changes, commit messages, and change file path in order to offer developers with faster feedback. The study investigates the predicted accuracy of ML models using a large industrial dataset from a telecom company, which included 15 million test executions over 15 months. The results show that Random Forest outperforms Neural Network models in test case selection, with up to 97% accuracy achieved. Real-time evaluations conducted over a month show significant savings in test executions (88 % -90 %) and testing time (44 % -74%) across multiple regression testing activities, illustrating the potential of ML-driven techniques to optimize CI/CD pipelines and increase developer productivity.

Testing of a large-scale and complex software system requires many types of knowledge, skills and personality traits. Contrasting the idea of a perfect all-round tester, this paper presents the Testing Hopscotch model with six complementary profiles, and the key characteristics considered as most relevant for each profile. The model is based on 60 interviews with engineers from three large-scale companies in different industry segments. The Testing Hopscotch model was well received by three focus groups including 22 participants, which further strengthens the validity of the model.

Many test prioritisation techniques have been proposed in order to improve test effectiveness of Continuous Integration (CI) pipelines. Particularly, diversity-based testing (DBT) has shown promising and competitive results to improve test effectiveness. However, the technical and practical challenges of introducing test prioritisation in CI pipelines are rarely discussed, thus hindering the applicability and adoption of those proposed techniques. This research builds on our prior work in which we evaluated diversity-based techniques in an industrial setting. This work investigates the factors that influence the adoption of DBT both in connection to improvements in test cost-effectiveness, as well as the process and human related challenges to transfer and use DBT prioritisation in CI pipelines. We report on a case study considering the CI pipeline of Axis Communications in Sweden. We performed a thematic analysis of a focus group interview with senior practitioners at the company to identify the challenges and perceived benefits of using test prioritisation in their test process. Our thematic analysis reveals a list of ten challenges and seven perceived effects of introducing test prioritisation in CI cycles. For instance, our participants emphasized the importance of introducing comprehensible and transparent techniques that instill trust in its users. Moreover, practitioners prefer techniques compatible with their current test infrastructure (e.g., test framework and environments) in order to reduce instrumentation efforts and avoid disrupting their current setup. In conclusion, we have identified tradeoffs between different test prioritisation techniques pertaining to the technical, process and human aspects of regression testing in CI. We summarize those findings in a list of seven advantages that refer to specific stakeholder interests and describe the effects of adopting DBT in CI pipelines.

Abstract Several operations, ranging from regular code updates to compiling, building, testing, and distribution to customers, are consolidated in continuous integration and delivery. Professionals seek additional information to complete the mission at hand during these tasks. Developers who devote a large amount of time and effort to finding such information may become distracted from their work. We will better understand the processes, procedures, and resources used to deliver a quality product on time by defining the types of information that software professionals seek. A deeper understanding of software practitioners' information needs has many advantages, including remaining competitive, growing knowledge of issues that can stymie a timely update, and creating a visualisation tool to assist practitioners in addressing their information needs. This is an extension of a previous work done by the authors. The authors conducted a multiple‐case holistic study with six different companies (38 unique participants) to identify information needs in continuous integration and delivery. This study attempts to capture the importance, frequency, required effort (e.g. sequence of actions required to collect information), current approach to handling, and associated stakeholders with respect to identified needs. 27 information needs associated with different stakeholders (i.e. developers, testers, project managers, release team, and compliance authority) were identified. The identified needs were categorised as testing, code & commit, confidence, bug, and artefacts. Apart from identifying information needs, practitioners face several challenges in developing visualisation tools. Thus, 8 challenges that were faced by the practitioners to develop/maintain visualisation tools for the software team were identified. The recommendations from practitioners who are experts in developing, maintaining, and providing visualisation services to the software team were listed.

Developers often spend time to determine whether test case failures are real failures or flaky. The flaky tests, also known as non-deterministic tests, switch their outcomes without any modification in the codebase, hence reducing the confidence of developers during maintenance as well as in the quality of a product. Re-running test cases to reveal flakiness is resource-consuming, unreliable and does not reveal the root causes of test flakiness. Our paper evaluates a multi-factor approach to identify flaky test executions implemented in a tool named MDF laker. The four factors are: trace-back coverage, flaky frequency, number of test smells, and test size. Based on the extracted factors, MDFlaker uses k-Nearest Neighbor (KNN) to determine whether failed test executions are flaky. We investigate MDFlaker in a case study with 2166 test executions from different open-source repositories. We evaluate the effectiveness of our flaky detection tool. We illustrate how the multi-factor approach can be used to reveal root causes for flakiness, and we conduct a qualitative comparison between MDF laker and other tools proposed in literature. Our results show that the combination of different factors can be used to identify flaky tests. Each factor has its own trade-off, e.g., trace-back leads to many true positives, while flaky frequency yields more true negatives. Therefore, specific combinations of factors enable classification for testers with limited information (e.g., not enough test history information).

Summary Identifying the root causes of test flakiness is one of the challenges faced by practitioners during software testing. In other words, the testing of the software is hampered by test flakiness. Since the research about test flakiness in large‐scale software engineering is scarce, the need for an empirical case‐study where we can build a common and grounded understanding of the problem as well as relevant remedies that can later be evaluated in a large‐scale context is a necessity. This study reports the findings from a multiple‐case study. The authors conducted an online survey to investigate and catalogue the root causes of test flakiness and mitigation strategies. We attempted to understand how practitioners perceive test flakiness in closed‐source development, such as how they define test flakiness and what practitioners perceive can affect test flakiness. The perceptions of practitioners were compared with the available literature. We investigated whether practitioners' perceptions are reflected in the test artefacts such as what is the relationship between the perceived factors and properties of test artefacts. This study reported 19 factors that are perceived by professionals to affect test flakiness. These perceived factors are categorized astest code,system under test,CI/test infrastructure, andorganization‐related. The authors concluded that some of the perceived factors in test flakiness in closed‐source development are directly related to non‐determinism, whereas other perceived factors concern different aspects, for example, lack of good properties of a test case, deviations from the established processes, and ad hoc decisions. Given a data set from investigated cases, the authors concluded that two of the perceived factors (i.e., test case size and test case simplicity) have a strong effect on test flakiness.

Today it is very common for software systems to be built by teams located in more than one country. For example, a project team may be located in the US while the team lead resides in Sweden. How then should students be trained for this kind of work? Senior design or capstone projects offer students real-world hands-on experience but rarely while working internationally. One reason is that most instructors do not have international business contacts that allow them to find project sponsors in other countries. Another reason is the fear of having to invest a huge amount of time managing an international project. In this paper we present the general concepts related to "International Capstone Project Exchanges", the basic model behind the exchanges (student teams are led by an industry sponsor residing in a different country) and several alternate models that have been used in practice. We will give examples from projects in the US, Germany, Sweden, Australia, and Colombia. We have extended the model beyond software projects to include engineering projects as well as marketing, and journalism. We conclude with a description of an International Capstone Project Exchange website that we have developed to aid any university in establishing their own international project exchange.

We suggest a Bayesian approach to the problem of reducing bug turn-around time in large software development organizations. Our approach is to use classification to predict where bugs are located in components. This classification is a form of automatic fault localization (AFL) at the component level. The approach only relies on historical bug reports and does not require detailed analysis of source code or detailed test runs. Our approach addresses two problems identified in user studies of AFL tools. The first problem concerns the trust in which the user can put in the results of the tool. The second problem concerns understanding how the results were computed. The proposed model quantifies the uncertainty in its predictions and all estimated model parameters. Additionally, the output of the model explains why a result was suggested. We evaluate the approach on more than 50000 bugs.

In a modern computing curriculum, large-project courses are essential to give students hands-on experience of working in a realistic software engineering project. Assessing such projects is, however, extremely challenging. There are various aspects and trade-offs of assessments that can affect course quality. Individual assessments may fairly grade individuals, but may lose focus of the project as a group activity. Extensive teacher involvement is necessary for objective assessment, but may affect the way that students work. Continuous feedback to students can enhance learning, but may be hard to combine with fair assessment. Most previous work focuses on some specific assessment aspect; in this article, we present an assessment model that consists of a collection of assessment activities, each covering different aspects. We have applied, developed, and improved these activities during a 7yr period. To evaluate the usefulness of the model, we perform questionnaire-based surveys over a 2yr period. Furthermore, we design and execute an experiment that studies to what extent students can perform fair peer assessment and to what degree the assessments of students and teachers agree. We analyze the results, discuss findings, and summarize lessons learned.

Using design models instead of executable code has shown itself to be an efficient way of increasing abstraction level of software development. However, applying established code-based software engineering methods to design models can be a challenge - due to different abstraction levels, the same metrics as for code are not applicable for the design models. One of practical challenges in using metrics at the model level is applying complexity-prediction formulas developed using code-based metrics to design models. The existing formulas do not apply as they do not take into consideration the behavior part of the models - e.g. State charts. In this paper we address this challenge by conducting a case study at one of the large telecom products at Ericsson with the goal to identify which metrics can predict complex, hard to understand and hard to maintain software modules based on their design models. We use both statistical methods like regression to build prediction formulas and qualitative interviews to codify expert designers' perception of which software modules are complex. The results of this case study show that such measures as the number of non-self-transitions, transition per states or state depth can be combined in order to identify software units that are perceived as complex by expert designers. Our conclusion is that these metrics can be used in other companies to predict complex modules, but the coefficients should be recalculated per product to increase the prediction accuracy.

Larger project courses, such as capstone projects, are essential in a modern computing curriculum. Assessing such projects is, however, extremely challenging. There are various aspects and tradeoffs of assessments that can affect the quality of a project course. Individual assessments can give fair grading of individuals, but may loose focus of the project as a group activity. Extensive teacher involvement is necessary for objective assessment, but may affect the way students are working. Continuous feedback to students can enhance learning, but may be hard to combine with fair assessment. Most previous work is focusing on some specific assessment aspect, whereas we in this paper present an assessment model that consists of a collection of assessment activities, each covering different aspects. We have applied, developed, and improved these activities during a six-year period and evaluated their usefulness by performing a questionnaire-based survey.

BACKGROUND: Agile software development methods have a number of reported benefits on productivity, project visibility, software quality and other areas. There are also negative effects reported. However, the base of empirical evidence to the claimed effects needs more empirical studies. AIM: The purpose of the research was to contribute with empirical evidence on the impact of using agile principles and practices in large-scale, industrial software development. Research was focused on impacts within seven areas: Internal software documentation, Knowledge sharing, Project visibility, Pressure and stress, Coordination effectiveness, and Productivity. METHOD: Research was carried out as a multiple-case study on two contemporary, large-scale software development projects with different levels of agile adoption at Ericsson. Empirical data was collected through a survey of project members. RESULTS AND CONCLUSIONS: Intentional implementation of agile principles and practices were found to: correlate with a more balanced use of internal software documentation, contribute to knowledge sharing, correlate with increased project visibility and coordination effectiveness, reduce the need for other types of coordination mechanisms, and possibly increase productivity. No correlation with increase in pressure and stress were found.

Maintenance costs can be substantial for organizations with very large and complex software systems. This paper describes research for reducing anomaly report turnaround time which, if successful, would contribute to reducing maintenance costs and at the same time maintaining a good customer perception. Specifically, we are addressing the problem of the manual, laborious, and inaccurate process of assigning anomaly reports to the correct design teams. In large organizations with complex systems this is particularly problematic because the receiver of the anomaly report from customer may not have detailed knowledge of the whole system. As a consequence, anomaly reports may be wrongly routed around in the organization causing delays and unnecessary work. We have developed and validated machine learning approach, based on stacked generalization, to automatically route anomaly reports to the correct design teams in the organization. A research prototype has been implemented and evaluated on roughly one year of real anomaly reports on a large and complex system at Ericsson AB. The prediction accuracy of the automation is approaching that of humans, indicating that the anomaly report handling time could be significantly reduced by using our approach.

When teaching software engineering courses it is highly important to have good text books that are well-founded, up-to-date, and easily accessible to students. However, currently available text books on the market are either very broad or highly specialized, making it hard to select appropriate books for specific software engineering courses. Moreover, due to the rapidly changing subject of software engineering, books tend to become obsolete, which make students hesitate to buy books even if they are part of the listed course literature. In this paper, we briefly explain and discuss an approach of using a web-based system for creating collaborative and peer-reviewed text books that can be customized individually for specific courses. We describe and discuss the proposed system from a use case perspective.

Software processes, such as RUP and agile methods, focus their requirements engineering part on use cases and thus functional requirements. Complex products, such as radio network control software, need special handling of non-functional requirements as well. We describe how we used the eclipse process framework to augment the open and minimal OpenUP/basic process with improvements found in management of capacity requirements in a case-study at Ericsson. The result is compared with another project improving RUP to handle performance requirements. The major differences between the improvements are that 1) they suggest a special, dedicated performance manager role and we suggest that present roles are augmented, 2) they suggest a bottom-up approach to performance verification while we focus on system performance first, i.e. top-down. Further, we suggest augmenting Um l-2 models with capacity attributes to improve information flow from requirements to implementation.

Capacity in telecommunication systems is highly related to operator revenue. As a vendor of such systems, Ericsson AB is continuously improving its processes for estimating, specifying, tuning, and testing the capacity of delivered systems. In order to systematize process improvements Ericsson AB and Linköping University joined forces to create an anatomy of Capacity Sub Processes (CSPs). The anatomy is the result of an interview series conducted to document good practices amongst organizations active in capacity improvement. In this paper we analyze four different development processes in terms of how far they have reached in their process maturity according to our anatomy and show possible improvement directions. Three of the processes are currently in use at Ericsson, and the fourth is the OpenUP/Basic process which we have used as a reference process in earlier research. We also include an analysis of the observed good practices. The result mainly confirms the order of CSPs in the anatomy, but we need to use our information of the maturity of products and the major life cycle in the organization in order to fully explain the role of the anatomy in planning of improvements.

There is evidence to suggest that the software industry has not yet matured as regards management of nonfunctional requirements (NFRs). Consequently the cost of achieving required quality is unnecessarily high. To try and avoid this, the telecommunication systems provider Ericsson defined a research task to try and improve the management of requirements for capacity, which is one of the most critical NFRs. Linkoping University joined in the effort and conducted an interview series to investigate good practice within different parts of the company. Inspired by the interviews and an ongoing process improvement project a model for improvement was created and activities were synthesized. This paper contributes the results from the interview series, and details the subprocesses of specification that should be improved. Such improvements are about understanding the relationship between numerical entities at all system levels, augmenting UML specifications to make NFRs visible, working with time budgets, and testing the sub system level components on the same level as they are specified

Understanding the dynamic behavior of a system is a key determinant to successful system maintenance. This paper contributes two studies at Ericsson Radio Systems of the perfective maintenance of large and distributed systems. Our approach is a holistic method based on tracing and the technical solution to acquisition of trace data is to use CORBA interceptors. Our method has proven useful in solving a wide variety of problems in design as well as implementation and test-all this at a small price. Examples of improvements are performance, new test cases and merging of objects.

The task of finding an optimal selection of requirements for the next release of a software system is difficult as requirements may depend on each other in complex ways. The paper presents the results from an in-depth study of the interdependencies within 5 distinct sets of requirements, each including 20 high-priority requirements of 5 distinct products from 5 different companies. The results show that: (1) roughly 20% of the requirements are responsible for 75% of the interdependencies; (2) only a few requirements are singular; (3) customer-specific bespoke development tend to include more functionality- related dependencies whereas market-driven product development have an emphasis on value-related dependencies. Several strategies for reducing the effort needed for identifying and managing interdependencies are outlined. A technique for visualization of interdependencies with the aim of supporting release planning is also discussed. The complexity of requirements interdependency analysis is studied in relation to metrics of requirements coupling. Finally, a number of issues for further research are identified.

Impact analysis is an essential activity as cost and effort estimation is based on its outcome. Impact analysis, as performed in an industrial object-oriented system, has been analysed and evaluated in this case study. Preceding the design phase of the fourth release of the PMR system (PMR R4), the consequences of a set of new requirements were analysed. Traceability links were established between each new requirement and the objects predicted to be changed in the design object model, which represent the C++ classes in the system. The impact analysis as performed in this study was successful in the sense that the software engineers in the study prefer this way of working. When asked, they claim they are sure the number of objects to be changed is correct, while they are less sure about the number of man-hours required. The analysis of the impact analysis shows that individual objects are often predicted almost correctly, but that the number of actually changed objects is often greater than the number of predicted objects. The study also shows that tracing by interviewing knowledgeable software engineers is far more common than consulting object models and other kinds of documentation. A factor that prohibits software engineers using object models more extensively is lack of detailed information. We see a promising potential for using traceability, together with the use of existing object models, to achieve a more structured impact analysis process and accurate prediction. © 1998 John Wiley & Sons, Ltd.

Traceability defined as the ability to trace dependent items within a model and the ability to trace correspondent items in other models is advocated as a desirable property of a software development process. Potential benefits of good traceability are clearer documentation, more focussed development, increased ease of system understanding, and more precise impact analysis of proposed changes. An industry-scale project applying the analysis and design method Objectory has been examined and documented with a number of traceability examples generated from the perspective of a maintainer attempting to understand the system. Four representative examples and a categorization of traceability are presented in this study in order to provide a concrete empirical basis for the application of traceability to systems development.