Standardization processes build upon consensus between partners, which depends on their ability to identify points of disagreement and resolving them. Large standardization organizations, like the 3GPP or ISO, rely on leaders of work packages who can correctly, and efficiently, identify disagreements, discuss them and reach a consensus. This task, however, is effort-, labor-intensive and costly. In this paper, we address the problem of identifying similarities, dissimilarities and discussion points using large language models. In a design science research study, we work with one of the organizations which leads several workgroups in the 3GPP standard. Our goal is to understand how well the language models can support the standardization process in becoming more cost-efficient, faster and more reliable. Our results show that generic models for text summarization correlate well with domain expert's and delegate's assessments (Pearson correlation between 0.66 and 0.98), but that there is a need for domain-specific models to provide better discussion materials for the standardization groups.

In this paper, we explore challenges in communication between metrics teams and stakeholders in metrics service delivery. Drawing on interviews and interactive workshops with team members and stakeholders at two different Swedish agile software development organizations, we identify interrelated challenges such as aligning expectations, prioritizing demands, providing regular feedback, and maintaining continuous dialogue, which influence team-stakeholder interaction, relationships and performance. Our study shows the importance of understanding communicative hurdles and provides suggestions for their mitigation, therefore meriting further empirical research.

As technology has allowed us to collect large amounts of industrial data, it has become critical to analyze and understand the data collected, in particular to find data anomalies. Anomaly analysis allows a company to detect, analyze and understand anomalous or unusual data patterns. This is an important activity to understand, for example, deviations in service which may indicate potential problems, or differing customer behavior which may reveal new business opportunities. Much previous work has focused on anomaly detection, in particular using machine learning. Such approaches allow clustering of data patterns by common attributes, and, although useful, clusters often do not correspond to the root causes of anomalies, meaning that more manual analysis is needed. In this paper we report on a design science study with two different teams, in a partner company which focuses on modeling and understanding the attributes and root causes of data anomalies. After iteration, for each team, we have created general and anomaly-specific UML class diagrams and goal models to capture anomaly details. We use our experiences to create an example taxonomy, classifying anomalies by their root causes, and to create a general method for modeling and understanding data anomalies. This work paves the way for a better understanding of anomalies and their root causes, leading towards creating a training set which may be used for machine learning approaches.

Metrics teams play an increasingly important role in handling data and information in modern software development organizations; they manage their companies’ measurement programs, collect and process data, and develop and distribute information products. Metrics teams can comprise several roles, and their set-up can differ between companies, as can the metrics maturity of host organizations. These differences impact the effectiveness and quality of a team’s measurement program. Our objective was to design and evaluate a model to describe the characteristics of a mature metrics team, which efficiently designs, develops, maintains, and evolves its organization’s measurement program. We conducted an action research study on four metrics teams of four distinct companies. We designed and evaluated a domain-specific model for assessing the maturity of metrics teams – MeTeaM – and also assessed the four metrics teams per se. Our results were two-fold: the creation of the metrics team maturity model MeTeaM and a template to assess metrics teams. Our evaluation showed that the model captures the characteristics of successful metrics teams and quantifies the maturity status of both the metrics teams and their host organizations. More mature metrics teams score higher in the MeTeaM model than less mature teams. The assessment provides less mature metrics teams with valuable insights on what factors to improve. Such insights can be shared with and acted upon successfully with their organizations.

Every discipline, e.g. medicine and engineering, has its own vocabulary to describe situations and tools. This dedicated language is important, because it allows for being specific, detailed and precise. On the other hand, this language, specific to each discipline, becomes a barrier for communication across disciplines. International software measurement standards are examples of such language. The standards are documents that provide definitions of terms used and describe processes specific to the discipline of software measurement. However, one major problem we have observed is that standards are difficult to read and to understand; even for the stakeholders that they are intended for. In this paper, we present our experience report from introducing standards to the work of a large software development organization at an infrastructure provider company. We describe one of the concepts that we found essential in making the measurement standards understandable, namely the notion of the indicator.

Big data and machine learning models have been increasingly used to support software engineering processes and practices. One example is the use of machine learning models to improve test case selection in continuous integration. However, one of the challenges in building such models is the identification and reduction of noise that often comes in large data. In this paper, we present a noise reduction approach that deals with the problem of contradictory training entries. We empirically evaluate the effectiveness of the approach in the context of selective regression testing. For this purpose, we use a curated training set as input to a tree-based machine learning ensemble and compare the classification precision, recall, and f-score against a non-curated set. Our study shows that using the noise reduction approach on the training instances gives better results in prediction with an improvement of 37% on precision, 70% on recall, and 59% on f-score.

Code reviews are one of the first quality assurance tasks in continuous software integration and delivery. The goal of our work is to reduce the need for manual reviews by automatically identify which code fragments should be further reviewed manually. We conducted an action research study with two companies where we extracted code reviews and build machine learning classifiers (AdaBoost and Convolutional Neural Network– CNN). Our results show that the accuracy of recognizing code fragments that require manual review, measured with Matthews Correlation Coefficient, was 0.70 in the combination of our own feature extraction and CNN. We conclude that this way of combining automation with manual code reviews can improve the speed of reviews while providing organizations with the possibility to support knowledge transfer among the designers.

Abstract Software developers in big and medium-size companies are working with millions of lines of code in their codebases. Assuring the quality of this code has shifted from simple defect management to proactive assurance of internal code quality. Although static code analysis and code reviews have been at the forefront of research and practice in this area, code reviews are still an effort-intensive and interpretation-prone activity. The aim of this research is to support code reviews by automatically recognizing company-specific code guidelines violations in large-scale, industrial source code. In our action research project, we constructed a machine-learning-based tool for code analysis where software developers and architects in big and medium-sized companies can use a few examples of source code lines violating code/design guidelines (up to 700 lines of code) to train decision-tree classifiers to find similar violations in their codebases (up to 3 million lines of code). Our action research project consisted of (i) understanding the challenges of two large software development companies, (ii) applying the machine-learning-based tool to detect violations of Sun’s and Google’s coding conventions in the code of three large open source projects implemented in Java, (iii) evaluating the tool on evolving industrial codebase, and (iv) finding the best learning strategies to reduce the cost of training the classifiers. We were able to achieve the average accuracy of over 99% and the average F-score of 0.80 for open source projects when using ca. 40K lines for training the tool. We obtained a similar average F-score of 0.78 for the industrial code but this time using only up to 700 lines of code as a training dataset. Finally, we observed the tool performed visibly better for the rules requiring to understand a single line of code or the context of a few lines (often allowing to reach the F-score of 0.90 or higher). Based on these results, we could observe that this approach can provide modern software development companies with the ability to use examples to teach an algorithm to recognize violations of code/design guidelines and thus increase the number of reviews conducted before the product release. This, in turn, leads to the increased quality of the final software.

Continuous deployment has become software companies' inevitable response to the economic pressures of the market. At the same time, software quality is crucial in order to meet customers' expectations and hence succeed in the market. Therefore, current quality management processes require transformation in order to keep up with the fast pace of the market while at the same time meeting customers' expectations. In order to figure out how the current quality management process should be transformed to keep up with the fast pace of the market while ensuring both product quality and continuous deployment, we conducted a qualitative study at a large infrastructure provider company. During the interviews we conducted with the quality manager, developer and test architect, we used a metrics portfolio consisting of 59 candidate metrics that can be used in the transformed quality management process. Our findings show that, out of these candidate metrics, 9 metrics should be used in the internal quality measurement dashboard for quality check at the end of the software development life-cycle (SDLC) before the software is released to customer site, while 3 metrics should be used by quality manager to monitor earlier phases of SDLC and 5 metrics should also be delegated to earlier phases of SDLC but without the involvement of the quality manager. To summarize, our study support the claim that quality managers should not be only gatekeepers, but also proactive controllers of quality by monitoring earlier phases of the SDLC.

Summary Automated measurement programs are an efficient way of collecting, processing, and visualizing measures in large software development companies. The number of measurements in these programs is usually large, which is caused by a diversity of the needs of the stakeholders. In this paper, we present the application of the self‐healing concepts to assure the availability of measurements to the stakeholders without the need for effort‐intensive and costly manual interventions of the operators. We study the measurement infrastructure at one of the development units of a large infrastructure provider. In this paper, we present how the Monitor, Analyze, Plane, and Execute with Knowledge model was instantiated in a simplistic manner to reduce the need for manual intervention in the operation of the measurement systems. Based on the experiences from the 2 cases studied in this paper, we show how an evolution toward self‐healing measurement systems is done both with a dedicated failure taxonomy and with an effective straightforward handling of the most common errors in the execution. The mechanisms studied and presented in this paper show that self‐healing provides significant improvements to the operation of the measurement program and reduces the need for daily oversight by an operator for the measurement systems.

Background: The last ten to fifteen years, there has been a change in the way modern software organizations and companies work. The "classic" set-up with a project, where the project leader together with his/her project management team set up a time plan and execute on it, has been replaced by autonomous software development teams. Traditionally, measures were given top down, but now teams are asked to define and follow-up on their progress, by establishing their own measures. Goal: The goal of our research is to develop a set of measures for software development teams, addressing the research question of What should software development teams measure, to effectively monitor their progress? Method: We use an action research collaborative project with an infrastructure provider company. We developed the measures together with nine agile software development teams, using survey of needs, workshops about requirements, development of a dashboard and its evaluation in a mature measurement program. Results: The results show that the five measures we developed, in three different areas, cover the most important needs of agile software development teams for effective monitoring of their progress. Conclusions: Based of the results, we conclude that using the set of measures developed, software development teams can effectively monitor their progress. The limited number of areas monitored and measures used, limits the effort that the teams have to put on developing and monitoring these measures.

Software development companies have access to large amounts of software data, e.g. data that relate to the performance of their products (both during development and in field), data about finance, competitors. The software data is collected and analyzed; tables and graphs are developed showing status, progress, trends etc. This "post-process" of collected data, generates even more data. This situation creates, in one hand, a golden opportunity for these companies while raising at the same time a number of challenges. A golden opportunity, since the companies can analyze data to get a better understanding of e.g. the performance of their organizations and products and take fact based decisions. On the other hand, the large amount of data creates several challenges, e.g. how should the data be collected and stored, what makes measurement systems reliable [3], how the infrastructure should be set-up to support automation. Another challenge is how to identify those few important measures that provide the insight that is necessary for the organization to know that e.g. the product is being developed as specified; to know that there are no problems with the performance of the product when used by customers? In other words, given this plethora of data (e.g. more than 4 000 measurement systems), what should be measured?

Background: Modern software development companies increasingly rely on quantitative data in their decision-making for product releases, organizational performance assessment and monitoring of product quality. KPIs (Key Performance Indicators) are a critical element in the transformation of raw data (numbers) into decisions (indicators). The goal of the paper is to develop, document and evaluate a quality model for KPIs - addressing the research question of What characterizes a good KPI? In this paper we consider a KPI to be "good" when it is actionable and supports the organization in achieving its strategic goals. We use an action research collaborative project with an infrastructure provider company and an automotive OEM to develop and evaluate the model. We analyze a set of KPIs used at both companies and verify whether the organization's perception of these evaluated KPIs is aligned with the KPI's assessment according to our model. The results show that the model organizes good practices of KPI development and that it is easily used by the stakeholders to improve the quality of the KPIs or reduce the number of the KPIs. Using the KPI quality model provides the possibility to increase the effect of the KPIs in the organization and decreases the risk of wasting resources for collecting KPI data which cannot be used in practice.

Challenges of technical risk assessment is difficult to address, while its success can benefit software organizations appreciably. Classical definition of risk as a "combination of probability and impact of adverse event" appears not working with technical risk assessment. The main reason of this is the nature of adverse event's outcome which is rather continuous than discrete. The objective of this study was to scrutinize different aspects of technical risks and provide a definition, which will support effective risk assessment and management in software development organizations. In this study we defined the risk considering the nature of actual risks, emerged in software development. Afterwards, we summarized the software engineers' view on technical risks as results of three workshops with 15 engineers of four software development companies. The results show that technical risks could be viewed as a combination of uncertainty and magnitude of difference between actual and optimal design of product artifacts and processes. The presented definition is congruent with practitioners view on technical risk. It supports risk assessment in a quantitative manner and enables identification of potential product improvement areas.

Predicting reliability of software under development is an important part of estimations in software engineering projects. In many organizations as the goal is that software products are released with no known defects, the process of finding and removing defects correlates with the effort for software projects. Software development projects estimate the resources needed to design, develop, test and release software products, and the number of defects which have to be handled. In this paper we present a model for consequence analysis of inaccurate predictions of quality in software projects. The model is a result of multiple case studies and is evaluated at two companies. The model recognizes the most common mispredictions - e.g. Over- and under-prediction, early- and late-predictions - and the combination of theses. The results from the industrial evaluation show that the consequences can be grouped according to under- and over-predictions and that the late- and early-predictions have the same consequences. The results show also that mispredicting the shape of the reliability curve has a significant consequence with regard to assessment of release readiness and resource planning.

Monitoring performance of software development organizations can be achieved from a number of perspectives - e.g. using such tools as Balanced Scorecards or corporate dashboards. In this paper we present results from a study on using code stability indicators as a tool for product stability and organizational performance, conducted at three different software development companies - Ericsson AB, Saab AB Electronic Defense Systems (Saab) and Volvo Group Trucks Technology (Volvo Group). The results show that visualizing the source code changes using heat maps and linking these visualizations to defect inflow profiles provide indicators of how stable the product under development is and whether quality assurance efforts should be directed to specific parts of the product. Observing the indicator and making decisions based on its visualization leads to shorter feedback loops between development and test, thus resulting in lower development costs, shorter lead time and increased quality. The industrial case study in the paper shows that the indicator and its visualization can show whether the modifications of software products are focused on parts of the code base or are spread widely throughout the product.

Abstract The process of measuring in software engineering has already been standardized in the ISO/IEC 15939 standard, where activities related to identifying, creating, and evaluating of measures are described. In the process of measuring software entities, however, an organization usually needs to create custom measurement systems, which are intended to collect, analyze, and present data for a specific purpose. In this paper, we present a proven industrial process for developing measurement systems including the artifacts and deliverables important for a successful deployment of measurement systems in industry. The process has been elicited during a case study at Ericsson and is used in the organization for over 3 years when the paper was written. The process is supported by a framework that simplifies the implementation of the measurement systems and shortens the time from the initial idea to a working measurement system by the factor of 5 compared with using a standard development process not tailored for measurement systems. Copyright © 2010 John Wiley & Sons, Ltd.